It's really strange, you see people claim the same on the Codes subreddit. I can also run Astra xHigh at least for a few days with my 20x, but I don't use any of the token hungry patterns like orchestrators and such (only the code reviews have a skill using either 4 or 8 sub agents, some being Terra/Luna and none Astra). I did have a month ago a period or 2-3 days when usage was draining like 5x faster than usual, maybe some people are seeing smth like that? But more likely many sub agents and other patterns eating tokens.
Yet. But it’s built by the guy who created Vitess and cofounded PlanetScale, so while it doesn’t have sharding implemented at the moment I think it would be foolish to discount it. Plus it’s open source.
The PlanetScale team spent years improving Vitess with lessons learned from many different high scale customer workloads. We continue to invest in and improve Vitess. It's a substantial and growing part of our business.
Vitess today is not the Vitess of 2021. About 70% of the current codebase was written in the last five years, based on lessons we learned running it at scale.
Neki is built for Postgres from the ground up, by the engineers who did that work.
On the Multigres comparison, I'll admit we get a bit salty, mostly because "Vitess for Postgres" invites the comparison. But to date there's no public evidence of a working sharding implementation in Multigres. Their sharding design doc landed this week, and it lists cross-shard query planning and resharding execution as explicitly out of scope.
Neki does sharding today, including online resharding that switches traffic without a maintenance window. It isn't open source, but you can spin up a cluster and try it yourself.
This doesn't even make sense. Postgres is and has been open source since the 90s and AWS has had an existing Postgres offering since approximately always.
And there is no single company developing. Its a healthy mix of engineers from many companies. A perfect model to prevent this issue here of being aquihired.
> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.
The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine.
I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.
> The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine.
Disclosure: I work on Supermetal
You don't need to wait for a provider, and the provider is arguably the wrong place for this. They all have an incentive to make their own OLAP engine the happy path. A dedicated CDC tool that writes Iceberg to your own storage and catalog keeps the tables and the engine choice yours.
We built exactly that, a native Iceberg destination with Merge on Read. Since Snowflake and Databricks reject equality delete files, there's also a positional deletes only mode that writes deletion vectors instead, so the tables are readable from whatever engine you use.
> I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.
we deprecated this feature in our ETL tool[0]. The functionality is still in there but we can't support some of the production features we'd need for data/schema guarantees
Iceberg is still nascent - only supporting single-table transactions (at least when we tried). A lot of important CDC/transactional semantics were "a work in progress" upstream. We shifted our focus to ducklake, which stores the catalog in Postgres
I’ve built a similar thing against Snowflake, with one added layer: tenant-scoped views with the ID baked in, and an associated tenant-scoped role that only has read access to its own views.
It also lets you make updates to the underlying table (shared between all tenant views) and then separately update the views. As long as you aren’t making breaking schema changes you can more easily validate the changes before releasing.
> A data warehouse on the other hand is an OLAP database and is optimized to work on columns
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
You are pedantically correct but technically wrong, as even optimized postgres is going to suffer on analytical patterns without extensions. With extensions (DuckDB or Citus) you can do large aggregations, but regular postgres at medium/large scale (billions of rows, 100s of GB) starts having a lot of foot guns and complex babying to do analytics. A bunch of indexes and you'll be fine though.
reply