So, Herman, you know how everyone was losing their minds over the vector database gold rush a couple of years back? It felt like every week a new company was raising a hundred million dollars just to store some lists of numbers. It was the peak of inflated expectations. Well, we are sitting here in March of twenty twenty-six, and the hangover has officially arrived. It is a math-heavy, cold-shower kind of hangover.
It is the brutal reality of the cost-optimization phase, Corn. We have moved past the wide-eyed wonder of "look, my computer can remember things" to the cold, hard accounting of "why am I paying five thousand dollars a month for a glorified index?" Herman Poppleberry here, by the way, ready to dive into the balance sheets and the memory maps. The industry has shifted from "how do I build this?" to "how do I keep building this without going bankrupt?"
Today's prompt from Daniel is the perfect catalyst for this. He is asking about the vector database landscape, specifically looking at the trade-offs between self-hosting, managed services, and how to actually run this stuff on serverless frontends without the bill making you weep. And honestly, it is about time we had this conversation because the pricing models have become a complete maze. You have got the old guard like Pinecone facing off against these scrappy newcomers like Turbopuffer, and the tension is palpable.
It really has shifted. If you look at the landscape right now, we have this clear bifurcation. On one side, you have the specialized, high-performance engines like Qdrant, Milvus, and Weaviate that are essentially the Ferraris of vector search. They are built for speed and massive scale. On the other side, you have the integrated cloud giants like Amazon Web Services Bedrock and Google Vertex AI, who are trying to make it a one-click experience for the enterprise crowd. And then, right in the middle, you have the serverless-native players like Turbopuffer and Upstash who are trying to solve the architectural nightmare of the edge.
Before we get into the weeds of the providers, I want to tackle the resource question Daniel raised. This is the "should I stay or should I go" moment for a lot of developers. If I want to stop paying Pinecone or Zilliz and just run my own instance of Qdrant on a virtual private server, what am I actually looking at? Is it going to eat my RAM for breakfast, or can I run it on a potato?
That is the big fear, but the answer depends entirely on how much performance you are willing to trade for cash. If you want everything to be lightning-fast, you have to pay the RAM tax. The standard formula for full in-memory storage is actually pretty straightforward, but it scales aggressively. You take the number of vectors, multiply it by the number of dimensions, then multiply by four bytes, since these are typically thirty-two-bit floats. Then, you have to add a fifty percent overhead for the indexing structures like HNSW and your metadata.
Okay, let's put some real numbers on that because "aggressive scaling" sounds like a polite way of saying "expensive." If I have one million vectors using the OpenAI text-embedding-three-small model, which is fifteen hundred and thirty-six dimensions, what does that look like on a server?
For one million vectors at fifteen hundred and thirty-six dimensions, you are looking at roughly nine gigabytes of RAM if you want everything sitting in memory for sub-ten-millisecond retrieval. That is manageable on a mid-tier virtual private server. But here is where it gets scary: if you scale that up to a hundred million vectors, you are suddenly looking at nine hundred gigabytes of RAM. At that point, your server cost is not just a line item; it is a mortgage payment. You are looking at thousands of dollars a month just for the hardware.
Right, so self-hosting for a massive dataset sounds like a nightmare if you are doing it all in RAM. But you mentioned a trade-off earlier. Is there a way to do this on a budget?
This is where the memory-mapped storage trick comes in, which is something Qdrant and some of the other Rust-based engines do brilliantly. It is often called "mmap." Instead of forcing every single vector to live in RAM all the time, you store them on the solid-state drive and let the operating system manage which ones are "hot" and need to be in the cache. If you use memory-mapping, you can run that same one million vector index on a server with just four to eight gigabytes of RAM, provided you have a fast NVMe drive. Your latency might go from five milliseconds to twenty-five milliseconds, but your monthly bill drops from hundreds of dollars to a twenty-dollar virtual private server.
Twenty dollars a month to host a million vectors? That seems like a massive win for anyone who does not need sub-ten-millisecond responses. I mean, twenty-five milliseconds is still faster than most human perception anyway, especially when you factor in the network latency of the LLM call that usually follows.
It is plenty fast for most retrieval-augmented generation use cases. The real "killer" of the self-hosted dream usually is not the database itself, though, it is the operational overhead. You have to manage the backups, the high availability, and the security patches. But from a pure hardware perspective, the "mmap" strategy has made self-hosting viable for almost any startup. It democratizes the technology. You can actually compete with the big boys on a twenty-dollar server if you know how to tune your storage.
Let's talk about the other side of the ledger, which is the embeddings themselves. Daniel asked about the cost of generating these things. I feel like people often forget that you pay twice: once to turn the text into numbers, and once to store and search those numbers. It is like paying for the translation and then paying for the library shelf space.
You are spot on. And the pricing for embeddings has actually seen some of the most aggressive cuts in the last year. If you are using OpenAI's text-embedding-three-small, you are looking at about two cents per million tokens for standard requests. But here is the pro tip that most people miss: if you use their batch API, it is half that. One cent per million tokens.
So if I have a massive library of documents and I am not in a rush to index them in real-time, I should just batch them and save fifty percent? Why isn't everyone doing that?
Mostly because people are impatient or they haven't updated their pipelines. If you have a million average-sized documents, say five hundred tokens each, that is five hundred million tokens. At the standard rate, that is ten dollars. It is almost negligible compared to the storage costs over time. The real expense comes if you get fancy and use the "large" models. The text-embedding-three-large is about thirteen cents per million tokens. That is a six-times increase in cost. You have to ask yourself if that extra precision is actually translating into better search results for your users.
It usually does not, does it? I feel like for eighty percent of apps, the small model is more than enough.
For most retrieval-augmented generation, the bottleneck is not the embedding dimensions; it is the quality of your chunking strategy and the context window of your LLM. Moving from fifteen hundred dimensions to three thousand dimensions is rarely the silver bullet people think it is. In fact, it often just makes your database slower and more expensive without improving the actual answer quality.
Now that we have looked at the hardware math and the token costs, let's talk about the architectural nightmare of running this on the edge. Daniel wants to know if you can use these things in applications deployed on serverless frontends, like Vercel or Cloudflare Workers. I know we have talked about this before, but the persistent connection problem is still a thing, right?
It is the bane of the serverless developer's existence. Traditional databases like Milvus or even a self-hosted Qdrant often rely on gRPC or persistent TCP connections. When you spin up a Vercel function, it lives for a few seconds and then dies. If it has to spend three hundred milliseconds establishing a new handshake with a database every time it runs, your performance goes out the window. It is like trying to have a conversation where you have to re-introduce yourself every time you speak.
So if I am building on the edge, I am basically locked out of the self-hosted world unless I put an HTTP proxy in front of it?
Pretty much. But the market has responded with some really clever "HTTP-native" options. This is why we are seeing the rise of players like Turbopuffer. They are essentially built on top of object storage like Amazon S3. Instead of a persistent server cluster that has to stay awake, they treat the storage layer as the source of truth and use aggressive caching.
I have been seeing Turbopuffer's name everywhere lately. Cursor, Notion, and Linear are all using them, right? What makes them different from, say, Pinecone Serverless?
Architecture and economics. Pinecone Serverless is great, but their pricing is based on "read units," which can get very expensive if you have a high-volume app. Turbopuffer is built to be stateless. You interact with it entirely over HTTP, so there is no connection overhead for your serverless functions. It is also incredibly cheap for high-scale reads because of how they leverage object storage. They have essentially bet that S3 is the future of database storage, and for serverless apps, they might be right.
Is there a catch? There is always a catch when someone says it is "stateless and cheap."
The catch is usually cold-start latency and the lack of a free tier. Turbopuffer starts at about sixty-four dollars a month. If you are a hobbyist just trying to build a weekend project, that is a steep entry fee. For the hobbyist level, you are better off looking at something like Upstash Vector or even Cloudflare's own Vectorize. Upstash uses a Redis-like model with a REST API that is very friendly to edge functions, and they have a generous free tier.
Cloudflare Vectorize is an interesting one. Since it lives right inside the Cloudflare ecosystem, does it bypass that whole "external API latency" issue?
It does, but it is a bit of a walled garden. It is brilliant if you are already all-in on Cloudflare Workers. They have a free tier that lets you store up to five million vectors, which is massive for a free offering. But it is strictly vector search. If you need "hybrid search," where you combine vector similarity with traditional keyword searching, Vectorize is still catching up. Turbopuffer and Supabase are much stronger in that hybrid category.
That hybrid search bit is actually a huge point. We did a deep dive on why "pgvector" is often the "good enough" solution in episode twelve twelve, and one of the big reasons was that you get to keep your traditional database features. How does pgvector hold up in this twenty twenty-six serverless world?
It is actually winning the "middle of the road" category. If you use a provider like Neon or Supabase, they have built custom connection poolers that make Postgres work beautifully with serverless functions. Neon, specifically, can scale to zero, so you are not paying for the database when nobody is using your app. For a developer who wants one database for their users, their posts, and their vectors, it is really hard to beat. They added HNSW support in version zero point five point zero, which closed the performance gap with specialized engines for most use cases.
So the "one size fits all" dream is still alive with Postgres?
To a point. If you have ten million vectors, pgvector with an HNSW index is fantastic. If you have a billion vectors and you need to perform complex filtering on metadata while searching, that is when you start looking at a dedicated engine like Qdrant or Milvus. But for eighty percent of the people listening to this, pgvector is the correct answer. It prevents "architectural sprawl," which we talked about in episode eleven twenty-four.
I want to go back to something you mentioned earlier. You called Cloudflare Workers AI a "sleeper pick" for embeddings. Why is that?
Because it solves the "double hop" problem. Usually, your serverless function has to call OpenAI to get an embedding, wait for that to come back, and then call your vector database. That is two external network requests. Cloudflare has built-in embedding models that run on their own GPUs at the edge. You call one function inside your worker, it generates the embedding locally in the same data center, and then you send it to your database. It is faster, and for the first ten thousand requests a day, it is free. It uses the B-G-E base model, which is very solid.
That is actually a huge deal for latency. If I am trying to build a chat interface that feels "snappy," cutting out that extra trip to OpenAI's servers is a massive optimization. It is the difference between a UI that feels alive and one that feels like it is lagging.
It really is. And it brings us back to Daniel's question about resources. If you use the Cloudflare model, you are using zero of your own "resources" and zero of your API budget. It is a very clean way to build a production-grade RAG pipeline on a shoestring. You are essentially offloading the hardest parts of the math to Cloudflare's infrastructure.
Okay, let's look at the enterprise side for a second. We have been talking a lot about startups and hobbyists. What happens when you are a massive company with eighty million queries a month? At that scale, does Pinecone Serverless still make sense?
This is where the math gets really interesting and where some companies are actually moving away from managed services. If you are doing eighty million queries a month on Pinecone, your bill could easily be north of fifteen thousand dollars just for the "read units." If you take that same workload and put it on a cluster of self-hosted Qdrant nodes on dedicated hardware, your cost might drop to two thousand dollars a month.
So at the high end, we are seeing a "re-repatriation" of data? People going back to managing their own servers just to escape the "serverless tax"?
We absolutely are. It is the classic cloud cycle. You start with serverless because it is easy and you have no users. Then you get successful, your bill explodes, and you hire a DevOps engineer to move everything to bare metal or a fixed-cost virtual private server to save eighty percent on your margins. We are seeing this with companies like Zilliz too. Their dedicated tier starts at around a hundred and fourteen dollars a month, which is their way of trying to capture that middle ground before people flee to self-hosting.
It is funny how that works. We spend all this time inventing abstractions just to realize that the most efficient thing is often just a well-tuned server with a lot of NVMe storage.
Precisely. The "mmap" trick I mentioned earlier is the key to that transition. It allows you to buy a server with a decent amount of RAM but massive storage, and the operating system does the heavy lifting for you. It is the ultimate "work smarter, not harder" strategy for database engineering.
Let's talk about the "Big Cloud" integrated options for a second. AWS Bedrock Knowledge Bases and Google Vertex AI Vector Search. Are those just for people who are already trapped in those ecosystems, or is there a genuine technical reason to use them?
It is mostly about security and compliance. If you are a bank or a healthcare provider, and all your data is already in an Amazon S3 bucket, it is much easier to get your legal team to approve "AWS Bedrock" than it is to get them to approve "Turbopuffer." Technically, they are very capable. Google's Vertex AI Vector Search, for instance, has incredible low-latency performance for massive scales, but the developer experience is... let's just say it is very "Google." It is complex.
That is a polite way of putting it. I have tried setting up some of those Google Cloud services and I felt like I needed a PhD in IAM roles just to see a "hello world" response.
You and everyone else, Corn. But for a certain scale of enterprise, that complexity is a feature because it comes with the "nobody ever got fired for buying IBM" level of support. If you are at that scale, you aren't worried about a twenty-dollar VPS; you are worried about five-nines of availability and a service level agreement.
Alright, so we have covered a lot of ground. We have the self-hosted math, the embedding costs, and the serverless landscape. Let's try to distill this into a decision matrix for the listeners. If someone is starting a project today, how should they choose?
I think there are three clear paths. Path one: you are a developer building a standard app and you want to keep things simple. Use pgvector on a platform like Supabase or Neon. It is free to start, it scales to ten million vectors easily, and you do not have to learn a new query language. It is the "default" choice for a reason.
Simple, effective. I like it. What is path two?
Path two: you are building a "serverless-first" app on Vercel or Cloudflare and you need it to be fast and maintenance-free. Use Turbopuffer if you have a budget and need high performance for a professional app, or Upstash Vector if you want a great free tier and a Redis-like experience. These are built for the edge and won't give you connection headaches.
And path three? The "I have a billion vectors and a CFO who is breathing down my neck" path?
Path three is self-hosting Qdrant or Milvus on a dedicated server or a high-performance virtual private server. Use the memory-mapped storage configuration to keep your hardware costs low, and put an HTTP proxy in front of it if you need to talk to it from serverless functions. This is where you get the most "bang for your buck" once you pass that one-million-vector mark.
That feels like a very solid roadmap. It is interesting to see how much the "mmap" strategy has changed the math for the little guy. You can actually compete with the big boys on a twenty-dollar server.
It has democratized the technology in a way that I think people are still waking up to. We saw a similar thing with the "Database Explosion" we talked about in episode eleven twenty-four. Specialized tools eventually get optimized to the point where they can run on almost anything. The "vector database" is no longer a mystical black box; it is just another tool in the shed.
Before we wrap up, I have to ask: do you think vector databases as a standalone category will even exist in five years? Or is this all just going to be a feature of every database, like full-text search is today?
That is the million-dollar question. My take is that "vector search" will be a feature of every database, but "vector engines" will remain a specialized category for the high end. Just like how every database can store a JSON blob, but people still use MongoDB for massive document workloads. If you are doing basic retrieval-augmented generation, you will just use your primary database. If you are building a recommendation engine for a billion products, you will still want a specialized engine like Qdrant or Milvus.
That makes sense. The "good enough" solution will swallow the bottom of the market, and the specialized tools will retreat to the high-performance peaks. It is the natural evolution of software.
And honestly, that is a win for everyone. It means developers can start simple and only move to the complex stuff when they actually have the scale to justify it. No more over-engineering on day one.
Well, I think we have given Daniel and everyone else a lot to chew on. The math is clear: batch your embeddings to save fifty percent, use memory-mapping if you are self-hosting to save ninety percent on RAM, and choose your serverless provider based on whether you need a free tier or high-volume performance.
And don't over-engineer it on day one. Start with pgvector and only move when your database starts sweating. Most people never actually reach the point where they need a dedicated vector cluster.
Solid advice as always, Herman. I'll refrain from making any more sloth jokes for at least five minutes as a reward for that thorough breakdown.
I'll take what I can get, Corn.
Big thanks to our producer, Hilbert Flumingtop, for keeping the gears turning behind the scenes. And a massive thank you to Modal for providing the GPU credits that power this show's generation pipeline.
If you found this deep dive into vector economics useful, a quick review on your podcast app really helps us reach more curious nerds like you. It keeps the lights on and the servers running.
This has been My Weird Prompts. We will be back next time with whatever strange topic Daniel or the rest of you throw our way.
See you then.
