What a vector database actually solves, the honest intuition behind approximate search, and the pgvector-or-dedicated-engine decision. The plain, complete version.
Your prototype holds three hundred chunks. When a question arrives, the code does the least clever thing possible: it compares the question's vector against every stored vector, keeps the closest five, and hands them to the model. The whole search takes a blink. Then the pilot goes well. Two more teams add their documents, then someone loads the entire company wiki, and the corpus is four million chunks. The same loop now makes four million comparisons for every question, while somebody watches a spinner.
Notice what didn't break: the idea. Nearest-point-wins is still exactly the right search. It just stopped being fast, and that is the entire reason vector databases exist. Storage was never the point; any file can hold a list of numbers. The point is keeping the search quick.
If you keep one thing from this page, keep this: a vector database buys you a single trade — "find the nearest vectors" stays fast at any corpus size, and the price is that the answer becomes approximately right instead of exactly right. Everything else on the product pages, the dashboards and SDKs and pricing tiers, is wrapped around that one bargain.
How it works
Start with the baseline. Exact nearest-neighbor search compares the question to everything, so it's perfectly correct, and its cost grows in a straight line with your corpus. Ten times the chunks, ten times the work. At a few thousand vectors that line is so cheap nobody notices it. At a few million, it's the spinner.
The fix is to stop looking at everything. Ahead of time, the database builds an index, a structure that lets a search skip almost every vector it holds. The one you'll meet most often is called HNSW. It sits inside pgvector, Qdrant, Weaviate, and most of their peers, and the best intuition for it is a road network.
Picture every vector as an address on the map from the embeddings piece. The index connects each address to its near neighbors, the local streets, then lays sparser layers of long-range links on top, like highways over avenues over streets. A search enters on the highway layer, takes a few big hops toward the right region, drops down a layer, refines, drops again, and finishes on the local streets of one small neighborhood. It checked a sliver of the map and still landed next to the target.
Now the catch. The search moves greedily, always toward whichever connected point sits closest to the question, exploring only a narrow frontier around its path and rarely doubling back down roads it skipped. Occasionally the true nearest vector sits down one of those skipped roads. So the results are approximately the nearest: usually the set exact search would return, sometimes missing one. Engines expose this as a dial. Turn it up and the search looks wider and misses less, at the cost of speed. Turn it down for the reverse.
In practice the miss matters less than it sounds. Embeddings are already a fuzzy map, so first-nearest versus third-nearest is rarely a right answer versus a wrong one, and a decent pipeline retrieves a batch of candidates and lets a re-ranker sort the ordering anyway. Approximation is the price of speed, and it's usually a fair price. Usually.
Vector search alone is rarely enough
Real questions arrive with conditions attached. Only this customer's workspace. Only the current policy version, not the three it replaced. Pure vector search knows nothing about any of that, because nearness has no tenant column. So every serious setup pairs the similarity search with metadata filters: the labels you attached back in the chunking piece, finally paying rent.
Most systems also run keyword search beside the vector search, the hybrid setup from the embeddings piece, because meaning-search fuzzes exact strings and users keep typing order IDs. Vectors plus filters plus keywords: that's the normal shape of production retrieval, and a better test of an engine than any raw speed claim.
Where it breaks
Recall is a dial, not a guarantee. An approximate index sometimes misses the true best chunk and won't tell you. From the outside it looks identical to a chunking or embedding failure: the right passage existed and didn't come back. Before you rebuild anything upstream, remember the dial exists and check where it's set.
Filters fight the index. Filter after the search and you can keep nothing: the ten nearest chunks were all from the wrong tenant, and the right answer sat at rank eleven. Filter before it and a strict filter turns most of the graph into dead nodes the search must route around. Engines carry real machinery for this, but it's a hard problem everywhere. Test with your real filters at their real selectivity, not an unfiltered demo query.
Stale vectors after re-chunking or re-embedding. The embeddings piece had one iron rule: documents and questions must share a model. The database is where that rule comes due. Swap the model and every stored vector is a coordinate on a map that no longer exists, which means a full re-embed and re-insert, not an update. Re-chunk and it's the same story. Let documents change while the index doesn't, and the system faithfully quotes text nobody can find anymore; a surprising share of reported hallucination is really an index out of sync with its documents.
Rebuilds are a real operation. An index's shape is decided when it's built. Change the embedding model or retune the build settings and you're rebuilding, which on millions of vectors takes real time and memory while queries still need answering. Plan it like a migration, not a checkbox.
It's one more system. A dedicated engine means another thing with backups, upgrades, authentication, monitoring, and a sync job keeping it honest against your source-of-truth database. You pay that cost in ops attention, and for a small team it's routinely the deciding one.
The decision: array, pgvector, or a dedicated engine
There are three tiers.
A plain array is genuinely fine when the corpus is small, meaning a few thousand chunks or even tens of thousands. Load the vectors into memory, brute-force the comparison, take the top five. It's exact, it's a blink, and there is no infrastructure to run; the only thing that can fall out of date is the array itself. One product's docs, one team's runbooks: this tier covers more real projects than anyone admits, and no architecture diagram should shame you out of it.
pgvector, when you already run Postgres (which is most teams). It's an extension, so vectors live in a column next to the rows they describe, and filters are just WHERE clauses. You keep one backup story and one connection string instead of gaining a second system. Without an index it's the brute-force loop with SQL around it; add its HNSW index as the corpus grows and it carries you comfortably into the hundreds of thousands of vectors, often well past. Here's the judgment call: a two-person team already on Postgres with under half a million vectors should almost always just use pgvector. A dedicated engine there hands a team with no slack a second data system, complete with its own failure modes and its own 2 a.m. pages.
A dedicated engine, when the numbers force it. Many millions of vectors and climbing. Latency budgets that stay tight under heavy filtered and hybrid load, or search that has to scale separately from the application database. That's when the Pinecone / Qdrant / Weaviate class earns its keep, and the differences between them matter far less than the difference between needing one and not. Managed means paying to make the ops someone else's problem; self-hosting means having a team with room to own it. Choose by ops appetite, not by feature table.
And starting small is not a trap. Moving up a tier is mostly re-inserting your vectors — at worst re-embedding your chunks, the operation the stale-vectors bullet says you'll do now and then anyway. The chunking and embedding pipeline is the part you keep. The database is a decision you're allowed to revisit.
Whichever tier you land on, measure it against real questions before you trust it, filters and all. (More on that when we reach evals.)
Where to go next
Vector databases are the third piece of the knowledge retrieval model: chunking cut the documents, embeddings turned each piece into a point, and the database is where those points get searched at speed. The rest of the map:
Re-ranking — the second, sharper pass that fixes "close, but not the best," and part of why an approximate miss costs less than you'd fear
Evals — proving that a database swap, an index rebuild, or a turn of the recall dial actually helped
then the project: build a production RAG, end to end.
New here? Start at What is RAG, really?. The two pieces before this one, Chunking and Embeddings, make the points this page is searching. And to watch retrieval fail on real users and get fixed, that's the war story: Your RAG worked in the demo. Then real users arrived.
The vector database is the most branded and most argued-about component in the stack, and the least decisive one. Teams spend weeks comparing engines while shipping chunks that average into mush and embeddings nobody ever measured. The database only searches the map it's given; it cannot sharpen a blurry point or resurrect an answer cut in half upstream. Pick the smallest thing that fits, and put the attention you saved where retrieval is actually won, which is upstream: in the cuts, in the map, and in the evals that tell you whether either one works.
