Do You Need a Vector Database?

Posted on In

Most RAG tutorials start by setting up a vector database. For a lot of projects that is the first piece of infrastructure they did not need yet. For others it is the right call, and knowing which group you are in saves a migration later.

This post covers what a vector database actually does, how far plain exact search gets you, what Postgres can do with pgvector, and the point where a dedicated store earns its place.

What a vector database does

Strip away the marketing and a vector database does two jobs. It stores vectors next to an id and some metadata, and it finds the vectors closest to a query vector, quickly, while respecting filters.

Animated: a chunk with metadata is embedded, added to an index and stored with its id; later a question is embedded the same way, the index is walked, filters apply, and the top matches come back as ids.

The "quickly" is the whole product. Finding the nearest vectors is easy if you compare the query with every stored vector. A vector database adds an index that lets it look at a small part of the data instead, and accepts that it will sometimes miss a true neighbour. That trade is called approximate nearest neighbour search, or ANN.

Everything else follows from that one feature: tuning how much accuracy to give up, keeping the index fresh while you write to it, and making filters work with an index that was built without knowing about them.

Start with exact search

Before you add an index, do the arithmetic. A typical embedding has 768 dimensions, stored as 4-byte floats, so one vector is about 3 KB. A hundred thousand chunks is around 300 MB. That fits in memory on a laptop.

Searching it exactly means one dot product per stored vector: 100,000 of them, about 77 million multiply-adds. Modern CPUs do that in tens of milliseconds on one core, often less. The result is exact, so recall is perfect by definition, and there is no index to build, tune or rebuild.

For a personal knowledge base, a documentation site or one team's tickets, that is often the end of the story. Keep the vectors in SQLite or a file next to the text, load them, compare. My own small RAG project keeps text, a keyword index and vectors in a single SQLite file, and the vector search is a straight scan.

Exact search stops being enough when the numbers grow. At ten million chunks you are scanning 30 GB per query, and no tuning saves that. That is the point where you need an index, which is not the same as needing a new database.

What an ANN index buys, and what it costs

The two index types you will meet most are HNSW and IVF.

  • HNSW builds a graph in layers. Each vector links to a few close neighbours, and a search walks the graph greedily from the top layer down, getting closer at each hop. It visits a few hundred vectors instead of millions.
  • IVF clusters the vectors first. A search finds the closest few clusters and scans only those lists.

Here is HNSW on 80 points in two dimensions. Exact search measures all 80. The graph walk starts on the top layer, drops a layer whenever it cannot get closer, and finds the same nearest point after 14 distance checks.

Both have a dial that trades speed for recall: how wide the graph walk is, or how many lists to probe. Turn it up and you find more true neighbours, more slowly. HNSW usually gives a better speed-recall trade than IVF, but takes longer to build and uses more memory. IVF needs representative data before you build it, because the clusters come from that data.

So an index brings three new problems: recall that is now a number you have to measure, memory and build time, and a setting someone has to own. Take them on when exact search is measurably too slow, not before.

Postgres is probably enough

If your data already lives in Postgres, the pgvector extension is the obvious next step. It stores vectors in a normal column, searches exactly by default, and adds HNSW and IVFFlat indexes when you need them.

What you get for free is the part people underestimate. The vector sits in the same row as the text and the metadata, so filters are ordinary SQL, a delete removes the vector in the same transaction, and backups, access control and monitoring are the ones you already run. There is no second copy to keep in sync.

Two limits are worth knowing before you start:

  • Indexes on the standard vector type support up to 2,000 dimensions; the half-precision type goes to 4,000. Some newer embedding models produce more than that, so check your model's output size before you pick it.
  • Filters are applied after the index scan. With the default HNSW search width of 40, a filter that matches 10% of rows leaves about 4 results on average, even if you asked for 10. pgvector 0.8.0 added iterative index scans, which keep scanning until enough rows pass the filter. Turn them on for filtered queries.

That filter trap applies to every ANN index, not just pgvector. It is the most common reason a RAG system "works in testing" and returns thin results in production, where every query is scoped to one tenant or one date range.

When a dedicated store earns its place

A dedicated vector database (Qdrant, Milvus, Weaviate, Pinecone and others) makes sense when one Postgres node is genuinely outgrown: hundreds of millions of vectors, high query rates, or search load you do not want on your transactional database.

Animated decision path: start with exact search; if it is too slow and you already run Postgres, use pgvector with HNSW; if filters thin results, turn on iterative scans; only when one node is outgrown, move to a dedicated store and plan the sync.

What they bring is real. Several of them build filtering into the index search rather than applying it afterwards, sharding and replication are part of the product, and quantization (storing vectors in fewer bits) is built in and tuned, which cuts memory several times over. Many also combine keyword and vector search in one query.

What they cost is a second system that holds a copy of your data. Every insert, update and delete now has to reach two places, and the day they disagree you will be answering questions from deleted documents. Plan that sync from the start: write changes to an outbox table in the same transaction and ship them, or stream them out of the database's change log. Dual writes from application code drift the first time one of them fails.

Two things that matter more than the database

Whatever you choose, two decisions shape answer quality more than the store does.

The embedding model is part of your schema. Vectors from different models are not comparable, so changing the model means re-embedding everything. Record which model and version produced each vector, and keep the text, so a re-embed is a batch job rather than a rescue.

Vectors are bad at exact strings. An error code, a function name or a product SKU often lands near the wrong neighbours. Run a keyword search next to the vector search and merge the two rankings (reciprocal rank fusion is the simple, robust way). Postgres full-text search or SQLite FTS5 is enough for that half.

The short version

  • Exact search over a few hundred thousand vectors is fast, simple and perfectly accurate. Start there.
  • Add an ANN index when a measurement says exact search is too slow, and measure recall when you do.
  • If you run Postgres, pgvector keeps vectors next to the data they describe. Check the dimension limits, and turn on iterative scans for filtered queries.
  • Move to a dedicated store when one node is truly outgrown, and plan the sync before the first write.
  • Store the embedding model with every vector, and pair vector search with keyword search.