Embeddings and vector search for RAG

Two passes over the same model

Vector search runs the embedding model at two different times. At index time, every chunk is embedded once and the vector is stored next to the chunk text and its metadata. At query time, the user's question is embedded and the index hands back the chunks whose stored vectors are nearest. Index-time work is done in bulk and amortized; query-time work happens on the hot path, once per request.

The non-negotiable rule is that both passes use the same model. A query vector and a document vector are only comparable if they live in the same space, and each model builds its own space. Swap the model on one side and the distances stop meaning anything. Some models go further and are asymmetric: they want a short prefix like query: on questions and passage: on documents, because a question and the paragraph that answers it do not look alike on the surface. If your model expects that, skipping it quietly degrades every search.

What "close" is measured with

Closeness needs a number, and there are three common ones: cosine similarity (the angle between vectors), dot product, and Euclidean distance (straight-line gap). For text, cosine is the usual default because it ignores vector length and scores only direction, which is where meaning lives. The detail that bites people: you have to use the metric the model was trained for. An index configured for Euclidean distance over vectors meant to be compared by cosine will return subtly wrong neighbors. Many setups sidestep this by normalizing every vector to unit length at index time, which makes dot product and cosine rank results identically.

Approximate nearest neighbor search

The honest way to find the nearest vectors is to compare the query against every stored vector and keep the closest. This is exact, and it is what a "flat" or brute-force index does. It is also linear in the size of your corpus, so at a few million chunks it gets too slow for an interactive query.

Production indexes use approximate nearest neighbor (ANN) search instead. They give up the guarantee of finding the exact nearest vectors in exchange for being orders of magnitude faster. Two families dominate. HNSW builds a layered graph of vectors and walks it greedily, hopping from node to nearer node until it settles, so it never has to look at most of the corpus. IVF clusters vectors around centroids and, at query time, only searches the few clusters nearest the query. Both expose a knob (HNSW's ef_search, IVF's nprobe) that trades recall for latency: turn it up to look at more candidates and miss fewer true neighbors, turn it down to go faster.

This is why two RAG systems with identical data can retrieve differently: the index settings, not just the embeddings, decide whether the best chunk comes back. Recall is measurable. Pick a set of known question-and-source pairs and check how often the right chunk lands in the top results, then tune the knob against that number rather than guessing.

Filtering and search fight each other

Most real queries are not pure similarity. You also need to restrict by tenant, language, product version, or document status. There are two ways to combine a filter with ANN search, and both have a failure mode. Post-filtering runs the vector search first and then drops candidates that fail the filter; if the filter is selective, it can throw away almost all of your top results and leave you with too few chunks. Pre-filtering restricts to matching vectors first and then searches; it returns the right population but can be slow, and on a graph index a very selective filter can strand the search in a poorly connected region and hurt recall.

Good vector databases offer filtered search that integrates the two, but you should know which mode you are getting. A retrieval bug that only appears for users in a small tenant or an uncommon language is very often this interaction, not the embeddings.

Similarity is not relevance

Vector similarity answers a narrow question: are these pieces of text close in embedding space? A genuinely relevant answer also depends on freshness, permissions, document type, exact terms, product version, and whether the chunk actually contains the answer rather than merely discussing the topic. The model is happy to return a paragraph that talks around a question without answering it, because on the surface it looks similar.

A query like "how do I rotate keys?" might retrieve chunks about API keys, encryption keys, keyboard shortcuts, or SSH keys. Semantic similarity gets you into the right neighborhood. It does not always pick the right door, which is exactly why the next lesson adds keyword search and reranking on top.

Choosing an embedding model

Every embedding model defines its own vector space, so switching models means re-embedding the whole index. Pick one that fits your domain, latency budget, and index size, then measure recall on your eval set rather than chasing leaderboard points alone.

Use public benchmarks (MTEB and task-specific leaderboards) as a shortlist, not a verdict. A model that tops general English retrieval may still miss your error codes, internal acronyms, or legal phrasing. Run your labeled question-and-source pairs through two or three finalists before you commit.

Latency in the table is order-of-magnitude for a single query embedding on typical hardware or a hosted API, not the full retrieval path. Index RAM scales with dimension count: a million chunks at 1536-dim float32 is roughly 6 GB of vector storage before metadata and index overhead.

What the vector database actually owns

A vector database or vector index usually handles:

• Storage for vectors, chunk IDs, and metadata.
• Nearest neighbor search over the embeddings, with the recall and latency knobs above.
• Metadata filters such as tenant, source, language, product, or date.
• Deletes and updates when source documents change, so stale chunks leave the index.
• Index tuning for the three-way budget of latency, memory, and recall.

The database does not understand your product truth by itself. It only searches what you stored and filters by what you modeled. Two costs are easy to underestimate. The index usually lives in memory for speed, and at a few million chunks with thousand-dimension vectors that is real RAM (vector storage is roughly chunk count times dimensions times four bytes for plain floats, before any compression like product quantization). And changing the embedding model invalidates everything: a new model means a new space, so you must re-embed every chunk and rebuild the whole index, which is a migration, not a config flip.

Common vector search failures

• Semantic drift. Results are topically related but do not answer the question.
• Exact-token misses. Error codes, function names, SKUs, and IDs may need keyword search.
• Embedding mismatch. The model embeds queries and documents poorly for your domain, or one side skips the expected query/passage prefix.
• Metric or knob mistakes. The index uses the wrong distance metric, or the ANN recall knob is set too low.
• Filter interactions. Search finds good chunks from the wrong tenant or version, or pre/post-filtering silently starves the result set.
• Score over-trust. A high similarity score is treated as proof that the chunk supports the answer.

Quick check