The RAG retrieval pipeline

Two paths, one answer

A working RAG system is really two pipelines that meet at the index. The offline path runs before anyone asks a question: it turns source material into searchable units and writes them to the index. The online path runs at request time: it interprets the question, searches the index, selects evidence, and asks the model to answer.

The split is not cosmetic, because the two halves have opposite constraints. Offline ingestion can be slow, batched, and careful; it runs on a schedule or a queue and nobody is waiting on it. Online retrieval sits on a user-facing request with a latency budget measured in hundreds of milliseconds. Mixing them is a common early mistake. If you find yourself parsing a PDF or re-embedding a whole document inside the request handler, work that belongs offline has leaked online.

The offline path

Ingestion starts with source documents: HTML pages, PDFs, Markdown files, tickets, database rows, transcripts, code, or anything else the system should be able to search. The job is to turn those sources into chunks, each carrying a stable ID, clean text, metadata, and an embedding. Roughly, the stages are extract, clean, chunk, enrich with metadata, embed, and write to the index.

Extraction is where quality quietly leaks. A PDF is a layout format, not a text format, so naive extraction drags in page footers, headers, line numbers, and reflowed columns; HTML brings nav menus, cookie banners, and duplicated sidebars; tables collapse into word soup. None of that is what the user wants to retrieve, and if the index stores junk, search will confidently find junk. Treat extraction and cleaning as real work, not boring plumbing, and dedupe near-identical content so the same boilerplate does not crowd out real answers.

Every chunk should record where it came from. At minimum keep the source URL or file path, title, section heading, updated time, access scope, and a chunk ID that is stable across re-ingestion. Stable IDs are what let you update or delete a single chunk later instead of rebuilding the world, and they are what make a citation point back to a real place when the model finally answers.

The online path

At request time the system receives a question and has to turn it into one or more searches. Sometimes the raw question is enough. Often it helps to do a little query understanding first: rewrite a vague follow-up into a standalone question using the chat history, expand acronyms, attach filters the user implied ("our policy" means their tenant), or split a broad question into sub-queries that are each searchable.

From there the online path is a funnel. Retrieval pulls a broad candidate set, usually far more chunks than the model will ever see. Then ranking, filtering, and deduplication narrow it, and context assembly picks the final few that go into the prompt. Each stage throws information away on purpose. The whole art is throwing away the right things, and most RAG quality is won or lost between "we retrieved 100 candidates" and "we sent 5 to the model." The mechanics of that narrowing, vector search, hybrid search, and reranking, are the subject of the next two lessons.

Query transformation (optional upgrades)

Naive RAG embeds the user's question as-is. That works when the question is clear and self-contained. It struggles when the query is vague, uses different vocabulary than the docs, or is a follow-up that only makes sense with chat history. Query transformation rewrites the question before search, as an optional layer on the online path.

Two patterns worth knowing:

• Multi-query retrieval. An LLM (or a template) generates several search queries from one user question, each phrased differently. You run retrieval for each, merge the candidate pools, and dedupe. This raises recall when a single embedding of the question lands in the wrong neighborhood.
• HyDE (Hypothetical Document Embeddings). Ask the model to draft a short hypothetical answer or passage, embed that text instead of the raw question, and search with it. The fake passage often sits closer in embedding space to real docs than a terse question does. It costs an extra LLM call and can drift if the hypothesis invents facts, so it is best treated as a recall booster you validate on your eval set, not a default.

Step-back prompting (rewrite a specific question into a broader one) and conversational query rewriting (turn "what about contractors?" into a standalone question using history) fit the same family. They are cheap wins when recall is the bottleneck and the index is otherwise healthy.

Metadata is part of retrieval

Embeddings capture semantic similarity, but metadata gives you control. If a user asks about the current vacation policy, you probably want filters like doc_type:policy, status:published, and department:hr alongside the semantic search. If a customer asks about their contract, access control and account ID matter more than how similar the text is, and they are not optional: getting them wrong is a data leak, not a ranking miss.

So metadata is not decoration you bolt on at the end. It is part of the retrieval query, and it has to be modeled during ingestion, because you can only filter on fields you actually stored. Without it the retriever has to infer everything from text similarity, which is fragile. With it you can shrink the search space to the eligible population before ranking even runs.

Assembling the context window

The last online stage before generation is often the most underrated: turning a ranked list of chunks into the actual text block the model reads. This is not just concatenation. You deduplicate chunks that say the same thing, order them so the strongest evidence sits where the model attends best, and stop adding once you hit a deliberate token budget rather than dumping everything you retrieved. Each chunk should carry its source label into the prompt so the model can cite it and so you can trace the answer afterward.

Two failure modes live here. Pass too little and the answer is ungrounded; pass too much and the real evidence gets buried among loosely related chunks, costs more, and runs slower. Context assembly is where the retrieval pipeline hands off to prompting, which the prompts and citations lesson picks up in detail.

Freshness and the sync problem

The index is a copy of your sources, and copies drift. When a document changes, the indexed chunks are wrong until ingestion runs again, and the gap between "source changed" and "index updated" is real lag a user can hit. Worse than slightly stale is content that should be gone: if a document is deleted or a chunk is removed and the index still holds it, retrieval will happily surface a deleted policy as if it were current.

This is why stable chunk IDs and a real update-and-delete story matter. You need a way to re-ingest only what changed, propagate deletes, and know how fresh the index is. Reindexing the whole corpus on every change does not scale, and never reindexing is how RAG systems quietly start lying. You do not have to solve incremental indexing today, but you should design the offline path knowing the index is a cache that has to be invalidated.

Common pipeline failures

• Stale index. The source changed, but the indexed chunks did not.
• Lost provenance. The chunk has text, but no useful source URL, title, or section.
• Bad access control. A user can retrieve content they should not see.
• Overstuffed context. Too many chunks bury the useful evidence.
• No retrieval logs. You cannot inspect which chunks were returned for a bad answer.

Go deeper

Quick check