Chunking documents for RAG
Chunking is the part of RAG that sounds simple until it ruins your answers. The retriever does not search a whole document. It searches the pieces you chose to create.
The retriever never searches your documents. It searches the chunks you cut them into, so chunking decides what can be found at all. A chunk has to be small enough to retrieve precisely and large enough to carry the context that answers the question, and the right boundary is almost always a semantic one, not a character count.
Why chunks exist
Most source documents are too large to be a useful unit of retrieval. A policy page might cover vacation, sick leave, holidays, payroll, and approval rules. Embed that whole page and you hit a subtler problem than wasted prompt space: the embedding from lesson 04 is a single vector that averages everything in the chunk, so a page about five topics lands in a blurry middle of the space and matches a question about sick leave only weakly. This is embedding dilution, and it is the core reason whole documents make bad chunks.
Cutting the page into focused chunks fixes both problems at once. Each chunk's vector points somewhere specific, so retrieval is sharper, and the citation improves as a side effect: pointing a user at "Leave policy, Sick leave" is far more useful than pointing them at a 40-page PDF. The flip side is the warning for the rest of this lesson: cut too aggressively and a chunk no longer contains enough to answer anything.
The chunk size tradeoff
This is the tension at the heart of chunking. Small chunks match precisely but can strand the answer across a boundary or lose the context that makes it meaningful. Large chunks preserve context but dilute the embedding, match too many unrelated questions, and burn prompt budget. There is no universal best number, only the size that fits your content and your questions.
For prose, a few hundred tokens per chunk with a little overlap is a sane starting point. But "tokens" is the guardrail, not the goal. For API docs, code, legal text, tables, and transcripts, the natural unit is a heading section, a function, a clause, a row group, or a speaker turn, and those should drive the boundaries.
Split on semantic boundaries first, then use size limits only as guardrails. The common implementation is recursive: try to split on headings, then paragraphs, then sentences, and fall back to a fixed token window only when a single unit is still too big. Boundaries follow the document's own structure rather than a blind character count.
Overlap helps, but it is not free
Overlap repeats a little text between adjacent chunks so an answer is less likely to depend on a sentence that got cut away. It helps with boundary problems, especially in plain prose.
But overlap increases index size, retrieval duplicates, and prompt repetition. Too much overlap can make the top results look diverse while actually returning the same paragraph five times. Use enough to preserve continuity, then dedupe during retrieval.
Titles and headings carry meaning
A chunk often makes more sense when you prepend its document title and section path. The text "Requests must be approved by a manager" is vague on its own, and its embedding will be vague too. Prefix it with "Expense policy > Travel meals > Approval" and both retrieval and the model can interpret it correctly. The heading is cheap context that sharpens the chunk's meaning.
You can push this further with contextual enrichment: prepend a short, generated sentence that situates the chunk in its document, like "This section of the 2026 expense policy covers approval limits for travel meals." A chunk that begins with "It must not exceed the daily cap" is nearly unretrievable on its own because the pronoun has no referent; a one-line context header fixes the embedding and the answer at once. It costs a little extra processing at index time and pays off in recall. Anthropic's contextual retrieval write-up popularized this pattern: generate a short context prefix per chunk before embedding.
This is also why HTML and Markdown are easier to index than messy PDFs: their structure is explicit, so the headings and boundaries are already there for the taking. When structure exists, keep it rather than flattening everything to plain text.
Retrieve small, feed large
The chunk-size tradeoff feels like a forced choice between precise retrieval and enough context, but you do not actually have to pick. A common production pattern, sometimes called small-to-big or parent-child retrieval, splits the difference: you index small, focused chunks so matching is precise, but each small chunk points to a larger parent (its full section or a window of neighbors), and once a small chunk is retrieved you hand the parent to the model.
So the unit you search and the unit you read can be different. Search on the sharp sentence that matched the query, then give the model the surrounding paragraph or section so it has room to actually answer. This keeps embeddings clean without starving generation of context, and it is one of the higher-leverage changes when answers are accurate but feel thin or cut off.
Special cases
- Tables. Preserve headers with each row or row group. A row without column names is often meaningless.
- Code. Keep functions, classes, imports, and comments together when possible. Splitting in the middle of a function hurts both retrieval and explanation.
- Transcripts. Keep speaker labels and timestamps. The same sentence can mean different things depending on who said it.
- FAQs. Keep question and answer together. Embedding only the answer loses the phrasing users may search for.
Chunk IDs and source traceability
Every chunk should have a stable ID. That ID should point back to a source, version, and position. If a user reports a bad answer, you want to see exactly which chunk was used and whether that chunk still exists in the current source.
Good chunk metadata lets you rebuild indexes, compare old and new retrieval behavior, enforce permissions, and render citations that users can inspect.
Chunking changes are data migrations. If you change chunk size or boundaries, old retrieval scores, cached answers, and evaluation baselines may stop being comparable. Version your chunking strategy.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- What is embedding dilution, and why does it make whole documents bad chunks?
- Why can both tiny chunks and huge chunks hurt answer quality?
- How does recursive, structure-aware splitting choose boundaries?
- How does small-to-big retrieval get precision and context at once?
- Why is changing your chunking strategy a data migration?
Quick check
- Because fixed windows are always too slow
- Because fixed windows can cut through the middle of a meaningful unit
- Because embedding models cannot embed fixed windows
- They tell the retriever and model what the chunk is about
- They reduce token count
- They automatically enforce access control
- Its single vector averages all the topics, so it matches any one of them weakly
- The vector becomes too large for the index to store
- Because vector search cannot match long text at all
- Retrieve on small chunks but feed the larger parent section to the model
- Make every indexed chunk much larger
- Add heavy overlap between every chunk