What is retrieval-augmented generation?

RAG gives an LLM a search step before it answers. Instead of asking the model to rely only on what it learned during training, you retrieve relevant source material and ask the model to answer from that context.

The one idea

RAG is a system pattern, not a kind of model: retrieve evidence first, generate second. The model still writes the answer, but the system decides which facts are allowed into the room. Get that selection right and a general model answers accurately about your private, current data without ever being retrained.

The basic loop

A plain LLM call has one main input: the prompt. The model reads it, uses its learned weights and the current context window, then generates tokens. That works well for general reasoning and writing, but it is weak when the answer depends on private docs, fresh product state, internal policies, tickets, logs, or exact citations.

RAG adds a retrieval step before generation:

The user asks a question.
The system searches a document collection for relevant chunks.
The system puts those chunks into the model context.
The model answers using the retrieved evidence.

RAG is not a model type. It is the retrieval and prompting machinery around a model.

Two kinds of memory

It helps to think of a RAG system as having two kinds of memory. The model's weights are parametric memory: everything it absorbed during training, baked in and frozen. It is fast and fluent but fixed at a knowledge cutoff, impossible to update without retraining, and unable to tell you where a fact came from. When asked about something just outside that memory, a model tends to produce a confident, plausible, wrong answer, which is the failure we call hallucination.

Retrieval adds non-parametric memory: an external store you can read at request time and change whenever you like. Your company handbook, current pricing, last night's incident, a customer's contract terms, and private code live there, not in the weights. The operational story is the payoff: update the document, reindex, and the next answer reflects the change, with a citation that points back to the source. RAG is the bridge that lets a frozen model answer from living data.

RAG, fine-tuning, or a bigger context window?

Newcomers often treat these as competitors. They solve different problems, and knowing which to reach for is half of using RAG well.

RAG changes what the model knows for a given question by fetching facts at request time. Reach for it when the answer depends on a large or changing body of knowledge, when you need citations, or when access control and freshness matter.
Fine-tuning changes how the model behaves: tone, format, a specialized skill, or a domain's style. It is the wrong tool for keeping facts current, because the facts get frozen into weights again. The two combine well: fine-tune for behavior, retrieve for knowledge.
A bigger context window lets you paste more text into a single prompt, but it is not a substitute for retrieval once your corpus is larger than a prompt, or when stuffing everything in is slow, costly, and dilutes the model's attention. RAG is how you choose which slice of a big corpus to put in that window.
Tools and function calling let the model take actions or query live systems. Retrieval is often just one tool among several in an agent; "search the docs" is a tool call.

Rule of thumb

Need current or private facts, with sources? That is RAG. Need a consistent behavior or style? That is fine-tuning. The question "how do I make the model know our docs?" almost always means RAG, not training.

What RAG is good at

RAG is a strong fit when the answer should be grounded in a known corpus:

Internal knowledge base search with natural language answers.
Support agents that cite help center pages.
Developer assistants that read project docs and code snippets.
Legal, policy, or compliance lookup where source traceability matters.
Research assistants that summarize a bounded set of documents.

The common thread is not "make the model smarter." The goal is narrower: put the right evidence in front of the model at the right time.

What RAG does not solve

RAG reduces unsupported answers, but it does not make a system automatically correct. It adds a pipeline, and every stage of that pipeline is a place a good answer can die before generation even starts. The rest of this course is essentially a tour of those failure points:

Indexing can store junk or never ingest the source at all (lesson 02).
Chunking can split the needed detail across a boundary or dilute it (lesson 03).
Retrieval can miss the right chunk, or rank it below noise (lessons 04 and 05).
The prompt can let the model ignore the evidence, blend conflicting sources, or invent a citation (lesson 06).
A stale index can serve confidently wrong, out-of-date facts.

This is why production RAG is far more than vector search. It is content processing, retrieval, ranking, context budgeting, prompt design, citation control, logging, and evaluation (lesson 07). The good news is that these stages fail in distinguishable ways, so a disciplined system can tell which one broke instead of just declaring "the model is bad."

Engineering reality

Most bad RAG systems fail before the LLM call. The evidence is missing, stale, too broad, duplicated, or badly ranked. The model then writes a polished answer from weak inputs, and everyone blames the model.

The boundary between retrieval and generation

Keep this separation clear:

Retrieval decides what evidence the model sees. This is where recall, ranking, filtering, access control, and freshness live.
Generation decides how to turn evidence into an answer. This is where summarization, tone, abstention, and citation formatting live.

When a RAG answer is wrong, debug the two halves separately. First ask whether the right evidence was retrieved. Then ask whether the model used it correctly.

Naive RAG vs an advanced pipeline

Naive RAG is the textbook loop from this lesson: embed chunks, search top-k, paste them into a prompt, generate once. It is a fine v1 and a useful mental model, but production systems quickly add stages the naive version skips: structure-aware chunking, hybrid search, reranking, query rewriting, citation validation, freshness sync, and evals that measure each stage.

An advanced pipeline keeps the same core idea (retrieve, then generate) but treats retrieval as a funnel with logging, filters, and tuning knobs rather than a single vector call. The lessons ahead walk through that funnel piece by piece. You do not need every upgrade on day one; you do need to know which layer broke when an answer fails.

Beyond one-shot RAG

Agentic RAG runs retrieval inside a loop: the model (or harness) can search, read results, decide it needs another query, search again, and only then answer. That pattern shows up in research assistants and support agents where one retrieval pass is not enough. It is the same evidence pipeline as this course, wrapped in an agent loop with tools. When you are ready for that shape, continue with the Agents, Tools & Harnesses course.

Checkpoint

You're ready for the next lesson if you can answer these from memory:

What extra step does RAG add before generation?
What is the difference between parametric and non-parametric memory?
When would you reach for RAG instead of fine-tuning or a bigger context window?
How does naive RAG differ from an advanced pipeline?
What is the difference between retrieval quality and generation quality?
Why does RAG not remove the need for evaluation?

Quick check

To write the final answer before the model sees it
To find relevant evidence and place it in the model context
To train the model on every document before each question

Whether the retrieved chunks contained the needed evidence
Whether the temperature was exactly zero
Whether the prompt can fit more documents

RAG, so the facts stay in a store you can update and reindex
Fine-tuning, to teach the model the documents
A bigger context window alone, with no retrieval