Evaluating RAG systems

Start with a real eval set

Everything here depends on a test set that looks like real use. Collect questions users actually ask, or questions that behave like them, and cover the hard cases on purpose: easy lookups, ambiguous questions, exact-identifier searches, multi-hop questions that need two sources, stale-document traps, permission-sensitive questions, and questions the system should refuse. A set that is all easy lookups will tell you the system is great right up until it ships.

For each question, store the expected source chunks or documents, not just a golden answer. This is the detail people skip, and it is the most important one. If you only store the answer, a response that looks similar will score as a pass even when retrieval found nothing and the model guessed from training memory. Storing the expected sources is what lets you separate retrieval from generation at all. A few hundred well-chosen, labeled questions beat tens of thousands of scraped ones, and keep them out of any prompt or fine-tuning data so the eval stays honest.

Evaluate retrieval first

Retrieval evaluation asks a narrow, answerable question: did the evidence the answer needs actually come back? The core metric is recall@k, the fraction of questions whose expected source appears in the top k results. Recall is what you protect most fiercely, because a chunk that never gets retrieved cannot be reranked, cannot be cited, and cannot be reasoned over. Precision and order matter too (metrics like MRR and NDCG reward putting the right chunk near the top), but a recall miss is unrecoverable downstream.

The trick that turns "RAG is bad" into a fixable bug is measuring recall at each stage of the funnel, not just at the end. Recall can only fall as the funnel narrows, so the stage where it drops is the stage that broke.

Evaluate the answer

Once you know the right evidence was retrieved, you can fairly judge the answer. It has several parts that are worth scoring separately:

• Faithfulness. Every claim in the answer is supported by the retrieved sources, with nothing invented.
• Correctness. The answer actually resolves the user's task.
• Citation quality. Citations point to sources that genuinely support the claim they are attached to.
• Abstention. The system refuses or asks for clarification when the evidence is not enough.
• Usefulness. The answer is clear and specific enough to act on.

Faithfulness and correctness are not the same thing, and the gap between them is where RAG quietly goes wrong. An answer can be faithful but wrong when the retrieved source is itself outdated or mistaken, and the model dutifully repeats it. An answer can be correct but unfaithful when the model reaches the right answer from its training memory rather than the evidence, which looks fine today and breaks the moment the underlying fact changes. For a grounded system you want both, and you have to measure both, because either one alone hides a real failure.

Some of these are scored by humans, some by scripted checks (does every cited ID exist in the retrieved context?), and some by an LLM judge. A judge is attractive because it scales, but it is itself a model that can be wrong, biased toward longer answers, or inconsistent. Treat the judge as something that needs its own eval: label a sample by hand, measure how often the judge agrees, and do not trust its scores on the rest until that agreement is high enough for your stakes. An uncalibrated judge gives you confident numbers that mean nothing. For calibration methods and bias traps, see Evaluation lesson 03.

Named metrics and frameworks (RAGAS, DeepEval)

Hand-rolled checks work, but teams increasingly standardize on libraries that implement the same metric names across projects. RAGAS and DeepEval are two common choices; hosted platforms like Arize Phoenix expose similar scores. The names matter because they separate retrieval quality from generation quality instead of collapsing everything into "the answer looked fine."

• Faithfulness (generation): every claim in the answer is supported by the retrieved context. Use this when hallucination after retrieval is the risk. Low faithfulness with good retrieval usually means prompt or model behavior, not search.
• Answer relevance (generation): the answer addresses the user's question, regardless of whether it stayed grounded. A fluent non-answer can score high on faithfulness if it only repeats context; answer relevance catches "technically supported but useless."
• Context precision (retrieval): how much of the retrieved context is actually useful for the question. High precision means you are not stuffing irrelevant chunks into the prompt.
• Context recall (retrieval): whether the retrieved context contains the information needed to answer. This maps to the recall@k idea from earlier, but scored per question against a reference or judge.

When to use which: tune retrieval with context recall/precision and your labeled expected sources; tune prompting and generation with faithfulness and answer relevance together. A retrieval upgrade that lifts context recall but drops faithfulness may be surfacing noisy chunks the model now has to wade through. Run both sides of the split before you ship.

RAGAS implements these as LLM-assisted or embedding-assisted scorers over traces; DeepEval wraps similar ideas with pytest-style test cases. You do not need a framework on day one, but adopting its vocabulary early makes cross-team debugging easier.

Measure production constraints

RAG systems also fail by being too slow or too expensive. Track retrieval latency, reranker latency, model latency, input tokens, output tokens, total cost per request, cache hit rate, and index freshness. These numbers shape the product as much as answer quality does.

A pipeline that answers perfectly after 18 seconds may be useless for chat support. A cheap pipeline with weak citations may be useless for compliance review. Evaluation has to match the product.

Offline evals and online signals

The labeled set above is an offline eval: a fixed benchmark you run on demand to compare versions before they ship. It is repeatable and lets you change one thing and see the effect, which is exactly what you need to make decisions. Its weakness is that it only contains questions you thought to write down.

Production then shows you the questions you did not. Online signals are the real-world feedback a live system produces: thumbs up and down, whether the user rephrased and asked again, whether they escalated to a human, whether they abandoned the session, and how often the system abstained. None of these is ground truth on its own, but together they point at where reality diverges from your test set. The healthy loop runs both ways: online signals surface new failure cases, you fold the hard ones back into the offline set as labeled examples, and the benchmark grows toward the questions that actually matter.

Build a failure taxonomy

Every bad answer should get a cause label. Useful labels include:

• Right source not indexed.
• Right source indexed but not retrieved.
• Right source retrieved but not selected for context.
• Evidence selected but answer ignored it.
• Answer cited the wrong source.
• Source was stale or conflicting.
• User lacked permission for the needed source.

Those labels tell you what to fix next. Without them, teams tend to randomly change chunk size, swap embedding models, rewrite prompts, and hope.

Shared eval infrastructure

This lesson covers RAG-specific metrics end to end. For the cross-product eval stack—golden set design, Cohen's kappa for judges, CI regression gates, OTel tracing, and the production incident loop—see the Evaluation & Observability course. In particular: L02 task evals, L03 LLM-as-judge calibration, and L05 CI regression.

What good looks like

A healthy RAG system has a trace for every answer, a labeled eval set, retrieval metrics, answer metrics, latency and cost metrics, and a process for reviewing failures. It also has a clear answer to "what should happen when the evidence is missing?"

That is the main lesson of the course. RAG is not just embeddings plus a prompt. It is an evidence pipeline with a language model at the end.

Quick check