You change the chunk size from 300 to 500. Is the bot better now? You try five questions, it feels fine, you ship it. Two weeks later support tickets rise and nobody knows which change caused it. This is how most RAG projects die — not from a bad model, but from having no way to tell better from different. This lesson is the fix, and it's also the question that ends senior interviews: "how do you know it works?"
Why trying five questions fails
Three reasons, all fatal. You test what you imagined — the five questions you invented, not the messy, ambiguous, out-of-scope things real users type. You can't detect regressions — a change that fixes leave questions may quietly break payroll ones, and you'd never know. You can't locate the failure — "the answer was wrong" doesn't tell you whether retrieval missed or the model ignored good context.
Evaluation solves all three: real questions, run automatically, split by stage.
Two zones, two metric sets
This split is the most useful idea in the lesson. Measure retrieval and generation separately, because end-to-end accuracy tells you it broke but never which half to fix. High recall + low faithfulness → your prompt or model is the problem. Low recall → nothing downstream can save you; go fix chunking, hybrid search and reranking.
Retrieval metrics — did the right text arrive?
recall@k — in what fraction of questions does the chunk that truly answers it appear in the top k? This is the first number to compute, always. If it's 40%, stop tuning prompts.
gold = {"q1": "c17", "q2": "c04", "q3": "c88"} # question -> right chunk
retrieved = {"q1": ["c17", "c05"], "q2": ["c11", "c04"], "q3": ["c50", "c61"]}
hits = sum(1 for q, g in gold.items() if g in retrieved[q])
print(f"recall@2 = {hits / len(gold):.0%}")
print("missed:", [q for q, g in gold.items() if g not in retrieved[q]])Result
recall@2 = 67% missed: ['q3']
Context precision — of the chunks you retrieved, how many were actually relevant? Low precision means you're paying tokens for noise and diluting the model's attention; the fix is reranking, filters or a smaller k. MRR (mean reciprocal rank) adds nuance: was the right chunk first, or fourth? A system where the answer is always ranked 5th needs a reranker, not a bigger index.
Generation metrics — did the answer use it?
Faithfulness / groundedness — is every claim in the answer supported by the retrieved context? Note carefully: this is the answer versus the context, not the answer versus reality. A perfectly faithful answer can still be wrong if the document was wrong — which is why you also measure correctness.
Answer correctness — does it match the expert answer? Citation accuracy — does the cited source actually contain the claim? (Models cite plausibly-looking sources when under-instructed.) And the one people forget: refusal correctness — when the corpus has no answer, does the bot say so? Count both failure directions: false answers and false refusals. A bot that never says "I don't know" isn't confident — it's a liability.
Building the golden set — where teams go wrong
A golden set is a fixed collection of questions with, for each: the expected answer, the chunk(s) that should be retrieved, and the expected behaviour for edge cases. 50–200 questions is plenty to start.
The crucial rule: sample from real traffic, not imagination. Pull questions from support logs, ticket histories, actual chat transcripts (PII-masked). Then deliberately include the awkward categories teams skip:
| Category | Why it must be in the set |
|---|---|
| Straightforward questions | the baseline — should never regress |
| Paraphrased / slangy phrasing | tests the vocabulary gap that kills real systems |
| Questions with no answer in the corpus | tests refusal — the safety-critical behaviour |
| Multi-part questions | tests whether one retrieval was enough |
| Near-duplicate policies | tests whether the bot picks the current/correct one |
Then run it as a regression gate: every change to prompts, chunking, embedding model, top-k or LLM runs the suite, and you compare the table before and after. That is the entire difference between engineering and guessing.
Wait — who grades 200 answers every time I change a prompt?
Not a human, or you'll stop doing it by week two. The standard answer is LLM-as-judge: a model scores each answer against a rubric.
JUDGE = """You are grading a RAG answer for FAITHFULNESS.
Reply with a number 0-1 and one sentence of reasoning.
1.0 = every claim is supported by the context
0.5 = partially supported / some claims unsupported
0.0 = key claims are not in the context
Context:
{context}
Answer:
{answer}"""It scales — but the judge is itself an LLM with biases: it favours longer, more confident answers, shows position bias when comparing pairs, and drifts when the judge model is updated. So: calibrate it against a human-labelled sample (measure agreement before trusting it), pin the judge model version, use precise rubrics with anchored examples, and keep periodic human spot-checks. And never let one model write the golden answers and grade them — that measures nothing at all.
Selection-round radar: "How would you evaluate a RAG system?" is the question that separates senior candidates. Structure: split retrieval vs generation → name recall@k and faithfulness → golden set sampled from real traffic including must-refuse cases → LLM-as-judge with human calibration → regression gate plus production monitoring. Five beats, ninety seconds.
Monitoring after launch
The golden set protects you from regressions; production tells you what you failed to imagine. Track: thumbs-down rate, escalation-to-human rate, refusal rate (a sudden spike usually means the index broke), retrieval score distributions (drifting down = content drifting away from user language), latency and cost per query. Sample twenty real conversations weekly for human review — and feed every reported failure back into the golden set so the same bug can never return silently.
Common mistakes
- Measuring only end-to-end accuracy, so failures can't be located.
- A golden set written by the team instead of sampled from real traffic.
- No must-refuse cases — the most dangerous behaviour goes untested.
- Trusting an LLM judge without ever checking it against human labels.
- Evaluating once before launch and never again.
- Changing three things at once, then not knowing which moved the numbers.
Quick recap
| Metric | Question it answers |
|---|---|
| recall@k | did the right chunk reach the model at all? |
| context precision / MRR | how much noise came with it, and how well was it ranked? |
| Faithfulness | is every claim supported by the retrieved context? |
| Answer correctness | does it match the expert answer? |
| Citation accuracy | does the cited source really contain the claim? |
| Refusal correctness | does it admit ignorance when the corpus is silent? |
| Golden set | real questions + expected answers/sources = the regression gate |
Practice Zone — PYQs from real selection rounds
Six MCQs and two tasks — compute recall@k by hand, then write the five-step plan for auditing an inherited RAG bot.
Faithfulness (groundedness) measures:
Asked in
Context precision vs context recall in RAG evaluation:
Asked in
What is a 'golden set' in RAG evaluation?
Asked in
Why must retrieval and generation be evaluated SEPARATELY?
Asked in
Your golden set shows 92% faithfulness, but users still complain. What's the most likely blind spot?
Asked in
Which is a legitimate use of LLM-as-judge in RAG evaluation?
Asked in
Hands-on tasks:
Given a small labelled set (question → id of the chunk that truly answers it) and what your retriever returned, compute recall@3 and say what a low score points to.
Asked in
gold = {"q1": "c17", "q2": "c04", "q3": "c88", "q4": "c31", "q5": "c02"}
retrieved = {
"q1": ["c17", "c05", "c22"],
"q2": ["c11", "c09", "c04"],
"q3": ["c50", "c61", "c72"],
"q4": ["c31", "c33", "c30"],
"q5": ["c02", "c07", "c19"],
}You inherit an undocumented RAG bot serving 2,000 questions a day. Write the 5-step plan to find out whether it's actually any good — with what you'd measure at each step.
Asked in
FAQ
What is Ragas?
An open-source library that implements exactly these metrics — faithfulness, answer relevancy, context precision and recall — using LLM-as-judge under the hood. Useful, but know the metrics first: tools change, the ideas don't.
How big should the golden set be?
Start with 50 real questions covering the categories in the table above; grow toward 200 as you find failures. Coverage of question types matters far more than raw count — 50 well-chosen beats 500 near-duplicates.
Can I evaluate without labelled data at all?
Partially — faithfulness can be judged from the answer and its retrieved context alone, with no gold answer needed. But recall@k and correctness need labels, and they're the metrics that tell you where to fix. Label 50; it's an afternoon that pays for itself repeatedly.
Next lesson: what breaks once real users arrive — Lesson 10: RAG in Production →