A team I know shipped a summarisation feature after everyone agreed the demo looked great. Three weeks later they discovered it was silently dropping the last section of any document over a certain length. Nobody had checked, because "it looked great" had felt like evidence.
Evaluating an LLM is harder than evaluating a classifier — there is rarely one right answer — but "hard" is not "impossible", and the teams that do it well have a specific, learnable process. This lesson is that process, and it applies whether you fine-tuned anything or not.
Why public benchmarks won't do
MMLU, HumanEval, GSM8K and the rest are useful for one thing: coarse screening, narrowing ten candidate models to three. They are close to useless for deciding whether your product works, for three reasons.
- They measure someone else's task. A two-point MMLU gap tells you nothing about your invoice-extraction accuracy.
- Contamination. Benchmark questions leak into training corpora over time, so scores drift upward for reasons unrelated to capability.
- They are aggregates. They hide exactly the category-level behaviour you care about.
The only evaluation that predicts your product's behaviour is one built from your product's inputs.
Your evaluation set
Two properties do all the work: drawn from real traffic, and frozen, so a score change means the system changed rather than the test.
| Portion | Share | Why |
|---|---|---|
| Typical traffic | ~60% | sampled to match the real category mix, not an even split |
| Edge cases | ~25% | ambiguous, empty, wrong language, two intents, hostile input |
| Past failures | ~15% | every reported bug, added the day it was reported |
Fifty cases is enough to start and infinitely better than none; 150–300 makes small differences readable. Grow it only by appending — editing a case until the model passes it means testing nothing at all.
💡 Cheapest way to build one: take a week of real traffic, sample 100 conversations, label them by hand. A dull afternoon, and the highest-return afternoon in the project.
Three levels of scoring
Level 1 — deterministic. Free, instant, and covers more than people expect. Run these first; a failure here needs no further judgement.
def check(case, output):
failures = []
if not parses_as_json(output): failures.append("not json")
if missing_required_fields(output): failures.append("missing fields")
if word_count(output) > 120: failures.append("too long")
if case.expected_label != label(output): failures.append("wrong label")
if entities_not_in(output, case.facts): failures.append("hallucinated fact")
if FORBIDDEN_RE.search(output): failures.append("forbidden phrase")
return failuresLevel 2 — reference comparison. Exact match works for labels. For free text it is nearly meaningless — many excellent summaries share no wording with your reference — so semantic similarity is at best a rough signal here.
Level 3 — LLM-as-judge. For open-ended quality, where the first two run out.
LLM-as-judge
You are evaluating a customer-support reply. You are not writing
one and not being helpful to the customer.
<ticket>{{ticket}}</ticket>
<facts>{{order_json}}</facts>
<reply>{{reply}}</reply>
Score each 1-5 with a one-line justification that quotes the reply:
1. GROUNDING - every factual claim appears in <facts>.
Score 1 if any figure, date or id is invented.
2. RESOLUTION - does it answer what <ticket> asked?
3. TONE - appropriate to the customer's evident frustration.
4. ACTION - ends with one concrete next step.
Return JSON only. Set "blocking_issue" when grounding is 1.Three design choices there, each deliberate. Named criteria rather than one "quality" score, because you can't act on a 3/5 overall. Justifications that quote the text, which reduces sloppy grading. A blocking flag, so the worst failure isn't averaged away by three good scores.
And the caveat that is always the follow-up question: the judge is a model, so it needs evaluating too. Have humans label 100 outputs, run the judge on the same 100, measure agreement. If it disagrees with your humans, the rubric is wrong — fix it before believing any of the numbers. Re-check when the judge model version changes. Known biases: judges favour longer answers, and favour outputs from their own model family.
💡 For subjective quality, pairwise preference beats ratings — for humans and for judges. People are far more consistent at "which of these two is better?" than at "rate this 1–5".
Metrics for retrieval-backed systems
If your system retrieves, three metrics separate three different failure points — which is the whole reason to have three:
| Metric | Question | Failure it isolates |
|---|---|---|
| Context relevance | did retrieval return the right material? | retrieval / chunking / index |
| Groundedness (faithfulness) | is every claim supported by that material? | the generation step inventing things |
| Answer relevance | does the answer address the question? | prompt or task framing |
A single "quality" score tells you the answer was bad. These three tell you which stage to fix. The RAG course goes deeper.
Wait — what about everything I didn't optimise?
This is the section most teams skip and most incidents come from. Your task eval set measures the thing you were improving. It says nothing about what you might have broken.
Build a small general-capability regression set — five to ten probes each across about ten categories — and run it before and after every model or prompt change:
- Ordinary conversation ("hi", "thanks")
- Off-domain questions
- Instruction following ("in exactly 3 bullets")
- Refusals — requests it should decline
- Other languages your traffic uses
- Multi-turn coherence with a pronoun reference
- Arithmetic and dates
- Uncertainty — does it still say "not stated"?
- Degenerate inputs — empty, one word, wrong format
- Injected instructions inside user-supplied content
Categories 4 and 8 are blocking regardless of task gains. A model that stopped refusing is a safety regression; a summariser that stopped saying "not in the document" has learned to hallucinate confidently, which is worse than the problem you were solving.
🎯 Selection-round radar: "How would you evaluate an LLM application?" is the design question that separates seniors. A frozen set of real inputs with edge cases and past failures → deterministic checks first, LLM-as-judge with a written rubric calibrated against human labels for the rest → per-category reporting, never one aggregate → a general-capability regression set alongside the task set → and online behavioural signals after launch. Mention re-running on every model version change and you are ahead of the field.
Online evaluation
Offline evaluation gates the deploy. Online evaluation tells you whether real people got what they wanted — and it is usually more honest.
- Edit rate — what fraction of drafts are sent unchanged, and how much gets changed. For any assistant-drafts-something product, this is the metric.
- Escalation rate — how often a human has to take over.
- Retry and rephrase rate — the user asking again is a failure signal nobody has to volunteer.
- Copy rate, abandonment, task completion.
- Explicit feedback — thumbs. Useful, but sparse and biased toward extremes, so never the only signal.
Behavioural signals are collected from everyone, for free, and they measure usefulness rather than politeness.
Common mistakes
- Choosing a model on public benchmark scores.
- An eval set authored from imagination rather than sampled from traffic.
- Editing a test case until the model passes it.
- Trusting an uncalibrated LLM judge.
- Reporting one aggregate that hides a category collapsing.
- Reading a 3% move on 40 cases as a real improvement — that is about one case.
- No general-capability regression set.
- Never re-running evals when the provider ships a new model version.
Quick recap
| Concept | One-liner |
|---|---|
| Public benchmarks | coarse screening only — contaminated, aggregate, someone else's task |
| Eval set | real inputs, frozen, ~60/25/15 typical / edge / past failures |
| Scoring | deterministic → reference → LLM-as-judge, in that order |
| Judge caveat | calibrate against human labels; prefer pairwise over ratings |
| RAG metrics | context relevance, groundedness, answer relevance — three stages |
| Regression set | ten probe categories; refusals and uncertainty are blocking |
| Online | edit rate, escalation, retries — behaviour beats thumbs |
Practice Zone — PYQs from real selection rounds
Six MCQs, then two design tasks: a full evaluation for a fine-tuned support model, and a compact regression set.
Why are public benchmarks (MMLU, HumanEval and similar) a poor basis for choosing a model for your product?
Asked in
In a RAG or generation system, what does 'groundedness' (faithfulness) measure?
Asked in
What must you do before trusting an LLM-as-judge score?
Asked in
Your fine-tuned model improves on the target task. What else must you measure?
Asked in
What is an offline versus an online evaluation?
Asked in
Why is a single aggregate quality score dangerous?
Asked in
Hands-on tasks:
You fine-tuned a model to write first-draft support replies. Design the full evaluation: what you measure, how, and the bar for shipping.
Asked in
Your team only tests the target task. Write a compact regression set — around 10 probe categories — that would catch catastrophic forgetting in a model fine-tuned for legal-document summarisation.
Asked in
FAQ
Which evaluation framework should I use?
Start with a CSV and a script that loops through it and prints a table — it removes every excuse not to begin. Dedicated evaluation and tracing tools earn their place once you have several prompts, want history and dashboards, or need to share results across a team. The framework is never the hard part; the eval set is.
How do I evaluate something genuinely subjective, like brand voice?
Blind pairwise preference with a small human panel: show two outputs, ask which sounds more like us, and count. Then, if you need scale, build an LLM judge and calibrate it against those human verdicts. What you should not do is invent a 1–5 "voice score" and trust it unvalidated.
How often should evaluations run?
On every prompt, retrieval or model change — and on a schedule regardless, because the provider can change the model underneath you. Daily is a reasonable default for the cheap deterministic checks, weekly for the full suite.
Next lesson: getting the model in front of users — memory, batching and the latency numbers that matter — Lesson 8: Serving & Deployment →