Your agent answered the customer correctly. Success? Not necessarily — it also called the refund tool twice, searched the same document four times, and took nine steps to do a two-step job. The answer was right; the run was a disaster. That gap is why agent evaluation is a separate discipline from chatbot evaluation, and why "how would you test an agent?" has become a standard senior interview question.
Why agents are harder to evaluate
A chatbot has one output to judge. An agent has a trajectory — a sequence of decisions, calls, observations and recoveries — and several of those trajectories may be legitimately correct while others are wasteful or dangerous.
Worse, agents are non-deterministic: the same input can produce different (valid) paths, so naive assertion-based testing goes flaky immediately. And the failures that matter most — an unnecessary side-effectful call, a silent loop — are invisible in the final answer. Evaluate the journey, not just the destination.
What to measure
| Metric | What it tells you |
|---|---|
| Task success rate | the headline — did it achieve the goal? |
| Tool-selection accuracy | did it pick the right tools with the right arguments? |
| Steps to completion | efficiency — rising steps mean it's struggling |
| Error-recovery rate | when a tool failed, did it correct itself or spiral? |
| Cost & latency per task | what product owners actually ask about |
| Human-intervention rate | whether the autonomy level is set right |
| Unsafe-action rate | side-effectful calls it should never have made |
Success rate is the number you report; the rest explain it. Two of them are underrated: steps-to-completion is an early warning that quality is slipping, and unsafe-action rate is the one that keeps you out of an incident review.
Tracing — you cannot debug what you cannot see
A failed agent run tells you almost nothing from its final message. The bug is a malformed argument at step 3, or a tool that returned an empty list, or two tools ping-ponging. Tracing records the whole trajectory: every prompt, thought, tool call with arguments, result, latency and token count.
Tools: LangSmith, Langfuse, Arize Phoenix, or OpenTelemetry-based setups. Whatever you use, the rule is the same — tracing is not optional for agents, and it should be on from the first prototype, not added after the first production mystery.
Wait — how do I test something non-deterministic that calls live APIs?
Mock the tools. This one decision makes agent testing practical: deterministic results, no live-system side effects, no cost, and — the important part — you can simulate failures on demand.
def test_order_status_happy_path():
tools = MockTools({
"get_order_status": {"8123": {"status": "out for delivery"}},
})
run = agent.run("where is my order 8123?", tools=tools, max_steps=5)
assert run.succeeded
assert "out for delivery" in run.final_answer.lower()
assert tools.calls == [("get_order_status", {"order_id": "8123"})]
assert len(run.steps) <= 3 # efficiency
assert not tools.called("issue_refund") # safety
def test_recovers_from_tool_error():
tools = MockTools({"get_order_status": Error("service unavailable")})
run = agent.run("where is my order 8123?", tools=tools, max_steps=5)
assert not run.crashed
assert run.escalated or "try again" in run.final_answer.lower()Result
2 passed
The second test is the one people skip — and failure handling is where real agents fall apart.
Notice the assertion style: outcome, correct tool with correct arguments, bounded steps, and the absence of side-effectful calls. Those last two are what make it an agent test rather than an output test.
LLM-as-judge for trajectories
Some qualities can't be asserted — was the reasoning sensible? was the order of steps reasonable? For those, a judge model scores the trajectory against a rubric, which scales to hundreds of runs.
Same caveats as everywhere else, with an agent-specific twist: judges favour longer and more confident output, so a naive judge will happily reward a wasteful agent that narrates elaborately. Calibrate against human labels, pin the judge version, use anchored rubrics, and never let one model both produce and grade.
Production monitoring — struggle before failure
Offline tests catch regressions on cases you imagined. Production tells you what you didn't. Track: task success rate, steps per task, human-intervention rate, cost per task, tool error rates, and time-to-completion.
The leading indicators matter most: an agent struggles before it fails — more retries, more steps, more escalations — so alerting on rising steps-per-task buys you days before users start complaining. And feed every real failure back into the test set, so the same bug can never return silently.
Selection-round radar: "How do you test an agent?" expects four beats: mock the tools for determinism → assert outcome AND trajectory (right tools, bounded steps, no unsafe calls) → include failure cases → trace everything and monitor steps/interventions in production. Most candidates give only the first half of beat two.
Common mistakes
- Judging only the final answer — the wasteful or unsafe run passes.
- Testing against live APIs: flaky, expensive, and it fires real side effects.
- No failure-path tests — tool errors, empty results, timeouts.
- Running agents in production without tracing, then guessing.
- Trusting an LLM judge that rewards verbosity.
- Watching only success rate, and missing the rising step count that predicted the drop.
Quick recap
| Concept | One-liner |
|---|---|
| The challenge | many valid paths; evaluate the trajectory, not just the answer |
| Core metrics | success rate, tool accuracy, steps, recovery, cost, interventions, unsafe actions |
| Tracing | every prompt, call, argument and result — mandatory |
| Mocked tools | determinism, no side effects, and simulated failures |
| Assertions | outcome + right tools + bounded steps + no unsafe calls |
| LLM-as-judge | scales trajectory scoring; calibrate, it rewards verbosity |
| Leading indicator | steps-per-task and intervention rate rise before success rate falls |
Practice Zone — PYQs from real selection rounds
Six MCQs and two tasks — write a deterministic agent test, then read a real trace and diagnose every problem in it.
Why is evaluating an agent harder than evaluating a chatbot?
Asked in
Which metrics matter specifically for agents?
Asked in
What does tracing give you that logs of the final answer cannot?
Asked in
How do you build a test set for an agent?
Asked in
In production, which signal most often reveals a degrading agent first?
Asked in
What is 'LLM-as-judge' used for in agent evaluation, and what's its main caveat?
Asked in
Hands-on tasks:
Write a deterministic test for a support agent handling "where is my order 8123?" using mocked tools. What do you assert beyond the final answer?
Asked in
A trace shows: search_docs("refund") → empty → search_docs("refund policy") → empty → search_docs("refunds") → empty → final answer inventing a 30-day policy. Diagnose every problem you can see.
Asked in
FAQ
How do I test when several trajectories are valid?
Assert on properties rather than an exact sequence: the required tool was called with correct arguments, forbidden tools were not, step count stayed within a bound, and the outcome is correct. Reserve exact-sequence assertions for flows that genuinely must be deterministic.
How big should an agent test set be?
Start with 20–30 real tasks covering the main flows plus failure cases (tool down, empty result, out-of-scope request, ambiguous input). Grow it from production incidents — a test set that only contains happy paths gives false confidence.
Should traces be stored in production?
Yes — you cannot investigate an agent incident without them. Treat them as user data: mask PII, control access, and set retention limits, since traces contain conversation content and tool results.
Next lesson: keeping an acting system safe — Lesson 10: Agent Safety & Human-in-the-Loop →