Why a single-turn eval stops working
A prompt eval is a function test. One input, one output, one grader. An agent is a program: it plans, calls tools, reads what came back, changes its mind, writes to systems that remember, and stops when it decides it is done. Feeding it a prompt and grading the last message tells you almost nothing about the twelve decisions in between, and the twelve decisions in between are where production failures live. The harness has to change shape, not just grow more cases.
Errors compound multiplicatively. This is the single most important number in agent engineering and it is arithmetic, not opinion. An agent that takes the right action 97 percent of the time will complete a twelve-step task about 69 percent of the time. At 99 percent per step it completes 89 percent of the time. The gap between a per-step accuracy that sounds excellent and a task success rate that is embarrassing is entirely explained by the exponent. If your eval reports per-step metrics, it is reporting the flattering number.
The environment is part of the system under test. A model that is perfect and a search tool that returns stale rows produce a wrong answer, and the transcript will show the model reasoning carefully over bad data. If your harness mocks the tools with clean fixtures, you have tested the model and shipped the system.
Variance is high and unavoidable. Run the same case ten times and you will see different trajectories. Two of them may succeed by luck. A pass rate from a single run is a sample of size one per case, and teams routinely celebrate or panic over movement that is inside their own noise floor.
Success is not binary. An agent can produce the right answer after nine wasted tool calls and forty seconds. It can produce a wrong answer confidently, or the right answer while writing to a record it should not have touched. One boolean cannot carry that.
You are probably here because
- The suite is green and users are still reporting the agent doing something odd
- A pass rate moved four points and nobody can say whether that was real
- Every regression is found by a customer and reproduced by hand from a screenshot
- A full eval run takes six hours and costs more than the feature earns in a week
Those are four symptoms of the same design gap: a harness built to score answers, pointed at a system that produces trajectories.
Four things to measure, and they do not collapse into one
Outcome. Did the world end up in the state the user wanted? This is the metric that matters and the hardest to compute, because it usually requires inspecting the environment after the run rather than reading the final message. If the agent files an expense report, the assertion is against the expense record, not against the agent saying it filed one.
Trajectory. Did it get there sanely? Number of steps, tools called, redundant calls, dead ends, whether it read before it wrote. A run that succeeds in nineteen steps where the reference path is four is a run that will fail as soon as the task gets slightly harder. Trajectory metrics are your leading indicator; outcome is the lagging one.
Cost. Input tokens, output tokens, tool calls, wall-clock time, per case and per suite. Cost is a first-class quality axis for agents because the cheapest way to raise a pass rate is to let the loop run longer, and a harness that does not measure cost will happily reward that.
Blast radius. What did it touch that it did not need to touch? Every write, every external call, every permission exercised. This is the axis with no partial credit: an agent that succeeds at the task and deletes an unrelated record has failed the run.
The fixture problem is the whole engineering job
People underestimate this by roughly an order of magnitude. Writing two hundred test cases takes a week. Building an environment those cases can run against, repeatably, without touching production and without being so clean it hides real failures, takes considerably longer and is where the harness is actually won or lost.
Sandbox the world, do not mock the tools. A mock returns what you told it to return, so your agent never sees the response that actually breaks it. Prefer a real dependency in a disposable state: a seeded database, a fake tenant on the real SaaS API, a local object store. Where a real dependency is impossible, use record-and-replay against captured traffic — cassettes recorded once from a live system and replayed deterministically — rather than a hand-written stub. The captured traffic contains the pagination quirk and the null field your stub will never have.
Freeze time and seed randomness. Any case that reads a clock or a "recent items" endpoint will rot within a month. Pin the clock. Seed every generator. Store the seed in the result record so a failure can be reproduced exactly.
Make setup and teardown cheap. If resetting the world takes ninety seconds, a two-hundred-case suite spends five hours doing nothing else. Snapshot-and-restore at the container or database level, run cases in parallel with isolated tenants, and treat suite wall-clock time as a design constraint from the first day.
Keep the environment versioned with the code. A fixture change that alters a pass rate is a result change, and if the fixtures live outside version control you will spend a day arguing about a number that moved for a reason nobody recorded.
Where the build effort actually goes — our observed split
Share of engineering days on the harnesses we have built or rebuilt. Teams consistently plan this as if the first row were the smallest.
Three ways to grade, and what each one is worth
Use all three. They are not competing options, they are a ladder from cheap-and-narrow to expensive-and-true, and the job of the top two rungs is to keep the bottom one honest.
| Grader | Good for | Cost per case | How it lies to you |
|---|---|---|---|
| Programmatic assertion Check the end state, not the text | Anything with a checkable result: a row written, a file produced, a number matched | Effectively zero after it is written | Passes a run that reached the right state by an unacceptable path |
| Model-judged rubric Scored against written criteria | Open-ended output, tone, explanation quality, whether a refusal was appropriate | Cents, plus real latency at scale | Prefers longer and more confident answers; drifts when the judge model changes |
| Human review Sampled, with a written rubric | Establishing ground truth and calibrating the other two | Minutes of a qualified person | Reviewers disagree with each other more than teams expect |
Push everything you can into assertions. The instinct to reach for a judge is usually a sign the case was written loosely. "Did it summarize the document well" is judge work. "Did it extract these four field values" is an assertion, runs free, and never drifts. Rewriting a third of your judged cases as assertion cases is the highest-return afternoon in this whole exercise.
Treat the judge as a model you also have to evaluate. A judge prompt is a system with an accuracy number, and you do not know that number until you measure it against human labels. Sample a hundred cases, have a person grade them blind, and compute agreement. Typical agreement on a well-written rubric lands somewhere in the high seventies to mid eighties as a percentage, which is useful and is nowhere near ground truth. Publish the number next to every judged metric so nobody mistakes it for one.
Control the known biases. Judges favour longer responses, favour the first option presented in a pairwise comparison, and favour output that matches their own house style. Randomise option order, cap or normalise length, and prefer pairwise comparison over absolute scoring — models are far more reliable at "which of these two is better" than at "score this from one to ten." Version the judge prompt and pin the judge model, because an unannounced judge upgrade will move every historical number you have.
Sample size, or how to stop reporting noise
This is where most agent eval programs quietly fail. Suppose your suite is sixty cases and you pass forty-five, so you report 75 percent. The 95 percent confidence interval on that estimate is roughly 63 to 85 percent. You cannot detect a five-point regression. You cannot detect a ten-point regression with confidence. Every release conversation about a three-point move is a conversation about nothing.
Two fixes, and you want both. First, compare paired. Run the old and new configurations over the same cases with the same seeds and count only the cases that changed direction. A paired test on the disagreements — McNemar's test is the standard one — is dramatically more sensitive than comparing two independent proportions, because it discards the enormous shared variance from cases both versions get right or both get wrong. In practice this turns an undetectable difference into a measurable one at the same suite size.
Second, size the suite for the effect you care about. Decide in advance what movement is worth acting on. If a five-point change should trigger a release hold, you need a suite in the low hundreds of cases, not sixty. If you only care about ten-point moves, a hundred cases is defensible. Write the number down, because otherwise the suite size is set by how many cases somebody had time to write.
Repeat runs to separate model variance from case difficulty. Run each case three to five times. Now you can distinguish a case the agent always fails from a case it passes two times in five, and those need completely different responses. Report both mean pass rate and the stricter all-of-k rate: for anything a user will run once and trust, all-of-k is the honest number and it is always lower than the one on the slide.
What the suite should be made of
Case coverage is a taxonomy problem, and the taxonomy is short enough to hold in your head. Every case belongs in one of these buckets, and a suite missing a bucket is missing an entire failure class.
- Golden path — the task done normally, with clean inputs. Should be your smallest bucket and is usually the largest.
- Ambiguous request — underspecified input where the right move is to ask rather than guess.
- Should refuse or escalate — out of scope, unsafe, or beyond the agent's permissions. Scored on the refusal being correct and legible.
- Tool failure injection — timeouts, 500s, empty results, malformed payloads, a tool that succeeds slowly.
- Hostile input — instructions embedded in documents or tool output, attempting to redirect the agent.
- Long horizon — tasks needing enough steps that context management and summarisation are actually exercised.
- Regression cases — one per production incident, added the day it is diagnosed. This is the bucket that compounds.
The regression bucket deserves its own discipline. Every incident, every "the agent did something weird" ticket, becomes a case before the fix is merged. Two things happen. The suite grows in exactly the direction reality is pushing, and the cases carry a provenance line naming the incident, which makes them impossible to delete casually a year later when someone is trimming runtime.
Store the whole trajectory, not the score
Every run should persist the full ordered record: each model call with the exact prompt sent after templating, each tool call with arguments and response, timings, token counts, and the seed. Storage is cheap and the alternative is debugging from a pass rate. The single highest-value piece of UI in any harness we have built is a side-by-side trajectory diff between two runs of the same case, because it converts "it got worse" into "it stopped calling the lookup tool on step three."
Send us your suite and we will tell you what it cannot detect.
Email your case list, your graders and the last two runs to contact@precisionfederal.com. You get back a short written note: the smallest regression your current suite can actually detect, which failure classes are uncovered, and the three changes we would make first. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comWhat a full run costs, and the tiering that follows
Do the arithmetic before you design the suite, because the arithmetic decides the design. Take three hundred cases, five repetitions each, and an average of twenty-five model calls per run at roughly four thousand input and four hundred output tokens per call. That is 1,500 runs, 37,500 model calls, and on the order of 150 million input tokens and 15 million output tokens per full pass.
At the per-million-token prices commonly quoted for a mid-tier model at the time of writing, that is tens of dollars. At frontier-model prices it is comfortably into the hundreds, and if your judge is also a frontier model you can add half again. Run that on every pull request across a team of eight and the eval bill becomes a line item somebody asks about. It should be a line item — it is cheaper than the incident — but it should be a deliberate one.
The answer is tiering, and three tiers is the right number:
| Tier | Contents | Budget | Gate |
|---|---|---|---|
| Smoke | 20–30 cases, one repetition, assertions only, no judge | Under 5 minutes | Blocks the commit. Must never be flaky |
| Pull request | ~100 cases, 3 repetitions, judge on the subset that needs one | 20–40 minutes | Blocks the merge on a paired regression |
| Nightly / release | Everything, 5 repetitions, full trajectory capture, cost report | Hours, in parallel | Blocks the release; produces the published diff |
The smoke tier has one rule that matters more than its contents: it must not be flaky. A gate that fails randomly once a week gets an override flag within a month, and the override flag never goes away. If a case is inherently variable, it belongs in a higher tier where the statistics can absorb it.
How much a harness change moves real-world reliability — our ranking
Judgment from harnesses we have built, not a benchmark. The ordering is the useful part, and the last row is where most teams spend their time.
Keeping the suite honest over a year
Two decay mechanisms will quietly ruin a good harness, and both are slow enough to miss.
Contamination. If your eval cases appear anywhere in prompt examples, fine-tuning data, or a retrieval index the agent can reach, the suite is now measuring memorisation. Keep a held-out set that never touches any training or prompting artefact, hash every case, and check for those hashes in any dataset you build. When a case does leak, retire it rather than arguing about how much it matters.
Overfitting to the suite. Nine engineers optimising against two hundred cases for six months will produce a system that is very good at those two hundred cases. Counter it by holding back a portion of cases the team cannot see, rotating a fresh slice in each quarter, and keeping a small periodic human review over live production traffic. The gap between your suite pass rate and your production satisfaction signal is the number that tells you the suite has drifted, and it is worth putting on the same chart.
The mistakes we get called in to fix
- Grading the final message when the observable fact is the state of a record the agent wrote
- Mocked tools that always succeed, so the suite has never seen a timeout it will see hourly in production
- A single run per case, then a release argument about a movement inside the noise
- An unversioned judge prompt edited to fix a specific complaint, silently rescoring a year of history
- No cost metric, so a change that raised the pass rate by tripling the step budget looks like a win
- All golden path, no tool failures, no ambiguity, nothing the agent should refuse
- Eval cases pasted into the prompt as few-shot examples, which is the purest form of contamination
- A flaky gate with an override flag that has been on since April
A three-week build that gets you a real harness
Agent Evaluation Build
Day nineteen onward is the part that matters and never appears on a plan: use it. Run it against a change you already believe is neutral and see whether the harness agrees. Run it against a change you already believe is a regression and see whether it catches it. A harness that has never been shown a known-bad build has not been tested, it has been written.
Before you trust the number
- Outcome is asserted against the environment, not read from the agent's own summary
- Every case runs at least three times with recorded seeds
- Every reported rate carries an interval, and comparisons are paired
- Cost and step count are reported next to every quality metric
- Tool failures, ambiguity and hostile input each have their own cases
- The judge prompt and judge model are pinned and versioned with the code
- Judge agreement with human labels is measured and published
- Full trajectories are stored and diffable between runs
- A held-out slice exists that the team optimising cannot read
- Every production incident has become a case
Bottom line
An agent eval harness is not a bigger prompt eval. It is a test environment plus a trajectory recorder plus a statistics layer, and the model is only one of the components under test. Build the sandbox first, because it is the expensive part and everything else depends on it. Assert on end state rather than text wherever you can, because those graders are free forever and never drift. Run cases repeatedly and compare paired, because otherwise you are reading noise. And put cost on the same chart as quality, because an agent will always be able to buy a better score with more steps if you let it.
Frequently asked questions
It depends entirely on the smallest regression you want to catch. Sixty cases cannot reliably detect anything smaller than about a ten-point move; the low hundreds gets you into five-point territory when you compare paired on the same cases and seeds. Decide the detectable effect first and derive the count, rather than shipping whatever number of cases somebody had time to write.
For open-ended output it is the only thing that scales, and it needs to be treated as a measurement instrument with a known error rate. Have a person grade a hundred cases blind, compute agreement, and publish that agreement rate next to every judged metric. Prefer pairwise comparison to absolute scoring, randomise option order, and pin both the judge model and the judge prompt so historical numbers stay comparable.
Real dependencies in a disposable state, wherever it is possible. A hand-written mock returns exactly what you anticipated, which means the harness never sees the malformed payload or the slow-then-empty response that causes real failures. Where a real dependency is impractical, record traffic from a live system once and replay it deterministically, and inject failures deliberately rather than hoping they appear.
Because the steps multiply. Ninety-seven percent per step over twelve steps is a task success rate near sixty-nine percent. Per-step metrics are useful for diagnosis and misleading as a headline, so report end-to-end task success as the primary number and keep the step-level numbers underneath it for debugging.
Hold back a slice the optimising team cannot read, rotate fresh cases in every quarter, hash cases and check they have not leaked into prompts or training data, and track the gap between suite pass rate and a production satisfaction signal on the same chart. When that gap widens, the suite has drifted from reality and the fix is new cases, not a new threshold.