Your eval suite is measuring the wrong thing

The score went up and the complaints did not go down

The pattern is consistent enough to be diagnostic. A team stands up an eval suite, four hundred cases, run on every merge. Over two quarters the headline accuracy moves from the high eighties into the low nineties. In the same two quarters, support tickets about wrong answers stay flat, one large customer stops expanding usage, and the solutions engineer keeps apologizing for the same class of output. Nobody is lying. The eval is measuring a real property of the system. It is not measuring the property that decides whether the product gets kept.

The reason is almost always the same. An eval set is built at a moment, out of the failures visible at that moment, using the graders that were easy to write at that moment. It then freezes all three. The product does not freeze. New document types arrive, a retrieval index gets rebuilt, a customer starts sending inputs three times longer than anything in the set, and the definition of a good answer shifts because the person reviewing the output changed jobs. The suite keeps passing because it keeps asking its original question, and the original question has quietly stopped being the important one.

This is not a failure of discipline. It is what happens to any fixed measurement pointed at a moving system. The fix is not more cases. It is measuring a different thing, in a way that stays attached to the outcome.

Six gaps between a score and an outcome

When a suite and reality disagree, the disagreement almost always lives in one of six places. Working through them in order takes an afternoon and usually finds two.

You are measuring the model; the customer experiences the system

A task in a production system passes through a long chain: input normalization, retrieval or lookup, prompt assembly, the model call, output parsing and validation, any tool calls, the retry policy, the fallback, and finally rendering into whatever the user reads. A model-level eval holds eight of those nine constant and varies one. In production, most incidents come from the eight.

Three that we see repeatedly, none of which will move a model-level score by a single point. A document ingest pipeline starts dropping table cells after a library upgrade, so retrieval returns text that is subtly missing the numbers. A stricter output parser starts rejecting a formatting variant the model has always produced occasionally, and the fallback path returns a truncated answer that looks confident. A retry policy silently swallows a rate-limit error and returns the first partial response instead. In all three cases the model is fine, the eval is green, and the user gets something wrong.

The remedy is not to throw out the model-level suite. It is fast, cheap and useful for prompt and model comparisons, and it should stay. The remedy is to add one suite that runs end to end against the deployed path, with the real retrieval index, the real tools, the real parser, and assertions written against the artifact the user actually receives. It will be slower and flakier. It will also be the only thing you own that can fail for the same reason production fails.

A practical rule for what belongs where: if a change to the code between the request and the response could break it, the assertion belongs in the end-to-end suite. If only a change to the prompt or the model could break it, the model-level suite is enough.

The average is hiding the failure that loses the account

An aggregate of ninety-one percent is compatible with a great many underlying shapes. One of them is ninety-eight percent on the common case that makes up four-fifths of the volume and sixty-two percent on a six percent slice that happens to be one customer's entire workload. That customer does not experience ninety-one. They experience sixty-two, and no amount of explaining the average will change what they see.

Slice on the dimensions where behavior genuinely differs, not on everything. In most systems the productive cuts are: tenant or account, input type, input length, language or dialect, source system, and time period. Add one more that teams forget — the newest feature, which has the least representation in a set built before it existed.

Then report a per-slice floor rather than only a mean, and be honest about sample size, because this is where slicing usually goes wrong in the other direction. A slice of thirty cases cannot tell you whether the true rate is eighty-five percent or ninety-five. The interval around a proportion measured on thirty items is wide enough to swallow most regressions you would care about. Distinguishing a five-point difference with any confidence takes cases in the hundreds per slice, not dozens. The exact number depends on the base rate and how much of a difference you want to detect, and it is worth having someone compute it once for your actual numbers rather than guessing.

The practical consequence: gate on slices where you have enough cases, and watch, chart and investigate the ones where you do not. A slice with forty cases going from thirty-eight passes to thirty-four is not evidence of a regression. It is a reason to go collect more cases in that slice.

Your grader has an opinion and nobody has audited it

Every eval contains a grader, and the grader is a model of what correct means. There are four kinds, and they fail differently.

Programmatic checks — schema validity, a numeric match, a required field present. These are cheap, deterministic and never wrong about what they measure, which is usually a narrow slice of what matters.

Reference comparison — similarity to a written-down correct answer. Useful when there is one right answer and dangerous when there are many, because it penalizes a correct answer phrased differently and rewards a wrong answer phrased similarly.

A model acting as judge — now the default, because it scales and it handles open-ended output. It is also the grader that quietly sets your roadmap, and it comes with documented tendencies: preferring longer answers, preferring the answer presented first, preferring confident phrasing, and preferring output that resembles its own. None of those correlate with being right.

A human expert — the ground truth you are approximating, and the one you use sparingly because it is expensive.

The audit is straightforward and almost nobody does it. Take a hundred items your automated grader has scored. Have the domain expert grade the same hundred, blind to the automated result. Measure the agreement, and look at the disagreements individually. That agreement rate is the ceiling on what your eval can tell you. If the grader and the expert agree seventy percent of the time, then a four-point movement in your headline number is inside the noise of the grader itself, and you have been reading it as signal.

Two things this audit reliably surfaces. First, the cases where the grader is systematically wrong in one direction — a rubric item that was written ambiguously, so the judge resolves it one way and the expert the other, on every case. Second, the cases where the expert is inconsistent, which is genuinely valuable information: it means the specification is underdetermined, and no model can hit a target that two qualified people place in different spots. Re-run the audit when the grader model changes, when the rubric changes, and at least a couple of times a year regardless.

Offline and online disagree, and the disagreement is the finding

Production is running an evaluation on your behalf every hour, and most teams do not read it. The cheap online signals are: task completion, escalation to a human, abandonment part way through, retry rate, time to resolution, and the one that is consistently underused — how much a human edits the output before it goes out.

Edit distance is worth building first because it is specific. An accuracy score tells you something is wrong. A pattern showing that reviewers rewrite the first paragraph of ninety percent of outputs, and change almost nothing else, tells you exactly what is wrong and roughly how to fix it. In a review workflow this data already exists; it is the difference between the draft and the version that was approved, and capturing it is a database column and a diff.

Then build the join. Give every production task an identifier and a small set of tags describing its shape. Tag your eval cases the same way. Now you can ask the question that matters: for the tasks where production went badly, what did the offline suite say? When the suite said pass and production said fail, you have found a hole in the set, and that specific case should be in it by the end of the week. This loop — incident becomes case — is the only mechanism we know of that keeps a suite from aging into irrelevance.

Not all errors weigh the same, and your eval says they do

A suite that counts passes and failures has assumed every failure is equally bad. Almost no real workflow works that way. In a document review product, a slightly awkward summary is cosmetic; a missed obligation date is expensive; and citing a clause that does not exist in the document is the kind of error that ends the relationship regardless of the aggregate score.

Write the severity classes down with the business, not with the engineering team alone. Three classes is usually enough: cosmetic, costly, and unrecoverable. Then treat the unrecoverable class separately rather than weighting it inside an average. Give it its own small suite, hand-built, reviewed by a human, with a gate that says zero. Fifty cases you have personally verified, where any failure blocks release, will protect you better than five thousand cases contributing to a weighted mean that can absorb one catastrophe without visibly moving.

This also solves a political problem. It is hard to argue for holding a release over a two-point drop in an average. It is easy to hold a release that produced a fabricated citation in a suite whose whole purpose is fabricated citations.

What to measure instead

A working measurement set for a product team, in the order we would build it:

• Task success on the deployed path, defined by an assertion an operator would sign off on, not by similarity to a reference string
• Per-slice floors on tenant, input type and length, with a stated minimum sample before a slice can gate anything
• A zero-tolerance suite for the unrecoverable error class, small, hand-verified, blocking
• Grader agreement with a human expert, measured on a sample, re-measured whenever the grader or rubric changes
• Human edit distance where a person reviews output before it ships
• Cost and latency per successful task reported next to quality, so nobody optimizes one into the ground for the other
• An incident-to-case pipeline with a service level: every production failure becomes a permanent case within a week

Two of these are worth more than the rest combined: the end-to-end assertion and the incident-to-case pipeline. The first makes the suite able to fail for the right reasons. The second keeps it able to, as the product moves.

The mistakes we get called in to fix

• A single headline number reported to leadership, with no slice underneath it
• Every case graded by a model nobody compared to a human, so the eval's ceiling is unknown
• Mocked retrieval and mocked tools in the only suite that runs on every change
• Gating on a slice of thirty cases, producing alarms and complacency in roughly equal measure
• An eval set that has not gained a case in six months while the product gained four features
• Fabrication and formatting failures scored identically, so the worst error is invisible in the mean
• Test cases that leaked into the prompt as few-shot examples, which is how a suite reaches ninety-nine percent and stops predicting anything
• Production signals nobody joined back to the offline set, so the same defect is discovered twice

Bottom line

An eval suite is a hypothesis about what matters, written down in code, at one moment. It ages the way any hypothesis ages when the world it describes keeps moving. When the number and the customer disagree, believe the customer, and then go find which of the six gaps produced the disagreement — the unit, the distribution, the aggregation, the grader, the severity weighting, or simply time. Fixing it rarely means more cases. It usually means measuring the whole task instead of the model call, reporting the worst slice instead of the mean, and spending one afternoon checking that your grader and your best expert are even talking about the same thing.

Frequently asked questions