Building an eval set that catches regressions

An eval that never goes red is not measuring anything

There is a specific artifact we find in most teams that have been shipping model features for a year. A file of two hundred test cases, written in a single afternoon early on, scoring somewhere in the low nineties, run occasionally, and green every time. Nobody trusts it enough to block a release on it and nobody wants to say so. The problem is not that the team was lazy. It is that the cases were written by imagining what might go wrong, and imagination produces the failures you already know about. The failures that cost you money are the ones nobody imagined.

A useful eval set has three properties the imagined one lacks. It is sourced from things that actually went wrong. It is large enough and sliced finely enough that a real regression is visible above the noise. And it is calibrated so that scores sit in a range where movement is detectable, which usually means somewhere between sixty and eighty-five percent rather than ninety-six.

That last point is unintuitive and worth stating plainly: an eval set scoring ninety-six percent has almost no room to show you a regression, and the four percent it is failing on is where all the information is. If your set scores in the high nineties, it is too easy. Add the hard cases.

Where the cases come from

Three sources, and the proportions matter more than the total.

Production failures, about half the set. Every complaint, every thumbs-down, every escalation and every silent correction a user made becomes a case, with the input preserved exactly as it arrived and the correct output written down. This is the highest-value source by a wide margin because it is drawn from the real distribution, including the parts of it you did not know existed. Build the pipeline that captures these before you write a single case by hand.

Adversarial construction, about thirty percent. Cases built deliberately to break the system: empty inputs, enormous inputs, ambiguity between two valid readings, contradictory instructions, content in an unexpected language, a document that is a scan of a fax of a table. Include the cases where the correct answer is a refusal or an admission of uncertainty, because those are the ones a system optimized for helpfulness fails silently.

Distribution coverage, about twenty percent. Sampled from real traffic to make sure the common path stays represented. Without it you build a set entirely of edge cases and lose the ability to notice that the ordinary request stopped working.

One discipline holds it together: every case carries the reason it exists. A one-line note saying which incident or which risk produced it. Cases without that line become undeletable, because in a year nobody will know whether it is safe to remove them.

How many cases before the number means anything

This is where most eval discussions go wrong, and the arithmetic is simple enough to settle it. If your set has n cases and you score p, the standard error on that score is the square root of p(1−p)/n. At a score of ninety percent, that gives:

Cases	Standard error at 90%	95% interval	What you can honestly claim
50	4.2 points	±8.3 points	Almost nothing. A five-point move is noise
100	3.0 points	±5.9 points	Large regressions only
200	2.1 points	±4.2 points	A workable minimum for a release gate
400	1.5 points	±2.9 points	Comfortable for whole-set movement
1,000	0.9 points	±1.9 points	Enough to slice and still say something per slice

Two consequences follow. First, the fifty-case set everyone starts with cannot detect anything short of a catastrophe, and teams routinely celebrate or panic over moves inside its noise band. Second, if you intend to report per-slice scores, each slice needs its own sample size. A thousand-case set split across twelve slices gives you eighty cases per slice, which is back in the noise.

There is one large saving available. Because you run the same cases before and after a change, you can compare them pairwise rather than as two independent samples. Only the cases that changed verdict carry information, and a paired test over those is far more sensitive than comparing two aggregate percentages. In practice this means: always diff the per-case results, and look first at the list of cases that flipped from pass to fail. That list is more informative than the score, and it is available at any sample size.

Slice it, or the average will hide the damage

An aggregate score is a weighted average, and weighted averages are excellent at concealing exactly the failures that generate complaints. A change that improves the common case by two points and destroys performance on a segment that is four percent of traffic shows up as an improvement. That segment might be your largest customer.

Pick your slices from the dimensions along which the system plausibly behaves differently, and commit to them before you look at results. Typical ones: input length, document type or source system, language, the customer or tenant, the intent category, and whether the correct answer is a refusal. Then set a floor per slice, not just an overall threshold, and fail the run when any slice drops below its floor even if the total went up.

The refusal slice deserves specific attention because it is the one almost nobody builds. A system that has learned to always produce an answer scores well everywhere except on the inputs where the right response is "the documents do not contain this." If you do not have cases where abstention is the correct answer, you cannot detect the most expensive failure mode in the category.

Graders: what to use for what

Grader	Cost and speed	Use it for	How it fails
Exact or normalized match	Free, instant	Classification, routing, extracted identifiers, dates, amounts	Marks a correct answer wrong over formatting
Programmatic assertion	Free, instant	Schema validity, cited source exists, number appears in the source, required field present	Only checks what you thought to assert
Field-level scoring	Cheap	Structured extraction, where partial credit is real	Hides which field is failing unless reported per field
Model judge with a rubric	Moderate; adds a call per case	Open-ended text where a rubric can be written	Agrees with your reviewers less than you assume; drifts when the judge changes
Human review	Expensive, slow	Calibrating the judge; final sign-off on a major change	Inconsistent between reviewers unless the rubric is written down

Use the cheapest grader that can detect the failure you care about. A great deal of what teams send to a model judge is decidable programmatically: did the answer cite a source that exists, is the extracted date parseable, does the total equal the sum of the line items, is the required disclaimer present. Those checks are free, deterministic, and they never drift.

When you do need a model judge, treat it as a measuring instrument that requires calibration. Score two hundred cases by hand, run the judge over the same cases, and report the agreement rate. Below about eighty percent agreement the judge is telling you about itself rather than about your system. Judges also carry known biases — toward longer answers, toward the first option presented in a comparison — so randomize order and control for length in the rubric.

And pin the judge. When the judge model version changes, every historical score becomes incomparable, which is a quiet way to lose a year of trend data. If you must upgrade it, re-run the full history on the new judge and keep both series.

Two sets, because one set cannot do both jobs

Keep a frozen regression set and a working set, and do not let them merge. The frozen set is versioned, changes only by explicit decision with a note, and is what release gates run against. Its value comes entirely from being stable — a score is comparable across six months only if the set did not change underneath it.

The working set is where new cases land, where you experiment, and where you tune. It grows weekly. Periodically, cases graduate from it into the frozen set, and that promotion is a deliberate act with a changelog entry.

The reason for the split is contamination. Once you have tuned prompts against a set of cases, that set measures how well you fit those cases, not how well the system generalizes. This happens gradually and invisibly: someone looks at three failures, adjusts an instruction, the score rises, and the number quietly stops meaning what it meant. Hold out a portion that nobody looks at during tuning and run it monthly. The gap between the tuned score and the held-out score is the size of the illusion.

What to gate and what to only watch

Not everything should block a deploy. Gate on the things that are unambiguous and cheap to check: the never-fail set, schema validity, no drop below any slice floor, and no drop of more than a defined amount on the frozen set. Watch, without gating, the things that are noisy or judgment-heavy: overall quality scores from a model judge, style and tone, latency distributions, per-slice trends.

The failure mode to design against is a gate that is too tight. A gate firing on noise gets overridden once, then routinely, then removed. A gate set at a threshold that only fires on real damage keeps its authority. Set the threshold from the observed run-to-run variance of your own suite, which you can measure by running it three times against an unchanged system — a step almost nobody takes and one that immediately tells you what your noise floor is.

The part that is actually hard

Everything above is straightforward engineering. What makes eval sets expensive is the labelling, and specifically that the correct answers usually have to come from someone whose time is scarce — the analyst, the underwriter, the clinician, the person who knows what the document actually means. Teams consistently underestimate this by a wide margin. Budget several weeks of a subject expert's attention for a first real set, spread over a couple of months, and design the labelling interface so a session takes twenty minutes rather than a morning.

The second unwelcome fact is that an eval set depreciates. Your product changes, your traffic changes, and cases written for last year's feature quietly stop reflecting anything. Budget maintenance in the same way you budget test maintenance: a recurring slot, an owner, and a rule that a case with no stated reason for existing gets deleted rather than preserved out of caution.

A four-week build for a team starting from nothing

Step four is the one that gets skipped and the one that determines whether anyone believes the suite. Until you know how much your own score moves when nothing has changed, every threshold you set is a guess, and the first false alarm costs the suite its credibility permanently.

The mistakes we are called in to fix

• Fifty hand-written cases being used to justify decisions the sample size cannot support
• A set that scores 96% and therefore cannot show a regression
• Only an aggregate score, so a collapse in one segment reads as an improvement
• No cases where the correct answer is "I do not know"
• A model judge that was never checked against human labels
• A judge model upgraded mid-stream, making a year of history incomparable
• Prompts tuned against the same cases used to report quality, with no held-out set
• Thresholds set by intuition, firing on noise until someone disables the gate
• Cases with no stated reason for existing, so nothing can ever be removed

Before you rely on it

• At least half the cases came from real failures, not imagination
• Every case records why it exists and who labelled it
• Sample size supports the size of the claims being made from it
• Slices are defined in advance, reported separately, and have floors
• Abstention is a scored outcome with its own cases
• Programmatic graders handle everything decidable without a model
• The judge is pinned and its agreement with human labels is published
• Frozen set versioned; working set separate; a held-out portion nobody tunes on
• Every run reports the per-case diff, not only the score
• Thresholds derived from a measured noise floor
• A never-fail set blocks release independently of the aggregate

Bottom line

Build the capture pipeline first, because production failures are worth more than anything you can invent. Make the set hard enough to have room to move. Get to a few hundred cases before you make decisions from the score, and read the per-case diff rather than the average. Slice it, and put a floor under every slice. Grade programmatically wherever you can and calibrate the judge wherever you cannot. Keep the gate loose enough that it retains authority and hard requirements separate so they cannot be averaged away. None of this is difficult. It is just work that nobody schedules, right up until a bad release makes it urgent.

Frequently asked questions