Evaluation infrastructure for production models

The day the system stops moving

There is a specific week that arrives two or three months after launch. Somebody proposes a change: swap the retrieval model, raise the chunk size, move to the newer base model that costs 40 percent less. The room goes quiet, because nobody can say what it would do. The model is in production, roughly working, and no one can tell an improvement from a regression. So the team changes nothing, and quality freezes at whatever it happened to reach on launch day.

That freeze is an infrastructure failure, not a modeling failure. The team has a model and no way to measure it, which makes every change a coin flip somebody has to own personally. Evaluation infrastructure removes the personal risk. It answers one question cheaply and repeatedly: did this change make the system better or worse, by how much, and are we sure.

We build these alongside the model itself, and the eval stack is usually the part that survives longest. Base models get deprecated and frameworks fall out of favor. The dataset, the scorers and the gate keep working across all of it.

Three questions that look like one

Teams say evaluation and mean three systems with different data, latency and owners. Conflating them produces a stack that does one job badly instead of three well.

Offline evaluation. Fixed inputs, known correct answers, run on demand. This is what gates a deploy: fast, repeatable, cheap enough for every pull request. It tells you whether a candidate beats what is running now, and nothing at all about the real world, because the inputs are frozen.

Online evaluation. Real traffic split between variants, measured on outcomes users generate. This is where you learn that the version scoring three points higher offline gets abandoned more often, because it writes longer answers nobody reads. Slow, needs volume, and the only method that measures what you care about.

Production monitoring. No ground truth at all. You watch input distributions, error rates, latency percentiles, refusal rates and tool-call failures. It tells you something changed, not whether the change is bad. Monitoring detects. Evaluation adjudicates.

Layer one: the dataset

Everything else is downstream of this. A scorer applied to a bad dataset returns a confident number that means nothing, and a bad dataset is easy to build by accident.

The gold set is a versioned collection of inputs with agreed correct outputs, sampled from real work rather than invented. Sampling matters more than volume. Pull 500 examples uniformly from a production log and the set is dominated by the easy majority case, so the headline number is pinned by inputs the system already handles. Stratify instead: define the slices that matter to the business and sample enough from each that the slice carries its own number.

For document extraction that means document type, vintage, scan quality and page count. For a support assistant it means intent, customer tier, and whether the answer exists in the corpus at all. That last one is the slice everyone forgets. A retrieval system needs questions with no correct answer, because refusing well is a behavior you have to test for. Without them the metric teaches the system that guessing is free.

Labels need provenance: who labeled each item, when, under which version of the guidance, and whether it was adjudicated after disagreement. Double-label a subset and you get an agreement number, and that number is the ceiling on your metric. If two competent humans agree 82 percent of the time about what a correct answer looks like, a model scoring 95 percent against your labels is fitting your labeling quirks rather than the task.

Version the set like code, next to the scorers or in object storage with a content hash recorded in every run, and make every number you quote name the dataset version it came from. Otherwise you get an argument six months later about whether the score rose because the system improved or because somebody added forty easy examples.

How many examples you actually need

This is where eval work quietly wastes itself. A team runs 50 examples, sees 84 percent against 81 percent, and ships. That comparison carries no information, and the arithmetic is not hard.

For a pass rate on n independent items, the standard error is the square root of p times one minus p, over n. At 200 examples and a true rate near 85 percent that is about 2.5 points, so the 95 percent interval is roughly plus or minus 5. Two systems within 5 points of each other on 200 examples are indistinguishable. At 1,000 examples the interval tightens to about plus or minus 2.2. Resolving a 1-point difference this way takes tens of thousands of labels, which nobody is going to produce.

The way out is a paired design, not more data. Run both candidates on the same items and compare per item. Most items pass for both or fail for both and carry nothing about the difference; only the disagreements do. If 1,000 items produce 60 disagreements, the noise on how those 60 split is about the square root of 60, near 7.7, so a real result needs something like a 38-to-22 split. That is a 1.6-point difference detected on the same items that could not resolve anything close to it unpaired.

Two consequences. Run the baseline in the same job as the candidate, on the same items, with the same scorer version, and never compare against a number recorded last month. Report the disagreement count next to the scores, because it is the honest measure of how much evidence the run holds: a run where the systems disagree on four items has told you nothing, whatever the percentages say.

Layer two: the scorers

A scorer takes an input, an output and a reference, and returns a number plus a reason. The reason is not optional. A scorer that returns 0.71 with no explanation produces a dashboard nobody trusts and nobody debugs.

Pick the cheapest scorer that detects the failure you care about. Teams reach for a language model judge where a regular expression would have been exact, faster and free.

Scorer type	Good for	Cost per 1,000 items	Failure mode
Exact and structural string match, JSON schema, numeric tolerance	Extraction fields, classification labels, tool-call arguments	Seconds, no marginal cost	Marks a correct answer wrong over formatting
Statistical precision, recall, F1, calibration error	Classifiers, retrievers, anything with a ranked list	Seconds	Aggregates away the slice that matters
Programmatic assertions rules written per example or slice	Must-mention facts, forbidden content, citation presence	Seconds to write and run	Brittle when phrasing legitimately varies
Model-graded a language model scoring a rubric	Open-ended answers, tone, faithfulness to a source	Dollars to tens of dollars	Systematic bias that looks like signal
Human trained reviewers, written guidance	Calibrating everything above, adjudicating disputes	Hours of labor, days of wall clock	Drift between reviewers

A working stack uses all five as a pyramid. Structural and statistical scorers run on everything, every time. Assertions cover the failures you have already been burned by. Model grading covers what the cheap scorers cannot see, on a subset. Human review runs on a sample to check that the layers above still tell the truth.

Scorers should be pure functions with versioned code, not notebook cells. Put them in the application repository, unit test them against fixtures with known scores, and record the scorer version in every result row. When a metric moves, the first question is whether the system changed or the measurement did, and a version field answers it in seconds.

Calibrating a model-graded scorer

Grading with a language model is legitimate, and often the only affordable way to score open-ended output. It stops being legitimate the moment you treat the grader as ground truth. A judge is an instrument, and an uncalibrated instrument produces numbers with a decimal point and no meaning.

Calibration is a measurement, and it is cheap. Take 150 to 250 items humans have already labeled, run the judge over them, and compute agreement. Report Cohen's kappa rather than raw agreement, which is inflated whenever one class dominates. Under the Landis and Koch convention, kappa above 0.61 counts as substantial and above 0.81 as almost perfect. Below about 0.4 the judge is not measuring your task, and the fix is the rubric.

• Position bias. In pairwise comparison, run every pair in both orders and count how often the verdict flips. That flip rate is the bias estimate, and it is often large enough to swallow the effect you are measuring.
• Length bias. Longer answers score higher for reasons unrelated to correctness. Regress score against output length on the calibration set; if the slope is meaningful, put a length term in the rubric.
• Self-preference. A judge from the same model family as the system under test is not neutral. Where the stakes are high, grade with a different family.
• Rubric drift. Pin the judge model version and the rubric text together in the run record. A vendor updating the model behind an unversioned alias moves every number overnight with no change in your code.
• Rubric ambiguity. If two humans reading the rubric disagree, the judge will too. Fix the rubric until the humans agree.

Ask for structure, not a bare number. A judge that emits a short reason, then a category, then a score gives you something to audit and something to aggregate. Keep the reasons in the run record and read twenty after every meaningful change. That fifteen-minute habit catches more broken evaluation than any dashboard.

Layer three: the runner

The runner executes a suite and produces a result artifact. Ordinary engineering, and where teams lose weeks by underestimating it.

Concurrency. A 1,000-item suite against a hosted model at four requests per second takes four minutes with the right concurrency and forty without it. Bounded parallelism, retry on specific transient errors, a hard per-item timeout. Record the retries, because a suite that silently retries hides a reliability signal you want.

Caching. Key the cache on the full input, the model identifier, the parameters and the prompt version. Re-running a suite after changing one scorer should not invoke the model at all. That decision separates a suite people run on every branch from one they run once a sprint.

Determinism, and its limits. Temperature zero does not make a hosted model deterministic. Floating-point addition is not associative, batched inference kernels reduce in an order that depends on how requests are grouped, and your request shares a batch with whatever else arrived. Identical inputs can produce different tokens. Do not fight this. Run the identical suite twice against an unchanged system and record the delta. That is your noise floor, and any change smaller than it is not a change.

Cost capture. Every item record carries token counts, wall-clock time, retries and cost. Free at build time, nearly impossible to backfill, and it is what makes the interesting comparison possible: the candidate scores 1.5 points lower at a third of the cost, which is a business decision nobody can have without both numbers.

Artifacts. Every run writes a durable record, not a terminal table. A file in object storage or rows in a database, detailed enough to reconstruct the run six months later without the branch it came from.

Run record field	Why it is there
run_id, timestamp, git SHA, dataset version, scorer version	Reproduce the run, and settle the "system or measurement" argument in seconds
model identifier and every generation parameter	Vendors move models behind unversioned aliases; the pin is the only evidence of what ran
per-item input, output, reference, score, scorer reason	Aggregates are for reporting, items are for debugging, and you always need the items
tokens in and out, latency, retries, cost	Makes the quality-versus-cost tradeoff a measured decision, not an argument
slice tags on every item	Per-slice numbers cannot be computed later if the tags were never carried
baseline run_id being compared against	Every result is a comparison; a score with no baseline is decoration

Layer four: the gate

Infrastructure that reports without blocking gets ignored under deadline pressure. Evaluation is not exempt. If the result is advisory it will be waived on the day it matters most, so make part of it blocking. Split the suite by speed and put each part where it fits.

Write the gate rule as a policy, not a threshold on the headline number. Ours reads roughly: no slice with at least 30 items may drop more than 3 points, no regression item may flip from pass to fail without a waiver naming a person, and the aggregate may not drop. That shape blocks the failure that actually happens, a change improving the average while destroying one segment.

Waivers are part of the design, not a leak in it. A gate with no override gets disabled the first Friday it blocks a release nobody can delay. Give it a documented path: named approver, written reason, expiry date, a record in the run artifact. Then read the waiver log monthly, because a slice waived three times running is a defect wearing a process costume.

The average is where failures hide

Aggregate scores are for reporting to people who do not own the system. Engineers read slices, and so should the gate.

A slice carrying 8 percent of traffic can lose 20 points of accuracy and move the headline number 1.6 points, which reads as noise, while every customer in that slice has a bad week. This is the most common way a change that tested fine produces an escalation.

Define slices from the business rather than the data: customer tier, region, language, document type, channel, first-time versus returning. Set a minimum count per slice, show the interval next to the point estimate, and mark slices under the minimum as unmeasured instead of printing a number from nine items. Slice tags cannot be added retroactively, so carry them from the first run.

The loop that makes the suite get better

A gold set built once and never touched decays into a test of last year's problem. The fix is a loop with one rule: every production failure becomes a permanent test case before the fix ships.

Be strict about the mechanics. A bad answer gets reported. Before anyone debugs it, the exact input goes into the regression suite with the correct output attached, and the suite runs to confirm the item fails. Then the fix. Then the suite again, to confirm the item passes and nothing else broke. A fix without a test gets undone by the next prompt edit and nobody notices.

Feed the loop from support escalations, in-product corrections, sampled traffic the judge flags as weak, and disagreements found in human review. Over a year that produces a suite shaped like your real failure surface rather than what the team imagined at kickoff.

Buy the plumbing, own the meaning

You do not have to write all of this. The split is consistent: buy execution, storage, tracing and display; own the dataset, rubrics, scorers and gate.

On the buy side, MLflow and Weights and Biases handle experiment tracking and artifact storage. OpenTelemetry covers tracing for multi-step systems, and its generative-AI semantic conventions give a vendor-neutral attribute set worth aligning to even under a hosted product. LangSmith, Braintrust and Arize Phoenix handle run storage, comparison views and dataset management. Evidently covers drift monitoring. promptfoo, DeepEval and Ragas ship harnesses and prebuilt metrics, reasonable starting points for retrieval-augmented systems.

None of those products can tell you what correct means for your business, which slices matter, what the rubric should say, or what threshold blocks a release. That judgment is the asset. Keep it in your repository in portable formats. We have moved eval stacks between platforms without losing a test case, because the meaning never lived in the platform.

One constraint to check early: if the inputs carry personal data or regulated records, sending them to a hosted platform is a processing decision with GDPR and HIPAA consequences and needs the same review as any other subprocessor. Plan a redacted variant of the gold set, and verify the redaction does not change what the suite measures.

Mistakes we see most often

• Evaluating on the data the system was tuned against. Prompt engineering against the test set is training on the test set.
• Reporting one number. A single accuracy figure with no interval, no slices and no baseline will not survive a hard question.
• Comparing against a number from last month. The model behind the alias moved and the dataset grew. Re-run the baseline in the same job.
• Treating the judge as ground truth. An uncalibrated model-graded scorer tracks the judge's preferences, and teams optimize straight into them.
• Building the dashboard first. It is the cheapest part and the least valuable, and teams reach for it because it is visible.
• Letting the suite take an hour per pull request. A slow gate gets bypassed within two weeks.

What it takes to build

For a system already in production this is roughly a six-week effort for a small team, front-loaded on the least glamorous part.

Two of those weeks are labeling, the part every plan tries to cut. It is also what decides whether the rest means anything. If the schedule compresses, cut the number of slices before cutting label quality, and cut the dashboard before either.

Bottom line

Evaluation infrastructure is not a quality chore bolted onto a model. It is the mechanism that lets a team keep changing a production system without gambling, and its absence is why so many machine learning projects reach a decent launch and then stop improving. Build the dataset first, from real work. Run baseline and candidate in the same job. Calibrate anything that grades. Measure the noise floor before believing a delta. Read slices, and turn every incident into a permanent test. Then let part of it block a merge, because advisory quality gets skipped.

Teams that do this ship faster, not slower. That is the part people find surprising, and it is the whole argument. When a change can be measured in twenty minutes, a team tries ten changes a month instead of one a quarter, and that compounds well past whatever the model was on launch day.

Frequently asked questions