Evaluation harnesses that survive a review

The reviewer is not reading your score

There is a specific kind of reader who shows up after the model works. A contracting officer's representative building a case for acceptance. A security assessor assembling an authorization package. A prime's chief engineer deciding whether to put your component in their bid. A diligence team with two weeks and a mandate to find what the seller did not mention. None of them is reading the score to be impressed by it. They are reading it to find the reason it is wrong, before their signature makes it their problem.

These readers almost never dispute the arithmetic. They ask where the number came from. What was in the test set. Who put it there. Whether the model had seen it. Whether the same command run tomorrow returns the same value. Whether anyone outside the team has ever run it. The score is the output of a machine, and the machine is what is actually under review. Build the machine so it answers those questions without a meeting, and the review gets short. Build only the score, and the review turns into a negotiation about your credibility.

Federal buyers have written this expectation into policy. OMB Memorandum M-25-22, Driving Efficient Acquisition of Artificial Intelligence in Government, issued April 3, 2025, directs that when evaluating proposals agencies "must, to the greatest extent practicable, test proposed solutions to understand the capabilities and limitations of any offered AI system or service." It reaches into the contract itself: contracts "must detail the examination, testing, and validation procedures of the vendor," and where the vendor performs the testing, agencies must "require testing results detailed enough for the testing to be independently verified or reproduced, if practicable." Reproducibility stopped being a research virtue and became a contract term.

A harness is not a benchmark

A benchmark is a dataset and a metric. A harness is the runnable system that turns a model into a defended number: the frozen data, the runner, the configuration, the scoring code, the report writer, and the record of exactly what ran. Benchmarks are usually borrowed. Harnesses are built, and the harness is what a review inspects.

Five parts. A harness missing any one of them will not survive contact with a skeptical reader.

Fixed evaluation sets with provenance. Each item traceable to where it came from, who labeled it, and when the set was closed to changes.

A runner with no manual steps. One command, one configuration file. No notebook cells run in a particular order, no "then I fixed the two obvious mistakes by hand."

Scoring code under version control. The metric is code, not a spreadsheet formula, and it lives beside the data hashes it was written against.

A run manifest. Every version of every moving part, written by the runner at execution time, not typed in afterward from memory.

A report a stranger can read. Someone who was not in the room should open it and tell what was measured, on what, against what, with how much uncertainty.

Rerun it in front of them

The first real test of a harness is whether it produces the same answer twice. Not approximately. The same answer, or a documented reason why it cannot be. Most harnesses fail this on the first attempt, and the failures are boring: an endpoint that silently moved to a new model version, a temperature setting nobody wrote down, a retrieval index rebuilt on Tuesday, a tokenizer that changed with a library upgrade.

Pinning is the discipline of making every input to the number explicit. The third column below is the point of the whole exercise. A gap in the manifest is not read as an oversight. It is read as a place where the number could have been chosen.

Test data is evidence, and evidence has custody

Treat the evaluation set the way a lab treats a sample. Where did each item come from. Who labeled it, under what written instructions. What was the agreement rate between labelers. When was the set frozen, and who has touched it since under what change record. Keep the labeling guide in the repository next to the data, because the guide is the operational definition of the metric. A reviewer who asks "who decided this one was correct" and gets a shrug has learned that everything downstream is soft.

Record disagreements rather than resolving them silently. Items two trained labelers scored differently are the most informative rows in the set, and they put a ceiling on any accuracy claim. A system cannot be measured as more correct than the definition of correct.

There is a second reason to be strict here, and it is commercial. M-25-22 tells agencies to run independent evaluations on data they define, and says that data "should not be accessible to the vendor, and should be as similar as possible to the data used when the system is deployed." Assume a set you will never see. Assume it looks like the real work, not like your demo. The harness that does well in that situation is one built against operationally representative data from the start, which is also what NIST's AI Risk Management Framework asks for in MEASURE 2.3: performance criteria "measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s)."

Contamination is usually bookkeeping, not misconduct

The strongest published evidence on benchmark leakage is the Grade School Math 1000 study, presented at NeurIPS 2024. The authors commissioned a fresh set of problems built to match the well-known GSM8k benchmark on human solve rate, solution steps, and answer magnitude, then evaluated leading open and closed models on both. They observed "accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes," and a positive relationship (Spearman's r² = 0.36) between a model's probability of generating a GSM8k example and the size of its gap. Frontier models showed minimal overfitting. The finding is not that everyone cheats. It is that a public benchmark degrades as a measuring instrument the longer it stays public.

Your own data leaks the same way, through ordinary process. Examples used to write the prompt end up in the test set. The few-shot block is drawn from the same pool as the evaluation items. A fine-tuning run reads a directory that also feeds the harness. None of that is dishonest. All of it inflates the number, and all of it is visible to a reviewer who asks for the lineage of ten random items.

The defense is structural: a holdout drawn after the model was frozen, from a source the build process never reads, with an explicit overlap check in the report. If that set does not exist, say so plainly and describe what you did instead. Where a genuinely independent set comes from is covered in testing on someone else's instrument.

The prompt is part of the instrument

At ICLR 2024, Melanie Sclar, Yejin Choi, Yulia Tsvetkov and Alane Suhr measured how much score moves from formatting changes that preserve meaning: separators, casing, spacing, the small choices nobody documents. Across widely used open models in few-shot settings they reported performance differences of up to 76 accuracy points on LLaMA-2-13B from formatting alone. Their recommendation is the practical one: report "a range of performance across plausible prompt formats, instead of the currently-standard practice of reporting performance on a single format."

Two things follow. The prompt template is a versioned artifact, changed under review like any other code, because changing it invalidates every prior number. And if the schedule allows, run a small format sweep and publish the spread. A reported range of 84% to 88% across eight plausible formats is a stronger claim than a bare 88%, and it preempts the reviewer's best move: reformat your prompt slightly and watch the score fall.

A number without an interval is not a measurement

Accuracy on a finite test set is an estimate. How good an estimate depends almost entirely on how many items were in the set, and the arithmetic surprises people who have not done it. The table gives 95% intervals computed with the standard Wilson score method, including two small-sample cases of the kind that turn up in pilots.

Read the first row again. A pilot that gets 19 of 20 right is consistent with true performance anywhere from 76% to 99%. That is not a bad pilot. It is a pilot that cannot distinguish a system fit for production from one that fails one case in four. Reporting it as "95% accurate" is the most common way a technically honest team loses a reviewer's trust, because the reviewer knows the interval and now knows you either did not, or did not mention it.

Sample size is the lever, and the curve flattens fast. Going from 100 to 500 items cuts the interval by more than half; going from 500 to 1,000 buys another point and a half. That shape turns evaluation-set budgeting into a planning problem with an answer, and it is why the target set size belongs in the statement of work next to the threshold, as covered in our note on acceptance criteria for a machine learning deliverable.

Comparisons need the paired test

Most real questions are comparative. Is the new version better than the old one. Is our system better than the incumbent. Overall accuracies do not answer that, because two systems evaluated on the same items make correlated errors, and the correlation is the whole story.

Run both over the identical set, then look only at the items where they disagree. Those discordant pairs carry all the evidence. If 60 items split 40 to 20 in your favor, McNemar's exact test returns p ≈ 0.014 and the difference is real. If the same 60 split 35 to 25, the test returns p ≈ 0.25 and you have measured nothing, even though the headline accuracies moved. Publishing the discordant counts signals that you know which comparisons your data can support.

Judges are instruments, and instruments drift

Using a model to grade model output is routine now, and it is defensible when the judge is treated as an instrument with known error. The foundational study is the MT-Bench and Chatbot Arena work presented at NeurIPS 2023, which reported that strong model judges "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans" — while naming the failure modes directly: position, verbosity, and self-enhancement biases, along with limited reasoning ability.

Later work sharpened the self-preference finding in a way that matters for harness design. A quantitative study of self-preference bias found that model judges "assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated." The bias runs toward text that looks familiar to the judge, not toward its own output specifically, which means switching to a third-party judge that did not generate the candidates does not make the problem go away.

The controls are cheap. Present candidate pairs in both orders and report the flip rate, since a high flip rate means position is doing the grading. Hold out a sample scored by qualified humans and publish judge-to-human agreement beside the headline number. Say plainly when the judge shares a family with the system under test, and pin the judge's prompt, version, and decoding settings like everything else. NIST's framework points the same direction in MEASURE 1.3, which asks that "internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates."

Averages hide the risk; slices carry it

A single aggregate is the wrong shape for a decision. The reviewer's question is not how the system does on average but how it does where the consequences are: the rare class, the degraded input, the unusual document format, the population the training data underrepresented. Define the slices before running anything, set a floor for each, and report every slice whether or not it cleared. One slice below its floor with a named remediation beats a clean average, because it proves the harness can detect a problem.

A harness that only runs on one laptop is a screenshot

Evaluation is not an event before delivery. It is a scheduled job that keeps running while the system is in use, and the federal expectation now says so. M-25-22 requires contract terms giving the agency the ability to "regularly monitor and evaluate (e.g., on a quarterly or biannual basis, based on the needs of the program) performance, risks, and effectiveness" of the system. It encourages agencies to require vendors "to meet performance standards before deploying a new version of an AI system or service or to roll-back to a previous version if a new version fails to meet performance standards." And it states that "vendors must provide the access and time necessary for agencies to complete independent evaluation."

None of that is workable if the harness lives in one engineer's environment. It has to run in continuous integration, on a fixed schedule and on every model or index change, writing its manifest and report to a durable location. This is old security engineering practice applied to a new artifact. Control SA-11 in NIST SP 800-53 Revision 5 already requires the developer to "develop and implement a plan for ongoing security and privacy control assessments," to perform unit, integration, system, or regression testing at an organization-defined frequency, depth and coverage, and to "produce evidence of the execution of the assessment plan and the results of the testing and evaluation." A model evaluation suite is regression testing. Wire it up the same way.

What the reviewer is actually holding

It helps to know the documents on the other side of the table. OMB Memorandum M-25-21, issued the same day as M-25-22, defines high-impact AI as "AI with an output that serves as a principal basis for decisions or actions with legal, material, binding, or significant effect on" six named categories, running from civil rights, civil liberties and privacy through access to programs and to critical government services, human health and safety, critical infrastructure, and strategic assets. For those uses it sets minimum risk management practices including pre-deployment testing, an AI impact assessment, and ongoing monitoring, and states that "if a particular high-impact use case is not compliant with the minimum practices then the agency must safely discontinue use of the AI functionality." Its 365-day clock from April 3, 2025 has already run, so these are live obligations rather than plans.

One clause is worth reading closely as a vendor. On pre-deployment testing, M-25-21 says that if an agency "does not have access to the underlying AI source code, models, or data, the agency must use alternative test methodologies, such as querying the AI service and observing the outputs or providing evaluation data to the vendor and obtaining results." That describes two requests headed your way: black-box access for their own evaluation, or their data through your harness with results returned. A harness built as a service that takes an arbitrary dataset and emits a manifest and report handles both without a scramble.

The standards layer underneath is still moving, and saying so is more useful than pretending otherwise. The released version of NIST's framework is AI RMF 1.0, published January 26, 2023, which NIST states "is being revised as part of the White House AI Action Plan"; its companion Generative AI Profile, NIST AI 600-1, was published July 26, 2024. NIST's Center for AI Standards and Innovation, reoriented under a June 2025 Commerce announcement, now presents itself as "industry's primary point of contact within the U.S. government to facilitate testing and collaborative research" on commercial AI systems. On the defense side, developmental test and evaluation sits with DTE&A under the Office of the Under Secretary of War for Research and Engineering, which carries test and evaluation of AI-enabled systems as a standing focus area. Tooling moves too: Stanford's HELM repository now carries the notice that "HELM entered maintenance mode on June 1, 2026," and the maintainers of EleutherAI's widely used harness describe comparing runs across different papers as a practice they discourage.

The lesson from that churn is not to wait for the standards to settle. It is to make sure the durable artifact is yours: your frozen sets, your manifests, your report format, your scoring code. Third-party runners are useful, and any of them can go quiet. A harness whose evidence survives the retirement of the framework it was built on is the one that is still usable in the review two years from now.

The bundle to hand over

When the request arrives, the difference between a good week and a bad month is whether this already exists as one directory.

• The evaluation sets, with hashes, freeze dates, and the labeling guide
• The run manifest for the reported result, written by the runner at execution time
• The exact command and configuration that reproduces it
• Point estimates with intervals, and the sample size behind every threshold
• Per-slice results, including the slices that did not clear their floor
• Judge calibration: human-agreement sample, order-flip rate, judge version
• The contamination check and what it covered
• The change log for the harness itself, separate from the product change log

Where these reviews go wrong

Bottom line

An evaluation harness is production software with a compliance job, and it deserves the same care as the model it measures. Pin every input. Give the test data custody. Assume the public benchmark has leaked and the private holdout is the one that counts. Put an interval on every number and a sample size behind every threshold. Compare on paired items. Calibrate the judge. Run it on a schedule, not before a meeting. Do that, and the review is a demonstration instead of a defense — the only version of that meeting that ends with a signature.

Frequently asked questions