The instrument you wrote is calibrated to you
When one team writes both the system and the test cases, the two artifacts share an author, and everything an author knows leaks into both. The same abbreviations. The same section headings. The same idea of which phrasings are common and which are exotic. The same instinct about what a hard case looks like. A high score on that test set is a real result, but it is a narrower result than it feels: the system reliably recognizes the way this team writes the concept down. Whether it recognizes the concept is a separate measurement, and an in-house set cannot make it.
This is not an accusation of sloppiness. It is a statement about what a given measurement can resolve. A ruler cut from the same stick you are measuring will always report a whole number. The reason blind evaluation exists in mature measurement fields is not that practitioners there are more honest than software engineers; it is that they gave up on self-graded instruments a long time ago and built the plumbing to avoid them.
Contamination is usually ordinary, not careless
The word "contamination" suggests a lapse. Most of it is not. There are three routes into a development set that a diligent team walks down without noticing.
Shared authorship. The engineer who wrote the matching rules also wrote the gold labels. Every judgment call about what counts as a hit was made once, by one mind, and applied to both sides of the comparison.
Shared corpus. The evaluation examples were drawn from the same source documents the system was built against, so the surface forms in the test set are the surface forms the system was tuned for.
Shared iteration. This is the quiet one. Every time a result is used to decide what to try next, information moves from the held-out data into the system. Dwork, Feldman, Hardt, Pitassi, Reingold and Roth described this precisely in The reusable holdout: preserving validity in adaptive data analysis (Science, 2015): data analysis is inherently adaptive, and a holdout set queried repeatedly under adaptive choices will be overfitted like any other. The leak does not travel through the data pipeline. It travels through the person.
Well-resourced teams with an explicit decontamination step still get caught. In the GPT-3 paper (Brown et al., Language Models are Few-Shot Learners, 2020), the authors describe searching for and removing overlaps between the training corpus and the evaluation sets, and then report plainly that a bug in the filtering caused them to ignore some overlaps, and that retraining was not feasible given the cost. They named it in the paper. That is what careful work looks like when it fails — not silence, a footnote.
Reading your failures is a one-way door
Here is the moment that decides whether a benchmark is still a benchmark. An engineer opens the failure log and reads the cases the system missed. From that instant, that set is a development set, permanently. Not because anyone cheated. Because the information is now in the head of the person who will write the next fix, and no procedure removes it.
The asymmetry worth internalizing is that you do not have to change any code for this to happen. Looking is enough. A team can read the failures, decide to change nothing, and the set is still spent — because the next architectural decision, the next threshold, the next lexicon entry, will be made by someone who has seen those items.
The practical consequence is a budgeting change. Held-out sets are consumable material, not fixed infrastructure. A team that plans to iterate five times needs more than one instrument, and needs to decide in advance which one is being spent on this round.
What each kind of set can actually tell you
| Kind of set | What it measures | What it cannot tell you |
|---|---|---|
| Development set | Whether the last change helped, on the cases you already understand. | Anything about held-out performance. It stopped being evidence the first time you tuned against it. |
| In-house held-out set, never read | Whether the system generalizes across items your team wrote but did not tune on. A real and useful number. | Whether the system depends on your team's vocabulary, formatting, or notion of a hard case. |
| Published instrument, written by others | Whether the system reaches the concept when somebody else chose the words for it. | Whether it holds under operational data, which no public instrument represents exactly. |
| Sequestered third-party evaluation | Performance on data the builder has never seen and cannot inspect afterwards. | Why it failed. Sequestration buys validity and costs you diagnostic access. |
Freeze first, then go looking
The procedure is unglamorous and the order is the entire point. If the instrument is retrieved before the model is frozen, no blind test has occurred — what has occurred is a slightly more expensive development iteration with better paperwork.
Order of operations for a blind run
Where blind instruments come from
Four established patterns, each solving the authorship problem a different way.
Published measures in the domain. Clinical, psychometric and education research have decades of instruments with documented development, published item wording, and citation trails. The point is not that they are better written than yours. The point is that somebody else chose the words.
Sequestered data. NIST's face recognition evaluations — the long-running Face Recognition Vendor Test, continued as the Face Recognition Technology Evaluation — have developers submit algorithms as black boxes, which NIST then runs against operational imagery held at NIST. The developer never sees the data and cannot tune against it.
Judgments produced after submission. In NIST's Text REtrieval Conference, running since 1992, participants submit ranked results first; the relevance judgments are then produced by NIST assessors from the pooled submissions. You cannot overfit to judgments that do not exist when you submit.
Blindness by timing. The Critical Assessment of Structure Prediction experiment, running since 1994, distributes sequences whose experimental structures have been determined but not yet released, and closes submissions before the structures become public. The instrument is separated from the predictor in time rather than in space.
Underneath all four is one idea: separate the authorship of the measurement from the authorship of the system, in space or in time, and enforce the separation with something other than good intentions.
A large drop is a coverage measurement, not a verdict
The most useful public data on what happens when you swap the instrument comes from Recht, Roelofs, Schmidt and Shankar, Do ImageNet Classifiers Generalize to ImageNet? (arXiv:1902.10811, ICML 2019). They built new test sets for CIFAR-10 and ImageNet by following the original dataset creation processes as closely as they could, then evaluated a broad range of existing models. Their reported result: accuracy drops of 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet.
Two things about that finding matter more than the headline. First, the concept did not change and the construction protocol did not change — only the specific items did, and the number still moved substantially. Second, the authors report that the drops were not caused by adaptivity but by models failing on slightly harder images, and that accuracy gains on the original sets translated into larger gains on the new ones. In other words, the ranking largely survived even though the absolute numbers did not.
That is the correct way to read your own blind drop. It is usually telling you about coverage of the space of surface forms, not about whether the system has understood anything. Treating it as a verdict on comprehension throws away the diagnosis.
Decomposing a 29-point gap
Suppose an extraction system scores 91% on 400 items its own team wrote, and 62% on a 150-item instrument published by somebody else. The 29-point gap is tempting to read as "it understands a third less than we thought." Decompose it instead. Say the published instrument uses 40 distinct surface forms for the events in question and the system's patterns cover 26 of them — then a portion of the misses are unreachable for lexical reasons that have nothing to do with the concept. The remainder are items the system had the vocabulary for and still got wrong.
Those two failure classes have different price tags. Lexical coverage is bounded, cheap, and re-testable. Concept failure is expensive and is not fixed by adding words. Until the gap is split, 62% is not an actionable number — it is a mood. Every figure in this box is fabricated for the arithmetic and describes no system.
Three diagnoses for the same gap
Lexical coverage. The instrument names the same thing differently. The evidence is that failures cluster on wording rather than on category, and that adding the missing surface forms recovers items without any change to the logic. This is the most common diagnosis and the least interesting one, which is exactly why it must be excluded first.
Distribution shift. The instrument's items are drawn from a different population — longer documents, different authors, different base rates for the categories. Failures cluster by document type rather than by phrase. No amount of lexicon work fixes this; it needs representative data.
Concept failure. The system had the words, saw the structure, and still produced the wrong answer. This is the diagnosis that justifies rework, and it is the one teams reach for first and confirm last. The published NLP literature contains a sharp warning here: Gururangan and colleagues showed in Annotation Artifacts in Natural Language Inference Data (NAACL 2018) that a plain text classifier reading only the hypothesis — never seeing the premise it was supposed to reason about — classified roughly 67% of SNLI correctly. A system can post a strong number on a task it is not performing.
Reporting both numbers honestly
The reporting discipline is short, and skipping any line of it costs the reader the ability to interpret the rest.
- Put the internal number and the blind number in the same table — never in different documents, never in different sections.
- Name who wrote each instrument, and when — provenance is what makes the pair interpretable.
- State the freeze point — the commit or artifact hash the numbers describe.
- State how many times the blind instrument has been run against this system. If the answer is more than once, that is material and it belongs in the text.
- Report denominators, not just percentages — 62% of 150 items and 62% of 5,000 items are different claims.
- Name the items the instrument cannot reach at all — categories out of scope, formats unsupported — rather than scoring them silently.
A reader who is handed two numbers with different provenance learns more from the pair than from either alone, and considerably more than from a single flattering figure. There is also a plain credibility effect: an unsourced high number reads as marketing, while two numbers where one of them is worse read as measurement. The same logic applies to precision and recall — a number reported alone can usually be bought by changing the system's willingness to answer.
The federal version of the same argument
Two places in federal practice already encode this idea, in different vocabularies.
Operational test and evaluation for major defense acquisition programs is statutorily separated from the program that built the system. Under 10 U.S.C. § 139, the Director of Operational Test and Evaluation is appointed from civilian life by the President with the advice and consent of the Senate, is charged with monitoring and reviewing all operational test and evaluation in the Department, and may communicate views directly to the Secretary of Defense and the Deputy Secretary of Defense without obtaining the approval or concurrence of any other official in the Department. The same section bars the Director from being assigned any responsibility for developmental test and evaluation — the testing the acquiring program itself runs — other than advising the officials who run it. The separation is written in from both ends. That structure does not exist because developers are assumed dishonest. It exists because developer-run testing measures a genuinely different thing, and both measurements are needed.
The NIST AI Risk Management Framework (AI RMF 1.0) says a version of the same thing for AI systems. Subcategory MEASURE 1.3 asks that internal experts who did not serve as front-line developers for the system, and/or independent assessors, be involved in regular assessments and updates. The framework is voluntary, but the expectation it encodes is increasingly what a government reader looks for: the author of the system should not be the sole author of the measurement.
How we work
We build extraction, retrieval and document-understanding systems intended for federal deployment. Rather than assert a testing philosophy, here is the one artifact we can point at: a published benchmark of our own in which the comparison went against our system — a frontier model out-recalled our deterministic extractor 97.9% to 68.2%, and our weakest category came in at 56.6%. The losing numbers and the per-category breakdown are in the document, because a benchmark a firm only publishes when it wins is not a benchmark.
That example also shows the limit of what we can currently claim, and the table above is the reason. Part of that corpus was written and labeled before the engine was ever run against it, and the engine was not touched afterwards — a genuine freeze, and a real held-out number. But we wrote those items, so it is an in-house held-out set, not an instrument somebody else authored. It answers the second row of the table, not the third.
For most of the problem classes we work in, no published instrument exists — common in federal work — and the fallback is not to skip the step. It is to separate authorship inside the effort: someone outside the build team writes items from source material the build team has not seen, working from a written rubric agreed before the items exist. That is weaker than a genuinely external instrument, and where we use it we intend to label it as what it is rather than let it carry the word "blind" unqualified.
Common objections
Our internal set is genuinely hard — we wrote adversarial cases on purpose.
Adversarial-for-you is still your idea of adversarial. Hand-written hard cases are enormously valuable for finding known failure modes, and they are structurally unable to find the failure modes nobody on the team has imagined. That is the specific gap a differently-authored instrument fills, and it is why the two kinds of set are complements rather than substitutes.
Doesn't a blind score understate what our system can do?
Frequently, yes — and that is part of what makes it informative. A blind instrument may use wording your system has never encountered, or score categories you never claimed. The correct response is to report the number, then decompose it, then state which portion of the gap is lexical coverage and which is not. What is not acceptable is quietly reporting only the number that flatters.
Can a blind instrument ever be reused?
In a bounded way. The reusable-holdout line of work — Dwork and colleagues in Science, 2015 — shows that statistical validity can be preserved across a limited number of adaptive queries if the mechanism adds noise and enforces a query budget. That machinery is real, and it is more apparatus than most small teams will build. The safer operating assumption at small scale is one instrument, one number, then treat it as a development set forever.
What if the published instrument is simply worse than ours?
It might be. Published does not mean good, and some instruments are old, narrow, or built for a related-but-different task. Say so in the report, with the specifics — which items are out of scope and why. What that objection cannot support is reverting to a single self-authored number. If the available instruments are poor, the honest reporting is two imperfect numbers plus a description of each instrument's limits.
Bottom line
A system evaluated only on cases its own authors wrote is being measured with a ruler cut from the thing it measures. The fix is procedural rather than clever: freeze the system, write the scoring rules while still blind, retrieve an instrument somebody else wrote, run it once, and publish both numbers with their provenance. When the blind number is much lower, resist the two easy readings — that the system is broken, or that the instrument is unfair — and do the decomposition instead. Most of the gap is usually vocabulary. The part that is not is the part worth the next quarter of engineering.
Frequently asked questions
An evaluation performed on items the system's builders did not write and have not inspected, using scoring rules fixed before the items were retrieved. The defining property is separation of authorship — between the person who built the system and the person who chose the test items — enforced either by sequestering the data or by fixing the model before retrieval.
Through shared authorship, shared source corpora, and adaptive iteration. The third is the hardest to see: every time a result guides the next decision, information moves from the held-out set into the system by way of the engineer. Dwork and colleagues formalized this in Science in 2015. Large teams with explicit decontamination steps still miss overlaps; the GPT-3 paper documents a filtering bug of exactly that kind.
Not as a held-out set. Once a person who influences the system has read the failures, that set measures how well the system handles cases the team has studied. It remains useful as a regression suite and as a development set. It can no longer support a generalization claim, and reporting it as though it can is the error worth avoiding.
Most often it measures vocabulary and distribution coverage rather than comprehension. Recht and colleagues rebuilt the CIFAR-10 and ImageNet test sets following the original procedures and reported drops of 3–15% and 11–14% respectively, while model rankings were largely preserved. Split your own gap into lexical coverage, distribution shift, and genuine concept failure before drawing conclusions.
The structure is already in federal practice. Under 10 U.S.C. § 139 the Director of Operational Test and Evaluation may communicate views directly to the Secretary and the Deputy Secretary of Defense without the approval or concurrence of any other official in the Department, and may not be assigned responsibility for the developmental test and evaluation the acquiring program runs. NIST's AI Risk Management Framework asks in MEASURE 1.3 that internal experts who did not serve as front-line developers, and/or independent assessors, be involved in regular assessments. Reporting a self-authored number and a differently-authored number together anticipates a question a technical reader is already trained to ask.