The number was never in dispute. The ruler was.
A model comes back at 94% and the room splits. Engineering believes it. The business owner who will live with the output does not, and cannot say why, so the objection comes out as a feeling. Six weeks later the pilot runs on real traffic and lands in the low eighties. Nobody lied. The 94% was computed correctly, on a set the building team assembled, from data the building team had already looked at, at a threshold picked after seeing the curve. Each of those choices moves the number up a point or three, and none is fraud. They are the ordinary gravity of a party measuring its own work.
The fix is structural and cheap. The buyer selects the evaluation items, holds the labels, and never shows them to the builder until the scored run. That removes every degree of freedom the builder had over the verdict. It also gets skipped, because it costs two days up front and there is always a reason to start modeling instead.
What follows is why the optimism creeps in even among careful people, what it costs in points, how to build the instrument, and what a vendor's reaction tells you.
Who holds the answer key holds the verdict
A test set is not data. It is an instrument with a fixed number of readings in it, and a reading is spent every time somebody sees a result and changes something. A team that evaluates against the same held-out set twelve times in a quarter has not tested twelve times. It has run a slow, well-intentioned search over that set, and the final number says how well the model fits those particular items rather than the next ten thousand. Tuning a hyperparameter, dropping an ensemble member that hurt the score, moving a threshold to where precision and recall balance nicely: each is defensible alone, and together they are gradient descent performed by hand on the answer key.
When the buyer holds the key, that failure mode becomes impossible rather than discouraged. There is nothing to optimize against, because the score does not exist until the run. The builder falls back on a development split carved from their own training data, which is where tuning belongs. The discipline comes from not having access, not from good intentions.
Trust in a reported score, by who controlled each part of the measurement
Editorial weighting from delivery practice and published evaluation guidance. Illustrative, not a measured statistic.
What the optimism actually costs, in points
The gap depends on how much searching happened and how big the set is, and rough anchors beat a shrug. On a few hundred items, a team that tuned across a dozen evaluation cycles typically reports two to five points above what a fresh sample returns. That is the careful case, and not the interesting one.
The larger gaps come from sampling. A test set built from the records that were easy to extract, or the ones with clean labels, or one quarter's worth, can sit five to fifteen points above production without any tuning at all. The document classifier tested on one business unit meets the second unit's formatting conventions and falls apart. Nothing was optimized. The instrument was pointed at the wrong population.
The third gap is temporal, and it costs the most because it appears after signature. A model tested on a random split of historical data was evaluated on a task it will never face: predicting the past from the past. Deployment predicts forward. A holdout drawn from a window strictly after the training data ends measures the real job, and on problems where behavior shifts the difference is regularly ten points or more.
The rule already exists in regulated work
Independent evaluation is not a demand invented for AI. Banking supervisors wrote it down in 2011. The Federal Reserve's SR 11-7, Guidance on Model Risk Management, requires validation by people independent of model development, reasoning that "staff conducting validation should have explicit authority to challenge developers" and should not have a stake in the outcome. Fifteen years on, the institutions living under it do not treat the separation as an insult to their modeling teams. It is plumbing. NIST AI RMF puts the same idea under Measure, and ISO/IEC 42001 asks for documented evidence of performance verification that an internal self-scored number satisfies only weakly.
Federal AI acquisition has now landed in the same place. OMB Memorandum M-25-22, Driving Efficient Acquisition of Artificial Intelligence in Government, issued April 3, 2025, instructs agencies to use data they have defined, such as agency validation and testing datasets, when conducting independent evaluations, and states that the data used "should not be accessible to the vendor, and should be as similar as possible to the data used when the system is deployed." It further requires that vendors provide the access and time necessary to complete an independent evaluation, that where a vendor performs testing the results be detailed enough to be independently verified or reproduced, and that contracts not prohibit an agency from internally disclosing how the vendor conducts testing or the results of it.
Building the instrument takes about two days
The work is not hard and it is not glamorous, which is why it gets deferred. Someone with domain knowledge and access to the operational records does the following, once, before any modeling starts.
Define the population in a sentence. Not "our claims data." Something like: claims from the web portal and fax intake, March through August, all four regions, excluding the two product lines being sunset. If the sentence cannot be written, the sample cannot be defended when the number comes in low.
Stratify deliberately, and include the ugly ones. A pure random draw underweights the cases that break systems, because those are rare by construction. Force in the low-quality scans, the handwriting, the edge product types, the acquired subsidiary's different field conventions. Record the strata and proportions so the overall number can be reweighted honestly.
Split by time, not at random, wherever time exists. Everything before the cut date is available for development. Everything after is the holdout. This single choice catches more failures before deployment than any other item on this list.
Label with a written rule and measure your own agreement. Two people label an overlapping subset independently and the disagreement rate gets recorded. If experienced staff disagree on 12% of items, no model will be scored meaningfully at 95%, and that should be known in week one rather than in an acceptance dispute.
Freeze it, hash it, and store the key separately. Inputs go to the builder. Labels stay in a place the builder has no credential for. Record a hash of the sealed set so both parties can prove months later that nothing moved.
Carve out a reserve partition at the same time. Seal a second, smaller set from the same draw and leave it closed. When the first attempt misses, the retry gets scored on fresh items. Without a reserve, the retry runs against a set the builder has now seen results from, and the arrangement collapses back into what it was meant to prevent.
Contamination has more entrances than the obvious one
Sealing the labels is necessary and not sufficient. The leaks that survive a careful sealing are the ones worth knowing by name.
Near-duplicates across the split. The same invoice submitted twice with a different reference number. The same incident written up by two shifts. The same product photographed twice. Fuzzy matching before the split takes an hour and catches most of it.
Group leakage. Splitting a medical dataset by image rather than by patient puts three scans of the same person on both sides. Splitting a support dataset by ticket rather than by customer does the same thing. Split by the entity, not by the row.
Features computed after the fact. A field populated during downstream processing carries information the model would not have at decision time, and so does any aggregate over a window that includes the target period. This has ended more projects than everything else here, because the model looks superb until the day it meets live inputs and the field is empty.
Preprocessing fit on everything. Normalization statistics, vocabularies, embedding models, and imputation values fit across the full dataset before splitting all carry holdout information into training. Fit them on the development portion only.
| Leak | How it shows up | The check |
|---|---|---|
| Near-duplicate records | Score falls two to six points on a clean resample | Fuzzy match across the split before sealing |
| Group leakage | Strong on known entities, weak on new ones | Split by patient, customer, site or case, never by row |
| Post-outcome features | Near-perfect offline, collapses in production | Timestamp every field against the decision moment |
| Preprocessing fit on all data | Small, consistent optimism of one to three points | Fit transforms inside the training fold only |
| Random split on time-ordered data | Good offline, degrades within weeks of launch | Cut by date; hold out the later window |
| Repeated evaluation | Score creeps up over the project with no real gain | Buyer holds labels; count and cap the attempts |
Typical points of offline optimism, by source of the gap
Ranges from delivery practice across document, tabular and vision workloads. Illustrative, not a measured statistic.
How many items, and the arithmetic nobody runs
A sealed holdout of forty items is theater with better manners; the confidence interval swamps any decision it could inform. A measured 90% on 100 scored items has a 95% Wilson interval of roughly 82.6% to 94.5%. On 500 items it tightens to about 87.1% to 92.3%, and on 2,000 to about 88.6% to 91.2%. If the acceptance floor is 88% and the set holds 100 items, a reported 90% has not cleared it in any defensible sense.
Comparison costs far more than measurement. Showing that a system at 90% beats a baseline at 85%, at 95% confidence with 80% power, takes about 690 scored items per arm. Separating 90% from 88% takes roughly 3,800 per arm. Most business cases rest on that kind of narrow margin while most evaluation budgets are sized for a point estimate. One choice buys much of it back: score the incumbent process and the new system on identical items and count only the disagreements. Per-item difficulty cancels, and a paired comparison on 300 items does what an unpaired one needs a thousand for.
Slice floors matter more than the headline. An aggregate of 91% with one region at 62% is a system that gets pulled after the first complaint from that region. Set a floor per stratum and size each stratum so its own number means something, which drives the total count up. Finding the weak slice during evaluation is far cheaper than finding it during rollout.
What it means when the vendor asks to hold the set
There are honest reasons. A team that has never been handed clean labeled data may offer to build the set because they suspect nobody else will. Those cases are workable: the buyer defines the sampling and labeling rules in writing, a separate team executes them, and the building team still never sees the answers.
The answer that should stop the conversation is one that treats the request as unusual. A firm that has delivered production models under acceptance criteria knows the arrangement protects them as much as the buyer: when the number clears the floor it clears cleanly, and nobody spends a quarter relitigating whether the test was fair.
The counter-asks are worth granting, because they are the price of a fair test. The vendor needs the sampling and labeling rules in advance so they build toward the right population. They need enough development data of the same character, and the disagreement rate among your own labelers, since it caps what any system can score. They need a stated operating threshold, or the same model can be made to look strong or weak by moving it. And they need a defined number of attempts against the reserve, so a first-run miss is an engineering problem rather than the end of the engagement.
Writing it into the agreement
Four sentences carry the arrangement, and they belong in the statement of work rather than an appendix nobody signs against. The buyer defines and holds the evaluation data, and it is not accessible to the vendor before the scored run. The scoring rule, operating threshold and per-slice floors are fixed in writing beforehand. The evaluation is reproducible by a third party from the delivered artifacts. The number of attempts is stated, and later attempts use reserves sealed with the first.
For work touching federal customers the same artifacts feed NIST AI RMF Measure documentation and a system card that has to survive an authorizing official's reading. On a controlled-unclassified program the sealed-set mechanics have to respect where the data can live under 800-171, so the evaluation design and the security boundary get decided together rather than in sequence.
Bottom line
The score on a machine learning system is a property of the measurement as much as the model, and whoever holds the answer key controls both. Handing that key to the buyer costs two days up front and removes an entire category of dispute at acceptance. Size the set for the decision, split it by time, seal a reserve, set floors per slice, fix the threshold before the run. The number that comes back will be lower than one a vendor computes on their own set. It will also be the one that survives production, which is the only number either party was trying to buy.
Frequently asked questions
A validation set is used during development to compare model variants and tune settings; it gets looked at repeatedly and its number drifts optimistic by design. A holdout set is opened once, at the end, to estimate performance on unseen data. Using them interchangeably is the most common way an inflated score gets reported in good faith.
It depends on what the number has to prove. A point estimate with a usable interval wants several hundred scored items. Showing a five-point improvement over an existing process at conventional confidence and power takes roughly 690 per arm, or far fewer if both are scored on identical items and only disagreements are counted. Per-slice floors raise the total further, since each slice needs enough items to stand on its own.
Yes, and the constraint often improves the arrangement. The evaluation runs inside your boundary against the sealed set, with the vendor supplying a container and receiving only the aggregate and per-slice results plus a confusion matrix. On CUI work the same pattern runs inside the 800-171 boundary, with the evaluation artifacts documented as part of the system record.
That is a normal engineering outcome if the agreement anticipated it. State the number of attempts in advance and score each later attempt against a reserve partition sealed at the same time as the first. What should not happen is rescoring against the original set after the team has seen where it failed, which converts the holdout into a tuning set.
No. The work is domain work: defining the population, drawing a stratified sample, applying a written labeling rule, and keeping the labels in a place the vendor cannot reach. Scoring is arithmetic on two columns. The skills that matter are knowing which cases are hard and which records are trustworthy, and those live with your operations staff rather than with a data scientist.