Nobody in this story is lying
The useful thing to understand about an inflated benchmark is that fraud is almost never the explanation. A team picked a test set, ran the model, and reported what came out. Every step was defensible on its own. The number on the slide is arithmetically correct and reproducible on the machine that produced it. It is still going to be wrong by fifteen to forty points when you run the same model on your own traffic, and if you go in expecting to catch someone in a lie you will not find one, and you will buy the system anyway.
This matters because the two ways buyers usually respond both fail. The first is to take the number at face value, sign, and discover the gap in month four of a deployment that has already been announced internally. The second is to distrust all vendor numbers, demand a pilot, and end up running an eight-week bake-off that measures the vendors' willingness to staff a pilot rather than the quality of their systems. Neither approach touches the actual mechanism.
What works is narrower and cheaper. A small set of structural facts about how a benchmark was constructed determines whether its number survives contact with your data. You can get at all of them in one meeting, without a data scientist in the room, and the answers are either available immediately or they are not, which is itself the finding.
The mechanism: a test set is an instrument, and instruments drift toward their owner
Take a team building a document classifier. They need a test set, so they take a sample from the data they have. The data they have is the data they collected while building the thing, which came from the customers who adopted earliest, in the format those customers happened to use, with the label definitions the team settled on in month two. Nothing here is careless. It is the only data that existed.
Then the team starts iterating. They run the model, look at what it got wrong, and fix those cases. That loop is how engineering works and no version of building a good system skips it. But every pass moves information from the test set into the model, through the developers. After forty iterations the set is no longer measuring generalization. It measures how well the team has fit the quirks of a few thousand examples they have all read. The score keeps rising after the underlying capability stopped improving.
The result is honest, reproducible, and specific to an instrument calibrated to its builder. It transfers to you as far as your documents, your label definitions, and your edge cases resemble theirs. Usually that is further than zero and well short of what the slide implies.
How much a reported score typically survives, by construction of the test set
Editorial weighting from practitioner reading. Illustrative of how far a number travels, not a measured statistic.
Contamination for models trained on the open internet
The version of this problem specific to large language models is simpler and worse. If a benchmark was published on the web before a model's training data was collected, the model may have read the answers. This is not a hypothetical concern and it is not evenly distributed across benchmarks. Sets that have been public for years, with solutions posted in tutorials, forum threads and scraped repositories, are the most contaminated. Sets released after a model's cutoff are clean for that model and contaminated for the next one.
The tell is a score profile that does not degrade the way capability does. A model that reasons about a problem class degrades gracefully as problems get harder and their surface form changes. A model that memorized a benchmark scores well on the published items and falls off a cliff on rephrasings a person would call equivalent. Changing variable names, reordering multiple-choice options, or restating a question should move a real score a point or two. Ten or fifteen points means you were measuring recall.
Ask which public benchmarks appear in the vendor's numbers and when each was published relative to the model's training cutoff. A vendor building on a third-party foundation model often does not know the cutoff precisely, and saying so is a fine answer. Saying the question does not apply is not.
The four questions
These take about twenty minutes and work on any vendor, in any domain, whether or not you can read code. Each has a right answer and a set of wrong ones, and the wrong ones are informative rather than disqualifying.
When was the test set created, relative to the work? The answer you want is a date before development started, with the set frozen and stored separately. The answer you often get is that the split was made from the full data pool at some point during the project. That is not fatal, but it means the set and the model grew up together and the number carries optimism you should discount.
How many times has the team looked at examples the model got wrong on that set? This produces the most useful silence in the meeting. The honest answer from a working team is "many times, that is how we improved it," which tells you the reported score is a fitting score rather than a generalization score. A team that says "never, we hold a separate development set for error analysis" has done the expensive, correct thing and will be glad you asked.
What is the score on data the vendor has never seen? The right answer is a number from a third-party set, a customer's held-out data, or a set built after the model was frozen. Frequently there is no such number, which means nobody has ever measured generalization, including the vendor. That is a gap, not a scandal, and it is fixable in about two weeks.
What does the failure distribution look like? Not the aggregate. Ask for accuracy on the hardest decile, on the categories your business cares about, and on inputs that arrive malformed. The aggregate hides the shape of the errors, and the shape decides whether the system is usable. A model at 94 percent that fails randomly is a different product from one at 94 percent that fails on every document from one large customer.
Why aggregate accuracy is the wrong number to negotiate over
A single accuracy figure assumes every error costs the same. Almost no real workflow has that property. In claims review, a false negative that pays a fraudulent claim costs a fixed dollar amount; a false positive that flags a legitimate claim costs a review hour plus some customer goodwill. In a safety-adjacent inspection workflow the asymmetry is far larger. The two error types have to be priced separately before an accuracy number means anything to a budget owner.
The consequence for a buyer is that the right instrument is not a benchmark at all. It is a cost model with two rates in it, applied to a confusion matrix from your own data. A vendor at 91 percent with errors concentrated in the cheap direction can be worth more than a competitor at 95 percent whose errors sit on the expensive side. That comparison is invisible on a slide that reports one number, and it is the comparison that determines whether the deployment pays for itself.
The operating-point question lives here too. Most systems expose a threshold that trades the two error types against each other. Ask where the reported number sits on that curve, and whether the threshold was tuned to maximize the metric being shown. Often it was, which is reasonable, and it means the number is the peak of a curve rather than a point you would run at.
The regulated-industry version of the same argument
If you sell into banking, insurance, health or anything the federal government buys, this stops being a purchasing preference and becomes a documentation requirement, and the language already exists.
The Federal Reserve and OCC guidance on model risk management, SR 11-7, has required since 2011 that validation be performed by people independent of model development, and that it include outcomes analysis comparing model outputs to actual outcomes. The document describes exactly the problem in this article: a model validated by the team that built it is not validated. Banks have answered this about statistical models for a decade and a half, and the same standard now lands on machine learning systems.
The NIST AI Risk Management Framework puts the same requirement in its Measure function. Measure 2.3 covers performance demonstrated against deployment conditions. Measure 4.2 covers whether measurement approaches are valid and whether the evidence would be recognized outside the team. ISO/IEC 42001 requires documented evidence of performance evaluation and internal audit against defined objectives. For systems handling federal data, NIST 800-53 assessment procedures and the FedRAMP process both expect evidence produced against a defined baseline rather than a self-reported figure.
None of these forbids a vendor-built test set. They say the same narrower thing: evidence must trace to a defined set, be produced under a documented procedure, and be reviewable by someone who did not build the model. That is a lower bar than it sounds, and most teams clear it in a few weeks once they decide to.
What an honest evaluation costs to build
Vendors rarely lack these numbers out of reluctance. Building an evaluation set is unglamorous work nobody schedules. Here is what it takes, so you can judge whether a vendor's timeline is credible.
A domain-specific held-out set of 500 to 2,000 examples, labeled by people who know the domain, with a written label guide and a double-labeled subset for measuring annotator agreement, runs roughly 80 to 200 hours of combined engineering and subject-matter time. At blended rates that is somewhere between $15,000 and $50,000, and calendar time of two to four weeks if the labeling can run in parallel. The largest variable is whether the label definitions are already settled. If two experts disagree about what the right answer is on 15 percent of cases, no model can score above 85 percent and the first job is the guide, not the model.
The harness that runs the set, versions results, and reports breakdowns rather than one aggregate is another 40 to 80 hours. Wiring it into continuous integration so every model change produces a scorecard, with the set stored where developers cannot casually inspect it, is 20 to 40 more. The whole thing lands between $30,000 and $90,000 for a serious first version, against deployments that routinely carry seven-figure annual cost.
Share of a first evaluation build, by where the hours go
Editorial weighting from practitioner reading. Illustrative of effort distribution, not a measured statistic.
The ongoing cost is smaller and it is the part teams skip. A held-out set decays as the world changes. Budget a refresh every two quarters, pulling new examples from live traffic and retiring items that no longer represent anything, at roughly a quarter of the original labeling effort. A set that has not been refreshed in eighteen months is measuring a business that no longer exists.
When you are the vendor being asked
If you build the system, these four questions are coming at you from the other side of the table, and the move is to answer them before they are asked. A vendor who walks in with the score on their own set, the score on a third-party set, and an explanation of the gap is doing something no competitor is doing. It reads as confidence because it is confidence. Nobody publishes their worse number unless that number is defensible.
The gap itself is the interesting artifact. A five-point drop from internal to external is normal and means the internal set is reasonably constructed. Twenty points means the internal set is contaminated or unrepresentative, and knowing that before a customer discovers it is worth considerably more than the twenty points. A vendor who can say "we score 88 on our set and 81 on an independent set, and the gap is concentrated in scanned documents from before 2019" has told a buyer more useful information than any single number could carry.
It is also the cheapest way to shorten a sales cycle with a regulated buyer. The evidence a bank's model risk function or a federal authorizing official needs is the evidence described above. Produce it once, properly, and every subsequent deal reuses the artifact instead of relitigating the question.
How we work on this
We spend the first week on the label guide rather than the model, because disagreement among your own experts sets the ceiling on every number anyone will ever report. We measure inter-annotator agreement before anything else, and that one measurement often changes what the project is about.
Then we build the set the boring way: frozen at the start, stored where the development loop cannot reach it, with hashes recorded so anyone can confirm later that it never changed. Error analysis runs on a separate development set the team may read freely. The two-set discipline is the whole trick. It costs almost nothing on day one and is expensive to retrofit at month nine.
We report breakdowns rather than the aggregate: by category, by input quality, by hardest decile, and by whatever segmentation your business cares about. And we write the harness to produce the artifact your reviewer needs, whether that is an independent validation function under SR 11-7, an auditor working to ISO/IEC 42001, or a federal assessor working from 800-53 assessment procedures. Measurement and documentation are the same work if you plan them together, and two projects if you do not.
Common objections
"We do not have enough data to hold any out." Usually this is a statement about labeled data, not raw data. Five hundred carefully labeled held-out examples beat five thousand used for both training and reporting. If the labeled pool is genuinely small, cross-validation with a written protocol is defensible, provided the protocol was fixed in advance and the folds were not re-drawn after seeing results.
"Our domain has no public benchmark, so there is nothing to compare to." True and beside the point. The comparison that matters is against the process the system replaces. Measure your current human workflow on the same held-out set. That gives you a baseline that is more relevant than any public number and it is often the measurement that justifies the budget.
"The model changes every month, so any fixed set goes stale." The model changing is the reason to have a fixed set. A frozen instrument is what lets you say whether last month's change helped. Refresh the set on a schedule and version it, so you can distinguish a real regression from an instrument change. Both happen and they look identical if you change the set and the model at once.
Bottom line
Benchmark inflation is a structural property of how systems get built, not a character flaw in the people who build them. The test set drifts toward its owner, the error analysis loop leaks information into the model, and public benchmarks get absorbed into training data. Each produces an honest number that will not transfer. Four questions about how the set was constructed tell you how much of a reported score to believe, and an instrument you can defend costs between $30,000 and $90,000 against deployments worth many times that. Buyers who ask get a real number. Vendors who answer first win the deals.
Frequently asked questions
It depends entirely on how the test set was built, which is why the construction questions matter more than the number. A set frozen before development and never inspected transfers well. A set the team read failures from across dozens of iterations can overstate performance by twenty points or more on your data.
A pilot measures the right thing but costs six to twelve weeks and mostly tests whether the vendor will staff it. Running your own held-out data through two or three vendors' systems in a week gives a cleaner comparison for a fraction of the effort, provided you built the set first.
A model scoring well on a test because the test's contents and answers appeared in its training data. Most common with public benchmarks published before a model's training cutoff. The tell is a score that collapses when questions are rephrased in ways a person would consider equivalent.
The guidance is written about models generally rather than any specific technique, and supervisors have applied it to machine learning systems used in covered decisions. Its core requirements, independent validation and outcomes analysis against actual results, translate directly.
Roughly every two quarters for most business workflows, faster where inputs change quickly. Refresh by adding new examples from live traffic and retiring items that no longer represent anything current. Version each refresh so you can tell a real regression from an instrument change.