How to verify an AI vendor's benchmark claim

A number without a protocol is not a result

A vendor tells you their model is 94% accurate. The number is probably real. The question is what it is a number about. Accuracy is a measurement, and a measurement carries no meaning without the protocol behind it: what data it was taken on, how many times, against what alternative, and how much of the gap between 94% and what you already do is signal rather than noise. Federal buyers see these figures in capability briefs, white papers, oral presentations, and Phase II proposals. Most of the time nobody asks.

The asymmetry is the whole problem. The vendor ran the experiment. You did not. Reproducing it costs you weeks you may not have, and the vendor knows that. So the practical move is a short list of questions whose answers are trivial for a firm that actually measured something and painful for a firm that assembled a demo. Nobody has to become a statistician. You have to know what a real answer sounds like.

Federal policy points the same way. The NIST AI Risk Management Framework (AI 100-1, January 2023) puts test, evaluation, verification, and validation under its MEASURE function. GAO's AI accountability framework (GAO-21-519SP, June 2021) gives performance its own pillar, including assessing a model against its intended use rather than a general capability. OMB's April 2025 AI acquisition memorandum (M-25-22) pushes agencies toward performance-based techniques and measurable criteria set before award. None of that helps if the criteria at award are the vendor's own numbers, unexamined.

Question 1. What dataset, and who assembled it

Ask for the name, the version, the size, the date range, and the label source. A vendor who measured something has all five in a file. A vendor who did not will describe the dataset in adjectives.

If the answer is a public benchmark, the follow-up is contamination. A model trained on a broad web crawl through some date has plausibly seen any benchmark published before that date, answers included. That is not a hypothetical: it is the reason a strong score on a well-known public set has become weak evidence about anything. Ask when the benchmark was published, when the training data was collected, and whether anyone checked for overlap.

If the answer is a vendor-assembled set, the follow-up is labels. Who produced them, what the agreement rate between annotators was, how disagreements were adjudicated, and whether any labels were generated by another model, which quietly converts your evaluation into a measurement of how well one model imitates another. The short version of the ask is one sentence: send me the data card. Documenting provenance, collection method, and intended use on a standard sheet has been common practice since Gebru and colleagues proposed datasheets for datasets, and a firm with a real evaluation pipeline usually has one.

Then the question that matters most to you and least to the vendor: does the dataset resemble your work. Class balance, document quality, scan resolution, vocabulary, form versions, the share of records that are simply a mess. A model measured on clean data will not hold its number on your data, and the size of the drop is the single largest source of disappointment in deployed federal AI.

Question 2. What split, and when was it drawn

The number you were shown should come from data the model never saw during training or model selection. That sounds obvious and is violated constantly, usually by accident.

Ask when the test set was drawn and how many times it has been scored since. A test set that development scored three hundred times is a validation set wearing a costume: every design decision that followed a score was fitted to it. The honest pattern is a held-out set touched once at the end, or a fresh set drawn for each reported milestone.

Two leakage patterns show up constantly in exactly the workloads agencies buy. Temporal leakage happens when time-ordered data gets a random split, so records from the future sit in training while the model is scored on the past. Maintenance prediction, fraud detection, claims adjudication, and forecasting are all vulnerable. The fix is a time-based split: train before a cutoff, test after. Group leakage happens when the same underlying entity lands on both sides, the same patient, vehicle, case file, or applicant, so the model recognizes the entity rather than the pattern. Ask what the split key was. If the answer is "random rows," ask what a row represents.

Question 3. How many runs, and what was the spread

One run is an anecdote. Ask for the number of runs and either the standard deviation or the minimum and maximum across them. If the vendor cannot say, the answer is one.

This is not pedantry. Work by Dodge and colleagues on fine-tuning pretrained language models showed that changing only the random seed produced swings large enough to look like a method improvement, and that reporting the best of several seeds inflates the apparent result. Any pipeline with random initialization, random data ordering, or stochastic decoding behaves this way. Two systems can differ by two points on a single run and be identical.

Sampling noise in the test set is the second source of variance and gets ignored even more often. A 500-example test set carries real uncertainty in every number computed from it. Ask for a bootstrap confidence interval. It is a few lines in any scoring script, and it converts "94%" into something you can reason about.

Question 4. What baseline, and was anyone paid to make it good

The comparison determines the claim. "Better than the current process" means nothing until the current process has been measured under the same conditions, on the same data, with the same scoring code.

The recurring failure in published machine learning is the undertuned baseline, and it is well documented. Dacrema, Cremonesi, and Jannach examined eighteen neural recommendation methods from top conferences in 2019; only seven could be reproduced with reasonable effort, and six of those seven were beaten by simple, properly tuned nearest-neighbor or graph-based baselines. Melis and colleagues showed at ICLR 2018 that a carefully tuned LSTM matched or beat the architectures reported to have replaced it. In information retrieval, Jimmy Lin's work on weak baselines made the same point about neural rankers against a properly tuned BM25. In each case the new method was real work. The reported margin was mostly tuning budget.

So ask about tuning budget directly. How many configurations were swept for the proposed system, and how many for the baseline. If the answer is "we swept ours and used defaults for the baseline," you have a result about search effort, not about method. Ask a second question too: is the baseline the thing you would actually do otherwise. Sometimes that is a rules engine, sometimes a regular expression, sometimes an existing system of record, sometimes three people and a checklist. Those are the alternatives you are choosing between, and a vendor who has never measured them is asking you to compare their number to your imagination.

Question 5. Does the metric match the decision

Accuracy on an imbalanced problem is close to meaningless. If 1% of records are the thing you care about, a model that answers "no" every time scores 99%. The metrics that describe the decision are precision and recall at the threshold you would deploy, and the two are not interchangeable because the costs of the two errors are almost never symmetric. A missed case and a false accusation are different events with different consequences, and the operating point should be chosen from the consequences, not from whichever number looks best.

Ask for the confusion matrix at the deployed threshold and what review workload it implies at your volume. ROC-AUC deserves specific suspicion on rare-positive problems: it is insensitive to prevalence and looks strong on systems whose positive predictions are mostly wrong at any usable threshold. Precision-recall curves and positive predictive value at the operating point are the honest versions.

Then check that the reported unit matches the unit of your work. Per-field extraction accuracy and per-document correctness are different quantities. A form with 40 extracted fields at 98% per field, with independent errors, is fully correct end to end about 45% of the time (0.98⁴⁰ ≈ 0.446). Nothing was misrepresented. The buyer read a field-level number as a document-level promise.

The claim	What it can hide	What to ask for
"94% accurate"	Class balance and the chosen threshold	Confusion matrix at the deployed threshold, plus prevalence in your data
"Outperforms the leading model"	Which model, which version, tuned by whom	Baseline configuration, tuning budget, paired difference with an interval
"State of the art on [benchmark]"	Whether the benchmark resembles your workload	Data card, date the set was drawn, contamination check
"Cuts processing time 3x"	The human review the output still requires	End-to-end time including correction and exception handling
"Validated on 10,000 documents"	How many were independently labeled	Label source, annotator agreement, adjudication rule
"Production-proven"	Whose production, at what load, for how long	Environment, load profile, measurement window, incident history

Question 6. What results were not shown

The vendor ran more experiments than they presented. That is normal and not by itself dishonest. It does change what the presented number means.

Ask how many configurations were evaluated before this one, and what the others produced. If forty were tried and the best was reported, the best is biased upward by selection alone. The arithmetic is familiar to anyone who has run a multiple-comparisons correction: at a 5% significance threshold, testing twenty independent variants yields roughly one "significant" result by chance, with no underlying effect. Reporting only the winner turns noise into a finding.

Watch for scoped subsets. "On the subset of high-quality scans" is a real result about a subset, useful only alongside the share of your corpus that subset represents and what happens on the rest. Same with "excluding documents that failed preprocessing." Those failures do not disappear at deployment. They become your exception queue, and the size of that queue is often the deciding cost.

Question 7. Where does it fail

Every real system fails somewhere. Anyone who has run a model against messy operational data knows the shape of its failures, because failure is what you spend your time on. If nobody at the vendor can name a failure regime, either the evaluation was shallow or the answer is being withheld. Either way it becomes your problem after award.

Good answers sound specific and slightly unflattering. Recall drops below 200 dpi. It confuses two entity types when the address block is missing. It degrades on forms printed before the 1998 revision because the field order changed. It cannot read cursive. Answers like these are evidence of contact with real data. Polish is not evidence of anything.

Two follow-ups make the answer useful. What fraction of a corpus like ours falls in that regime. And what does the system do when it is there: return a confident wrong answer, abstain, or route to a human. A model that fails loudly is deployable. A model that fails silently at 3% is a slow-moving audit finding. Ask whether confidence scores are calibrated, and ask for a reliability diagram. It takes five minutes to produce for anyone who already has one, and its absence is informative.

Question 8. Does it reproduce on your instrument

Everything above is a conversation. This one is a test, and it is the only step a vendor cannot configure in advance. It needs neither a large budget nor a large dataset. A few hundred carefully labeled examples that look like your real work beat fifty thousand sloppy ones.

Two details carry most of the weight. Ask for predictions rather than scores, because scoring code makes definitional choices nobody documents: partial credit, string normalization, whitespace, tie handling, what counts as a match on a date field. Two reasonable scripts can differ by several points on identical predictions. And read the disagreements by hand. An hour with fifty errors teaches you more about fit than any aggregate number, and it often reveals that some of the "errors" are label problems on your side.

Put the verification in the contract, not the kickoff meeting

A benchmark claim examined after award is a complaint. Examined before award, it is a specification. Performance-based acquisition gives you the structure, and the criteria belong in the SOW or PWS with acceptance tied to the inspection terms of your contract type. This matters most on small buys. Below the simplified acquisition threshold at FAR 2.101, currently $250,000, you may never run a full source selection, and an unexamined vendor number does more damage there than in a large competition where technical evaluators will pull the claim apart anyway.

Name the metric and the operating point. Not "high accuracy" but recall at or above a stated value at a precision floor, measured at the threshold that will actually be deployed.

Name the dataset and who holds it. Acceptance testing on government-held data that the contractor has not seen is the difference between a test and a demonstration. State who draws the sample and who runs the scoring script.

Name the reporting format. Mean and spread across a stated number of runs, the confusion matrix at the operating threshold, and the paired comparison against the incumbent process with an interval.

Name a monitoring window after acceptance. Models drift as inputs change, and a number measured at acceptance is not a number in month nine. A quarterly re-score on a fresh sample costs little and catches what acceptance testing structurally cannot.

What an honest "we haven't measured that" sounds like

Some of these questions will get the answer "we don't know yet." That is a good answer. It is more informative than a confident number, and it shows the firm knows the difference between a measured quantity and an estimate. Preliminary work with a small sample and a wide interval is legitimate engineering. The problem is preliminary work presented as settled.

What separates the two is whether the vendor knows what they have not looked at. A firm that says "we have not tested on handwriting, and we would want two weeks and a labeled sample from you before claiming anything there" is describing a plan. A firm that answers every question with a percentage has either done extraordinary work or has never been asked. You can tell which within about twenty minutes.

Bottom line

The eight questions cost you one meeting and cost a vendor with real measurements almost nothing. Dataset and provenance. Split and leakage. Runs and spread. Baseline and tuning budget. Metric fit to the decision. What was not shown. Where it fails. Whether it reproduces on data you control. Any one of them will separate most configured demos from most real results, and running all eight tells you more about a vendor's engineering discipline than the past performance narrative will.

Frequently asked questions