The number is the end of an argument you did not get to watch
A benchmark result is a compressed claim. Unpacked, it says: this system, on this data, under these scoring rules, compared against this alternative, at this moment in time, produced this number. Five of those six clauses are usually missing from the slide, and each is a place where a true statement can quietly stop being a statement about your problem. Nobody has to lie for this to happen. Most of the gap between a benchmark and a deployment comes from ordinary engineering decisions that were never written down.
What follows is a reading method, not an accusation. You are not trying to catch anyone; you are trying to learn what the measurement can resolve, so you know whether it supports the schedule and budget you are about to attach to it. Each red flag below comes with the question that exposes it, answerable in a short email. If a vendor cannot answer them, that is information too.
| Red flag | The question that exposes it | A checkable answer (illustrative form) |
|---|---|---|
| A percentage with no denominator | How many items were scored, where did they come from, and how many contained the thing being detected? | "1,204 records sampled at random from one quarter of live intake; 311 contained the event." |
| Recall reported alone | What is the paired error rate, and what counts as a match? | "Recall and false-extraction rate reported together; a hit requires the predicted span to overlap the gold span." |
| The vendor wrote the test set | Who wrote the items, and who wrote the answer key? | "Items from a published instrument we did not author; labels by annotators who had not seen the system." |
| No freeze point | When was the system frozen relative to the first run on this set? | "Commit tagged and artifacts hashed before retrieval; the set was run once, then marked spent." |
| A baseline nobody tuned | How much effort went into the baseline compared with the proposed system? | "Equal search budget on both, same hardware, same folds, configurations published." |
| One number, no spread | What is the variation across seeds, and what was the search budget? | "Median and range across five seeds, with the number of trials that produced the reported best." |
The right-hand column is illustrative wording — the shape of an answer you could verify, not results from any system.
Red flag 1 — a percentage with no denominator
"94% accurate" is not a measurement. It is a measurement with the interesting part removed. Accurate on how many items, drawn from where, collected when, and — the question that does the most work — how many of those items actually contained the thing the system is supposed to find?
Class balance is the hidden variable. A detector scored on a set that is half positive is being asked a different question than the same detector running on live traffic where the event is rare. Recall survives the move; the number of false alarms your analysts have to clear does not. The arithmetic is worth doing once by hand.
Illustrative arithmetic — invented numbers, not a measured result
Take a detector reported at 95% recall on a balanced 200-item set: 100 items contain the event, 100 do not. It finds 95, and suppose it also fires on 40 of the 100 clean items. The reviewer's queue holds 135 alerts, 95 of them real — about 70% worth reading.
Now run the same model where the event is rare: 200 items, only 2 of which contain it. Recall is unchanged, so it finds both; the false-alarm behavior is unchanged, so it fires on about 79 of the 198 clean items. The queue is now 81 alerts, 2 of them real — roughly one useful alert in forty. Nothing was misreported. The benchmark's class balance simply was not the deployment's.
Sample size belongs to the same question. Small evaluation sets cannot resolve small differences, and the difference being claimed is often smaller than the set can see. Card, Henderson, Khandelwal, Jia, Mahowald and Jurafsky measured this in With Little Power Comes Great Responsibility (EMNLP 2020, arXiv:2010.06595): underpowered experiments are common in the literature, small test sets in several GLUE tasks leave most comparisons against a state-of-the-art model inadequately powered, and a typical 2,000-sentence machine translation test set has roughly 75% power to detect a one-BLEU-point difference. Ask for the item count first.
Red flag 2 — recall without an error rate
Recall answers one question: of the things that were there, how many did the system find? It says nothing about how often the system announced something that was not there. Those are independent quantities, and only one of them is usually on the slide.
The reason this matters is mechanical rather than moral. Recall can be purchased with volume: a system that becomes more willing to fire finds more true items and more false ones, and pushed far enough, recall approaches 100% while the output becomes unusable. A single-sided metric cannot distinguish a better model from a looser threshold — and neither can an F1 handed to you without its components, since the same F1 can be assembled from very different pairs.
The companion question is the match rule, which is part of the metric whether anyone says so or not. Does a hit require an exact string, an overlapping span, the right value in the right field, or a human judging the two equivalent? Loosening the match rule raises every number in the table without changing the system at all. We work through the full arithmetic in the false-extraction rate, including why averaging across categories hides the one that carries the consequence.
Red flag 3 — the vendor wrote the test set
When one team writes both the system and the items it is scored on, everything the author knows about the domain leaks into both — the same abbreviations, the same headings, the same instinct about what a hard case looks like. A high score on that instrument is a real result, but narrower than it appears: it shows the system recognizes how this team writes the concept down, which is not the same as recognizing the concept.
Ask two separate questions, because they have different answers: who wrote the items, and who wrote the answer key? A vendor may have licensed a public corpus and still labeled it in-house, putting the judgment calls back in the builder's hands. The mature alternative enforces separation of authorship with something other than good intentions: NIST's face recognition evaluations have developers submit algorithms as black boxes to be run against imagery held at NIST, and NIST's Text REtrieval Conference has participants submit ranked results before the relevance judgments exist. We walk through the mechanics in blind benchmarks.
Red flag 4 — no freeze point, and no version pin
There are two freeze questions and they are easy to conflate. The first: when was the system frozen relative to the first run on this set? If the honest sequence was run, read the failures, fix, run again, then the number is a development number and should be labeled as one. Reading the failure log is enough to spend a set — no code has to change, because the information has already moved into the head of whoever writes the next fix.
The second freeze question applies to anything built on a hosted model: which version, on what date? Model endpoints behind a stable name do not hold still. Chen, Zaharia and Zou documented this in How Is ChatGPT's Behavior Changing over Time? (arXiv:2307.09009, later in Harvard Data Science Review), comparing March 2023 and June 2023 snapshots of GPT-3.5 and GPT-4 across several tasks. Their reported example is blunt: on identifying prime versus composite numbers, GPT-4 went from 84% accuracy in the March snapshot to 51% in the June snapshot, and both models showed more code-formatting mistakes in June. A benchmark run against an unpinned endpoint measures a moment, not a system. Ask for the version string, the run date, and what happens to the number when the provider ships an update mid-contract.
Red flag 5 — the baseline nobody tuned
Most of the reported lift lives in the comparison, and the baseline is the part of the experiment with the least incentive attached to it. The proposed system gets weeks of hyperparameter search; the baseline gets whatever the default configuration produced on a Friday afternoon. Both numbers are real. The gap between them is an artifact of unequal effort.
This has been measured repeatedly. Ferrari Dacrema, Cremonesi and Jannach examined 18 neural recommendation papers from top conferences in Are We Really Making Much Progress? (RecSys 2019, arXiv:1907.06902); only 7 could be reproduced with reasonable effort, and 6 of those 7 could often be outperformed by comparatively simple heuristic methods such as nearest-neighbor approaches. Musgrave, Belongie and Lim found flaws in the experimental methodology of numerous deep metric learning papers (A Metric Learning Reality Check, ECCV 2020, arXiv:2003.08505) and concluded that real improvements had been marginal at best once conditions were held fixed. Earlier still, Armstrong, Moffat, Webber and Zobel showed in Improvements That Don't Add Up (CIKM 2009) that a decade of reported retrieval gains failed to accumulate, largely because researchers compared against their own weak baselines rather than the best published results.
So ask what the baseline was, who configured it, how much search budget it received relative to the proposed system, and whether both ran on the same splits and hardware. Then ask the question underneath: what is the simplest method that would be embarrassing to lose to, and did anyone try it?
Red flag 6 — one number, no spread
A single figure with no variation attached is a best run, not a result. A different random seed moves the number, so does a different data split, and so does the number of configurations searched before the reported one was found. Dodge, Gururangan, Card, Schwartz and Smith made the last point precise in Show Your Work (EMNLP 2019, arXiv:1909.03004): reporting expected performance as a function of search budget, they identified published comparisons where a different budget would have produced a different conclusion.
The practical question is short. How many runs produced this number, what was the spread, and how many configurations were tried? A vendor who answers with a median and a range is telling you something you can plan against. A vendor who repeats the same single number is telling you something too.
The federal version: the conditions were not your conditions
There is a seventh flag that only bites once a system leaves a laptop. Every benchmark was measured somewhere, on something, with certain things turned off — and that somewhere is rarely where the system has to live. A throughput figure from a rack of commercial-cloud accelerators with generous batching does not carry into a fixed GPU allocation in a government cloud region, an air-gapped enclave, or a customer-controlled sandbox with an egress policy. Neither does a latency figure measured without the audit logging, content filtering, and authentication path an accredited deployment requires.
Ask for the hardware, batch size, concurrency, and a tail latency rather than an average. Ask whether the harness ran with production guardrails enabled. And ask whether the artifact can be delivered into the environment you control at all, which is a separate engineering problem from accuracy — we wrote about that in headless deployment into a customer sandbox. Federal buyers have a policy hook here: OMB Memorandum M-25-21 (April 3, 2025), which replaced M-24-10, sets minimum risk management practices for what it defines as high-impact AI. Three of them bear directly on a vendor number: pre-deployment testing with risk mitigation plans that "reflect expected real-world outcomes," a completed AI impact assessment before deployment, and ongoing monitoring for performance and adverse impacts afterward. A number never measured under conditions resembling the deployment does not satisfy that expectation, whatever the slide says.
When the judge is a model
Increasingly the score came not from a scoring script but from another language model grading free-text output. That is a legitimate technique with its own failure modes, documented by the people who popularized it: Zheng and colleagues, in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023 Datasets and Benchmarks Track, arXiv:2306.05685), report position bias, verbosity bias, self-enhancement bias and limited reasoning ability among a judge's limitations — alongside their finding that a strong judge can exceed 80% agreement with human preferences.
So when the grader is a model, ask which model, whether it shares a family with the system being graded, whether answer order was randomized, and what the measured agreement with human raters was on a sample. Self-enhancement bias is the one to watch in a vendor benchmark: it flatters the seller by construction.
What a claim has to state to be checkable
- Population and denominator — how many items, sampled how, from what source, over what period.
- Class balance — how many items contained the target, and how that compares with live prevalence.
- Match rule — what counts as a hit, a miss, a partial, and a non-applicable item.
- Instrument provenance — who wrote the items, who wrote the labels, and when it was published.
- Freeze point — the commit or artifact hash, the date, and the model version string.
- Baseline and its budget — what it was compared against and how much tuning each side received.
- Spread — variation across seeds and splits, plus the number of configurations searched.
- Run conditions — hardware, batching, concurrency, tail latency, and which guardrails were active.
What to request before you sign anything
- The raw confusion counts, not the derived percentages
- The evaluation set itself, or its provenance, authorship, and publication date
- The scoring script, including the match rule as implemented
- The commit hash or model version string, and the date the system was frozen
- The baseline configuration and the search budget each side received
- Per-run results for every seed, including runs that went against the system
- The hardware, batching, and latency conditions of the measurement
What we do here, and what we will not do
We build production machine learning for federal missions — computer vision, natural language processing, time-series forecasting, anomaly detection, tabular prediction — along with the evaluation harness, drift monitoring, model registry, shadow deployment and rollback paths that keep a deployed model honest after the demo is over, described on our machine learning capability page. Reading somebody else's benchmark and building the harness that produces a defensible one are the same skill in two directions.
Here is where the line sits, stated plainly so you can rule us out cheaply. We will not hold both sides of an evaluation: if we build the system, we are not also your independent assessor of it, because a finding we would have to remediate is a finding we have an incentive not to write down. We do not issue authorizations to operate — no contractor does; an authorizing official at your agency does. We are not a C3PAO and cannot assess you for CMMC. We hold no facility clearance today and do not perform classified work on classified networks; design and modeling happen on the unclassified side, and the cleared execution path runs through a partner who holds the clearance. We are not a training vendor and we do not write proposals for other firms. And we will not tell you that anything we build arrives pre-accredited, because nothing does — a system inherits controls and evidence, never an authorization.
We also will not hand you a headline number without its denominators. If the measurement we can produce in the time available is a development number rather than a blind one, we will say so in the deliverable and label it that way.
How a conversation starts
Send the claim you are trying to evaluate, or the problem you are trying to measure — a slide, a vendor white paper, or a paragraph describing the data and what has to be found in it. The first exchange is scoping and costs nothing: what the number would have to survive to be trusted, whether a defensible instrument exists or has to be built, what the class balance in your live data looks like, and whether your deployment environment changes the answer. If the work is a poor fit, that is a reasonable outcome of a first conversation and we would rather reach it early.
Common objections
No vendor will give me any of this.
Some will, and the ones that do are worth noticing. Where a vendor declines, you have learned that the figure is a marketing claim, and you can still buy the product on other grounds — reputation, references, a pilot. What you should not do is attach a schedule, a staffing plan, or a performance clause to a number nobody will describe.
Isn't this an unfair bar for a small company?
The expensive part of rigorous evaluation is sequestered third-party testing, and almost nobody outside a few federal evaluation programs does that. Everything else on this page is free. Reporting a denominator costs nothing. Naming the baseline and its tuning budget costs nothing. Tagging a commit before the run costs nothing. The cheap items are the ones most often skipped, which is why asking for them separates carefulness from size.
Our acquisition timeline has no room for a bake-off.
Then ask for the counts. A confusion table and a sentence about where the items came from can be read in an afternoon, and it resolves more uncertainty per hour than any other artifact in the package. A full independent evaluation is a project; reading a benchmark critically is a meeting.
The vendor says the training data is proprietary, so they can't answer.
Most of these questions are about the evaluation set, not the training set, and a vendor can answer them without disclosing anything proprietary. Where training data does matter is contamination: if the evaluation instrument was published on the open web before the model's training cutoff, the items may already be inside the model. You can ask for the publication date of the instrument and the model's cutoff date without either party revealing a corpus.
Frequently asked questions
How many items were scored, and how many of them contained the thing being detected. That one question recovers the denominator and the class balance at the same time, and those two numbers determine whether the reported figure has any bearing on what your reviewers will experience in production.
Not by itself. Recht, Roelofs, Schmidt and Shankar rebuilt the CIFAR-10 and ImageNet test sets by following the original construction protocols as closely as they could (Do ImageNet Classifiers Generalize to ImageNet?, arXiv:1902.10811, ICML 2019) and reported accuracy drops of 3–15% on CIFAR-10 and 11–14% on ImageNet across a broad range of models. How much of that gap is the models failing to generalize and how much is the replication itself is contested: Engstrom, Ilyas, Santurkar, Tsipras, Steinhardt and Madry (Identifying Statistical Bias in Dataset Replication, arXiv:2005.09619) report that after correcting for statistical bias they identify in the replication process, only about 3.6% of the roughly 11.7% ImageNet drop remains unaccounted for. The practical lesson survives the dispute in either reading: expect a real drop when the items change, budget for it, and treat the size of the drop as a measurement of coverage rather than a verdict.
Usually that it was held out of training, which is necessary but not sufficient. A set can be held out of training and still be spent, because every round of looking at failures and deciding what to try next moves information from the set into the system through the engineer. Ask how many times the set has been run and whether anyone has read the failures.
It can be useful, with known caveats. The authors of the MT-Bench work report position, verbosity and self-enhancement biases in model judges alongside high agreement with human preferences on their data. Ask which judge model was used, whether it shares a family with the system under test, whether answer order was randomized, and what the measured agreement with human raters was.
We can read a claim, reconstruct what it does and does not establish, and design an instrument that would settle the question. What we will not do is serve as the independent assessor of a system we built ourselves, and we do not issue authorizations to operate or perform CMMC assessments — those sit with an authorizing official and a certified third-party assessor respectively.