Choosing a baseline you can defend

The control decides the claim

An accuracy of 0.94, an AUC of 0.91, a 38 percent reduction in review time: none of those sentences carries information on its own. Every one of them is a difference, and a difference needs two terms. The reader who is being asked to fund something, buy something, or stake a program schedule on something is entitled to the second term, and in most reports it is either missing, vague, or chosen to lose. The control condition is not a formality that follows the modeling work. It is the modeling work's only unit of measurement.

This shows up in the plainest possible way in fraud detection, rare-event forecasting, and safety screening, where the positive class is uncommon. A classifier that predicts "no event" for every record in a population with a 2 percent event rate is 98 percent accurate. Nobody writes it that way, but every accuracy figure reported without the majority-class rate beside it is asking the reader to guess at exactly that comparison. The same trap has a version in every problem class. In time series it is last-value-carried-forward. In retrieval it is keyword search. In document processing it is a regular expression. In classification of tabular records it is a logistic regression on five features.

The uncomfortable half of this is the part practitioners avoid saying out loud. Sometimes the trivial control wins. When it does, that is not a failed experiment to be re-run with a friendlier comparison. That is the experiment's result, and it belongs in the report at the same size and in the same font as everything else.

Three controls, and a serious evaluation owes all three

One baseline is almost never enough, because the three questions a reader has are different questions and each needs its own control. Is the signal real at all? Is this better than what the organization does today? Is the complicated method earning its complexity over the simple one? A single comparison answers one of those and leaves the other two open for whoever reviews the work later.

Control	What it is	What it protects the claim from
Degenerate	Majority class, base rate, seasonal naive, last observation carried forward, random ranking	The class-imbalance illusion. It establishes the floor, and it is the cheapest run in the whole study. A method that does not clear it has produced nothing, whatever the headline metric says.
Incumbent	The process actually in place today, measured on the same period and the same records	The relevance objection. This is the only control that speaks to whether anyone should change anything, and it is the one buyers care about most.
Strong conventional	A well-tuned established method: gradient-boosted trees, penalized regression, BM25, ARIMA or exponential smoothing, a rules engine written by a domain expert	The complexity objection. It answers whether the new architecture earns its operating cost, its retraining burden, and its explainability penalty.
Ablated self	The proposed system with one component removed at a time	The attribution objection. Without it, a gain from better data cleaning gets credited to the model and the next team inherits a false lesson.

The incumbent control is the one most often skipped, for an understandable reason: measuring it is real work with no modeling in it. It means instrumenting a manual process, pulling a year of dispositions, and agreeing with the operations lead on what counts as a correct outcome. Skip it and the report answers a question nobody asked. A 0.91 AUC against a random ranking is a statement about the dataset. A 0.91 against the queue the analysts run today is a statement about the program.

The record of what happens when the control is weak

This is not a theoretical worry, and the evidence for it is unusually blunt. Several fields have run the audit and published what they found.

Recommendation. Ferrari Dacrema, Cremonesi and Jannach took 18 neural recommendation methods published at top conferences and tried to reproduce them. Seven could be reproduced with reasonable effort, and of those seven, six could often be outperformed by comparably simple heuristic methods such as nearest-neighbor and graph-based approaches. The paper won the best long paper award at RecSys 2019, which tells you the field agreed with the diagnosis.

Baseline tuning. Rendle, Zhang and Koren looked at the baseline numbers that had been carried through years of publications on the MovieLens 10M benchmark. Setting up a plain matrix factorization baseline carefully, they beat the reported baseline results and also outperformed several newly proposed methods. Their conclusion is the one to take away: empirical findings are questionable unless they come from standardized benchmarks where the community has tuned the baselines extensively.

Metric learning. Musgrave, Belongie and Lim examined a decade of deep metric learning papers claiming large accuracy gains, found flaws in the experimental methodology, and showed that actual improvement over time had been marginal at best once the comparison was made fairly.

Forecasting. Makridakis, Spiliotis and Assimakopoulos compared ten machine learning methods against eight statistical ones on 1,045 monthly series from the M3 competition. The machine learning methods were dominated across the accuracy measures used and across every horizon examined, at considerably higher computational cost. The line that should stay with anyone building a forecasting system: Naive 2, a seasonal random walk, was more accurate than half of the machine learning methods.

Clinical prediction. Christodoulou and colleagues screened 927 studies and analyzed 71 that compared machine learning with logistic regression for binary clinical outcomes. The review's title is its finding: no performance benefit of machine learning over logistic regression for clinical prediction models.

Tabular data. Grinsztajn, Oyallon and Varoquaux built a 45-dataset benchmark with a methodology that budgeted hyperparameter search honestly for both sides, and found tree-based models still at the state of the art on medium-sized tabular data, before counting their speed advantage.

Retrieval. The BEIR benchmark evaluated retrieval models across a heterogeneous set of tasks and found BM25 to be a strong zero-shot baseline, with dense retrieval models frequently underperforming it out of domain despite being more computationally efficient.

The pattern is not about any one technique. In each case the newer method looked better against the baseline its authors chose and stopped looking better against the baseline someone else chose. The variable that moved was never the model.

Tuning asymmetry is how most comparisons get rigged without anyone deciding to

Very few weak baselines are dishonest on purpose. They are the arithmetic outcome of where the effort went. A team spends three months on the proposed architecture: feature engineering, learning-rate schedules, augmentation, ensembling, and several hundred hyperparameter configurations. The baseline is a library call with default parameters, run twice, the Thursday before the report is due. The measured gap is real, and it measures attention, not method.

The fix is procedural and it costs almost nothing to adopt. Give both arms the same search budget and say in the report what that budget was: number of configurations, search strategy, wall-clock or compute spent, and who ran it. When the budget cannot be equal, state the ratio. A reader who knows the proposed system got 400 configurations and the baseline got 12 can price the claim themselves, and will trust the rest of the document more for having been told.

Two related asymmetries do the same damage more quietly. Feature asymmetry gives the deep model a rich engineered feature set and the regression raw columns, so a comparison between feature sets gets reported as a comparison between model families. Preprocessing asymmetry tunes imputation and outlier handling against one arm's failure modes and hands it unchanged to the other. Fix the data preparation first, freeze it, then run the arms.

The ordering is a judgement about what survives a reviewer, not a scoring rubric. The fourth row ranks low for a specific reason: a number lifted from another study was produced on a different split, a different label definition, and often a different version of the same nominal dataset, so the comparison has no shared denominator. The bottom row ranks lowest because improvement against your own prior checkpoint is a development log, not evidence, and a program office reads it that way.

A baseline is a system, not a model

The incumbent is almost never an algorithm. It is a person with a queue, a set of rules accumulated over a decade, a threshold somebody tuned by hand in 2019, an escalation path, and a body of judgement that never got written down. Comparing a model against that fairly means comparing workflows, and the comparison has to be scoped honestly in both directions.

Honest in the incumbent's favor: it is allowed to use information the model never receives. A claims examiner who phones the provider is using a channel that is not in the feature set, and pretending otherwise inflates the model's apparent lead. Honest in the model's favor: the incumbent has a throughput ceiling, a queue that grows, and an error profile that changes at 4:30 on a Friday. State the operating point. A system that matches human accuracy at twenty times the throughput is a strong result and should be reported as a throughput result, not dressed up as an accuracy result it did not earn.

The practical form is a comparison at a fixed workload: hold constant what the organization can spend, in reviewer-hours or dollars per thousand records, then count correct outcomes at that spend. That framing survives contact with a program manager, because it is the one they already use for everything else.

If the split is wrong, both arms are wrong and the flexible one wins

A baseline comparison inherits every defect in the evaluation split, and the defects are not neutral between arms. Leakage inflates the more flexible model more, because a high-capacity method is better at finding and exploiting an artifact than a penalized regression is. The comparison then reports the wrong winner with perfect internal consistency.

Kapoor and Narayanan surveyed fields that had adopted machine learning and documented leakage in 17 of them, affecting 294 papers, and organized the failure into a taxonomy of eight types running from textbook errors to open research problems. The everyday versions are the ones to check first: a random split on data with repeated subjects, so the same patient or the same vehicle appears in train and test; a temporal split that leaks the future through a feature computed over the whole period; a preprocessing step fit on the full dataset before splitting; and duplicate records across the boundary.

Then there is the artifact that is not leakage in the technical sense but ruins the comparison the same way. Zech and colleagues, working across 158,323 chest radiographs from three institutions, found that convolutional networks could identify the source hospital system from an image with 99.95 percent and 99.98 percent accuracy for two of the sites. Where disease prevalence differs by site, a model can score well by learning the site. Geirhos and colleagues named the general case shortcut learning: decision rules that do well on the standard benchmark and fail to transfer to more demanding conditions. A simple baseline usually cannot represent the shortcut, so it looks worse in the lab and does better in the field. That is a control condition telling you something true, and it is worth listening to. We have written separately on testing a model on somebody else's instrument, which is the strongest available defense against this class of error.

The separation has to be larger than the noise you did not measure

A gap of half a point between two arms is not a finding unless you know what half a point of run-to-run variation looks like on that dataset. Bouthillier and colleagues modeled the whole benchmarking process and showed that variance from data sampling, parameter initialization, and hyperparameter choice moves results markedly, which means a single-run comparison cannot separate a method effect from a seed effect. Run both arms several times, vary the sources of variation deliberately, and report the spread.

Then use a test that respects the fact that both arms saw the same records. Dietterich's 1998 study of approximate statistical tests for comparing supervised classification algorithms remains the practical reference: McNemar's test had acceptable type I error and is the test to use when each algorithm can only be run once, while a 5x2 cross-validation test is slightly more powerful and directly measures variation from the choice of training set when ten runs are affordable. For areas under ROC curves computed on the same subjects, the DeLong nonparametric method for comparing correlated ROC curves is the standard, and it exists precisely because the naive comparison ignores the correlation.

Metric choice belongs in this section as well, because a baseline can be beaten on one axis and lose on the one that matters. Saito and Rehmsmeier showed that on strongly imbalanced data the visual interpretation of an ROC plot can be misleading about future performance, while a precision-recall plot reflects what fraction of positive predictions are actually correct. When the operating question is how much of a reviewer's day gets spent on false alarms, precision at the working threshold is the number, and reporting AUC instead quietly changes the subject.

What the federal record already requires of a comparison

Buyers do not have to invent this discipline. It is written into documents that already govern the work, and a supplier who structures a report around them is speaking the customer's language rather than a research group's.

Instrument	What it says	What it means for a baseline
FAR 37.601	Performance-based service contracts shall include measurable performance standards in terms of quality, timeliness and quantity, and the method of assessing performance against them	The standard, the method of assessment, and therefore the comparison point are contract terms. A number without a stated method of assessment is not yet a performance standard.
OMB Circular A-94 (revised November 9, 2023)	Analyses should consider alternative means of achieving program objectives, and for a capital asset decision the alternatives generally include doing nothing	The do-nothing alternative is the government's own name for the incumbent control. Net present value is calculated on incremental benefits and costs, which is a difference, which requires a base case.
OMB M-25-22 (April 3, 2025)	Directs performance-based acquisition of AI using solicitation strategies that specify outcomes and measurable success metrics, with performance validation and pre-award testing for high-impact AI and continuing testing rights	Testing happens before award and continues after it, on the government's terms. A comparison that only exists in the vendor's notebook does not satisfy it.
NIST AI RMF 1.0 (AI 100-1)	The MEASURE function applies quantitative, qualitative or mixed methods to analyze, assess, benchmark and monitor AI risk, with test, evaluation, verification and validation running across the lifecycle	Benchmarking is named as an ongoing function, not a one-time acceptance event. The baseline has to be reproducible months later by someone else.
FDA, Health Canada and MHRA guiding principles on good machine learning practice (October 2021)	Reference datasets are built with accepted best available methods so that clinically relevant, well-characterized data are collected and the limitations of the reference are understood, and clinical benefits and risks are used to derive clinically meaningful performance goals	The comparison point is a reference standard with stated limitations, and the target is set by consequence rather than by what the model happened to achieve.
Federal Policy on Research Misconduct (65 FR 76260)	Falsification includes changing or omitting data or results such that the research is not accurately represented in the research record	A control arm that was run and then left out of the write-up is an omitted result. A finding of misconduct also requires a significant departure from accepted practice, intent or recklessness, and proof by a preponderance of evidence, so the formal bar is high. The reporting expectation is not.

TRIPOD+AI, published in the BMJ in April 2024, does the same job for clinical prediction models and is worth borrowing outside medicine. One reporting standard covers both regression and machine learning methods, which removes the convenient ambiguity where a machine learning study is held to looser expectations than the statistical study it claims to beat.

A protocol you can write into a statement of work

None of this needs a methods committee. It needs eight decisions made before the first model runs, and written down where the customer can see them.

• Freeze the evaluation split before any modeling, and split by the unit that repeats: patient, vehicle, facility, case, or time.
• Name the three controls in advance: degenerate, incumbent, strong conventional.
• Fit every preprocessing step inside the training fold only, and apply the identical pipeline to all arms.
• Set one search budget and give it to both arms, then report the budget and who ran each arm.
• Pre-register the primary metric and the operating threshold, chosen from the cost of each error type rather than from the results.
• Run every arm multiple times across seeds and report the spread, not the best run.
• Report a confidence interval on the difference and the paired test used to produce it.
• Publish every arm that was run, including the ones that beat the proposed system.

That last item converts a methods list into a commitment, and it is the one to put in writing. A clause requiring delivery of all evaluated configurations and their results, whatever the ranking, costs a supplier nothing when the work is sound and is the most informative thing a buyer can ask for. Keep the evaluation harness separate from the modeling code as well, so the customer can re-run the comparison after delivery without the original team in the room.

When the simple method wins

Say it plainly, early, and with the numbers attached. Then keep going, because the interesting work usually starts at that point rather than ending there.

A simple method that wins on the primary metric has still left most of the problem unsolved. The regression that matches the neural network on AUC may be badly calibrated at the threshold where decisions are made. The rules engine that beats the classifier today was tuned on last year's distribution and has no monitoring on it. The keyword search that wins in the aggregate collapses on the queries whose vocabulary the index has never seen. Each of those is a real deliverable worth more to the customer than a marginal architecture change, because each addresses a failure the organization is exposed to now.

There is also a reputational asymmetry worth being clear-eyed about. A supplier who reports that the simple method won has demonstrated something no benchmark score demonstrates: that the evaluation was capable of producing a negative result. Every subsequent number from that supplier reads differently. Buyers who have been through one deployment where the demo number did not survive contact with production learn to look for exactly this signal, and the ones who have not been through it yet should read our notes on red flags in an AI benchmark before the next vendor meeting.

Common objections

Frequently asked questions