The result nobody budgets for
Somewhere in the third month of a modeling engagement, a junior engineer runs the boring comparison. Predict last week's value. Predict the majority class. Apply the seven business rules the operations team has been using since 2014. The boring thing scores within a point of the model that cost seven figures to build, and on two of the four customer segments it scores better. What happens next is decided almost entirely by what the contract said in month zero, and almost never by the engineering.
The failure is not that the baseline won. The failure is that nobody planned for it to be able to. A program with no baseline in its evaluation plan cannot distinguish a model that adds nothing from a model that adds a great deal, because it has no zero point on the scale. Every number it reports is an absolute measurement of something that only matters as a difference.
This is not a research-hygiene argument. It is a spending argument. A model that beats a well-built baseline by twelve points justifies its inference bill, its monitoring bill, its retraining bill and the governance overhead that follows a model into production. A model that beats a well-built baseline by half a point does not, and the half-point is often inside the error bar anyway. The baseline is the instrument that tells the two apart, and it costs a few engineer-days.
Build cost and run cost, typical mid-size deployment
Engineering effort to first defensible number, from our own project planning. Bar length is relative effort, not a measured statistic.
A weak baseline is worse than none
The most common baseline in a vendor deck is a strawman, and it is usually built without malice. Somebody fits a logistic regression with default settings on unscaled features with no missing-value handling, gets 0.61 AUC, and puts it in the comparison table next to the 0.84 the real model achieved. The chart looks decisive. It is decorative.
A defensible baseline gets the same treatment the candidate model gets: the same feature engineering budget where features are shared, the same hyperparameter search proportional to its parameter count, the same class balancing, the same threshold tuning on the same validation split. When a linear model is fit properly on well-prepared features, the gap to a deep model on tabular data is routinely a couple of points rather than twenty. Anyone whose comparison table shows twenty has probably measured their own preparation effort rather than the model families.
The reverse mistake is subtler and more expensive. Tuning the baseline on the test set to make the comparison look honest burns the test set for both arms. Fit and tune every arm on the same training and validation partitions, then score every arm once on a held-out set that neither arm has seen. If that is impossible, say so in the report and treat the numbers as optimistic for all arms equally.
The four baselines worth running on almost anything
The trivial predictor. Majority class for classification, the training mean or median for regression, last observed value for a time series. It costs an afternoon and it sets the floor a metric can reach by doing nothing. On a task with 4% positives, a model reporting 96% accuracy has reported the trivial predictor's score.
The existing process. Whatever humans or rules do today, scored on the same held-out data with the same metric. This is the only baseline the person paying for the work actually cares about, and it is the one most often skipped, because reconstructing historical decisions from operational records is unglamorous archaeology. It is also where the real business case lives.
The strong simple model. Regularized logistic or linear regression, or gradient-boosted trees, fit properly on the same features. On tabular data this is frequently the winner outright, and it deploys with a fraction of the operational surface.
The information-leakage check. A model trained on shuffled labels, and a model trained with the most suspicious feature removed. If the shuffled-label model scores above chance, the evaluation split is broken. If removing one feature collapses performance, that feature is probably a proxy for the answer, and it will not exist at inference time.
Half a point is not a result at the sample sizes people use
The arithmetic here settles most arguments before they start. Comparing two systems is far more expensive than measuring one, because the quantity being estimated is a difference and the variances add. Distinguishing a system that is truly 90% accurate from one that is truly 85%, at 95% confidence with 80% power, takes roughly 690 items per arm. Separating 90% from 88% takes about 3,800 per arm. Separating 90% from 89.5% takes on the order of 60,000.
So a deck reporting that the model beat the baseline 90.4% to 89.8% on a 400-item evaluation set has reported noise with a decimal point on it. The honest statement is that the two systems could not be separated at the sample size available, which is a finding, and an actionable one: either the evaluation set grows or the decision gets made on operating cost and maintainability instead.
Two techniques make small comparisons much sharper. Paired evaluation scores both systems on the same items and analyzes the per-item differences, which removes item difficulty from the variance and often cuts the required sample by half or more. Bootstrap resampling over the evaluation set gives a confidence interval on the difference itself, which is the number that belongs in the report. Report the interval, not the point estimate. A difference of 1.2 points with an interval spanning minus 0.8 to plus 3.2 is a very different purchase decision from 1.2 points spanning 0.9 to 1.5.
Items per arm needed to separate a model from a 90% baseline
Two-proportion comparison, two-sided 5% level, 80% power, unpaired. Arithmetic from the standard sample-size formula, not measured. Paired designs on the same items need substantially fewer.
Where the baseline usually wins, and why
Four situations produce a winning baseline often enough to plan for. Each has a mechanism behind it, and the mechanism tells you whether more modeling effort would change the answer.
The signal is thin. Some outcomes are close to unpredictable from the data on hand. Equipment failure with no sensor coverage on the failing component, customer churn driven by a competitor's pricing move nobody recorded, fraud by an adversary who adapts faster than the label lag. More model capacity cannot manufacture information that is not in the inputs. The right response is a data-acquisition project, not a bigger model.
The volume is small. Deep models earn their advantage on scale. With 3,000 labeled examples and forty features, a regularized linear model or a small tree ensemble is frequently the best achievable estimator, and everything more elaborate is fitting noise that will not reappear next quarter.
The rules encode real knowledge. A rule set that operations has been tuning against reality for a decade is a model. It was fit by gradient descent on human attention, and it has already absorbed the exceptions your training data may not contain. Beating it requires either information the rules do not use or interactions they cannot express.
The metric is the wrong shape. A model that is genuinely better at ranking can lose on accuracy at a badly chosen threshold. A model better on the 6% of cases that carry most of the cost can lose on an aggregate metric that weights every case equally. Before accepting that the baseline won, check that the metric matches the decision the system will actually inform.
What to report on the day the baseline wins
The report that survives review has four parts and takes about a week to assemble properly.
First, the comparison itself, with confidence intervals on the difference and per-slice results, not just aggregates. A model that loses on average and wins by nine points on the highest-cost segment is a candidate for a hybrid, not a cancellation.
Second, the ceiling analysis. How much headroom exists at all. Fit the strongest model you can afford with no deployment constraints, allow it to peek at features that would not be available at inference time, and see how far the number moves. If the unconstrained cheat model reaches 91% against a 90% baseline, the task is close to its information ceiling and no amount of engineering will change that.
Third, the cost of the win if there were one. Inference cost per thousand decisions, p95 latency, monitoring burden, retraining cadence, and the review overhead the model triggers. For a regulated deployment this is not small. Under SR 11-7, the Federal Reserve and OCC supervisory guidance on model risk management, a model requires documented development evidence, independent validation, ongoing monitoring and periodic review. NIST AI RMF adds the Map and Measure work: intended purpose documentation, TEVV planning, and characterized performance. A rule set carries a fraction of that load. The comparison is not 90% against 89.5%. It is 90% at a governance cost against 89.5% at almost none.
Fourth, the recommendation with a mechanism attached. Not "the model underperformed." Instead: ship the rules now, instrument these four missing data elements, revisit in two quarters when the labeled set triples, and here is the specific reason to expect the answer to change.
Write the baseline into the contract before anyone builds anything
The reason negative results get suppressed is structural. If the deliverable is "a model," a report saying the model should not ship reads as nonperformance, and the incentive runs one direction only. Fix the incentive in the statement of work.
| Contract element | What it should say | What it prevents |
|---|---|---|
| Baseline definition | Named baselines, built and scored before the candidate model, with the same feature and tuning budget | A strawman comparison assembled after the fact |
| Acceptance metric | A minimum improvement over the named baseline, at a stated operating point, with a confidence interval that excludes zero | An absolute threshold the trivial predictor already meets |
| Evaluation set | Buyer-assembled or jointly assembled, frozen on a named date, sized for a comparison rather than a point estimate | A number measured on data the vendor chose |
| Negative-result deliverable | A defined, separately priced report covering ceiling analysis and the data required to change the answer | The incentive to bury the finding and ship anyway |
| Slice reporting | Named segments, each with its own result, reported whether or not the aggregate passes | An average that conceals a segment where the model is dangerous |
The fourth row is the one that changes behavior. Price the negative result as a deliverable in its own right and the team building the model has no reason to avoid finding it. Federal buyers have supporting language available: OMB guidance on AI acquisition directs agencies to run independent evaluations on data the vendor cannot access, which presumes a comparison rather than a self-reported score. Commercial buyers can write the same terms into a master services agreement with no memo to cite.
The intellectual honesty is also the sales pitch
A firm that reports a losing model has told you something checkable about how it works. That report is more informative about future engagements than a deck of wins, because wins are reported by everyone and the reporting standard behind them is invisible.
There is a practical version of this too. A team that ships a rule set in month two, tells the client why, and specifies the three data elements that would make a model viable is usually back within the year on a better-defined project with better data. A team that ships a marginal model is back within the year on an incident.
How we run it
Baselines go in first, before feature engineering, before any model selection. The trivial predictor and the incumbent process get scored in week one, and those two numbers go at the top of every status report for the rest of the engagement so nobody loses track of what the work is being compared against. Evaluation sets get sized for the comparison at the start, using the arithmetic above, and if the available labeled data cannot support the comparison the client hears that in week one rather than month six.
Every comparison ships with a bootstrap interval on the difference and per-slice results. Every model that reaches a shipping decision comes with a written cost of ownership beside its accuracy number: inference cost, latency, monitoring, retraining and, in regulated settings, the SR 11-7 or NIST AI RMF documentation load it brings with it. When the baseline wins we say so, in writing, with the ceiling analysis and the specific conditions under which the answer would change.
Bottom line
A baseline is the zero point on the scale a model is measured against, and building a real one takes days against months for the model. Without it, a program cannot tell an expensive tie from a genuine improvement, and it will pay production and governance costs for both. Build the baselines first, size the evaluation set for a comparison rather than a measurement, report intervals on the difference, and price the negative result as a deliverable. On the day the baseline wins, that is the cheapest answer the program will ever get, and the only reason it hurts is that nobody wrote it into the plan.
Frequently asked questions
One built with the same feature preparation, the same tuning proportional to its parameter count, the same class balancing and the same threshold selection as the candidate model, fit and tuned on the same partitions and scored once on the same held-out set. Anything less measures preparation effort rather than model families.
It depends on the difference being detected, and the relationship is quadratic. Separating a model from a 90% baseline takes roughly 690 items per arm at a 5-point true difference, about 3,800 at 2 points, and around 15,000 at 1 point. Paired designs on identical items need substantially fewer.
No, provided the tie was measured properly. A ceiling analysis showing the task is near its information limit tells a program to stop spending on modeling and start spending on data acquisition, which is a more valuable finding than a marginal model would have been.
SR 11-7 requires independent validation with benchmarking against alternative approaches for models in scope. NIST AI RMF's Measure function presumes characterized performance against a defined reference. OMB AI acquisition guidance directs agencies to evaluate on data the vendor cannot access, which presumes a comparison rather than a self-reported number.
The incumbent process, scored on the same held-out data with the same metric. Trivial predictors set a floor and simple models bound the achievable, but the number that decides whether to spend is the improvement over what the organization does today.