Selection bias: when the people you need are missing from the data

Your database is a record of a filter, not a population

A lender has fourteen years of loan performance data. Every row is a loan that was approved, funded, and either repaid or not. The team trains a default model on it, and the model is excellent. It is excellent at one specific thing: predicting default among applicants the old underwriting rules already liked. The applicants those rules rejected have no outcome column, because they were never given the chance to produce one. The model has never seen a single one of them, and the model does not know that.

This is the shape of the problem in every domain we get called into. A hospital's readmission data covers patients who came back to that hospital. A manufacturer's failure data covers parts that were inspected. A ratings firm's default history covers issuers that were rated. A recruiting model's success data covers candidates who were hired. In each case the rows that exist were produced by a selection mechanism, and the mechanism was not random. It correlated with exactly the thing being predicted.

Statisticians call the severe version missing not at random. The probability that a row is absent depends on the value the row would have had. That is the hard case, because no amount of cleverness with the observed data identifies it. Missing completely at random costs you sample size. Missing at random costs you some efficiency and is fixable with covariates you hold. Missing not at random costs you the answer, and it looks identical to a clean dataset from the inside.

The reject inference problem, and why it generalizes

Consumer credit named this problem first and has argued about it longest, which makes it the best worked example even for readers who have never priced a loan. The task is to estimate default risk across the whole applicant pool while holding outcome labels only for the approved slice. The industry calls it reject inference, and the honest summary of sixty years of literature is that no method recovers the truth from approved data alone. Every one of them smuggles in an assumption.

Augmentation reweights approved loans to look like the full applicant distribution, which assumes rejects behave like the approvals that resemble them. Parceling assigns inferred outcomes to rejects from the score distribution, which assumes the score ordering holds outside the range where it was fit. Both assume the thing in question. What actually works is expensive and is the reason the good lenders are good: approve a small randomized sample below the cutoff, take the losses, and buy real labels in the region where you have none.

Now change the nouns. A predictive maintenance program has failure data only from instrumented units, and units get instrumented after somebody worries about them. A clinical decision tool has outcomes only for patients who were tested, and testing follows suspicion. A fraud model has confirmed fraud only where an investigator looked. Same structure, same non-answer, same fix: buy a small window of labels where the old gate said no.

What the failure actually looks like in production

Selection bias does not announce itself as poor accuracy. It announces itself as excellent accuracy in validation followed by a specific, patterned failure in the field, and the pattern is always the same: the model is wrong about the population that was underrepresented, and it is confident while being wrong.

That combination is what makes it dangerous. A model that knows it is uncertain hands the case to a human. A model trained on a filtered population has seen nothing contradicting it in the missing region, so its posterior there is narrow. It reports high confidence because it has no evidence of difficulty, not because it has evidence of ease. Calibration checks on a held-out set do not catch this, because the held-out set came out of the same filter.

The second signature is a widening gap between offline and online metrics that nobody can explain. Offline evaluation reuses the historical selection. Online traffic does not, especially when deployment widens the funnel on purpose. The moment it does, the live population stops matching the training population and the metrics separate. Teams blame drift and retrain. Retraining on data the model itself selected makes it worse, and now you have a feedback loop.

Coverage analysis: the check that should run before training

The first engineering move is not a modeling technique. It is an explicit comparison between the population in the training data and the population the system will be asked about, on every axis where they could differ. We build this as a table with three columns: the segment, its share of the training rows, and its share of the deployment population taken from an external reference.

The external reference is the part teams skip, and it is the part that does the work. Census ACS tables for demographic and geographic distribution. BLS QCEW for industry and establishment size. CMS claims summaries for clinical case mix. Your own top-of-funnel logs before any filter fires, which almost every company has and almost nobody joins against the training set. The reference does not have to be perfect. It has to be independent of the filter you are measuring.

Any segment where the training share is materially below the deployment share is a coverage gap, and the size of the gap sets the size of the problem. A segment at two percent of training rows and eleven percent of the deployment population is one the model will meet five times as often as it learned from. That number goes in a risk register and in front of a customer, and it is worth far more than an adjective.

Why the aggregate number hides it and slice floors expose it

A model at 93 percent overall can sit at 64 percent on a segment holding six percent of volume, and the aggregate moves by less than two points. Nothing in a standard evaluation report will show it. The aggregate is a weighted average over whatever the test set happened to contain, and the test set inherited the same filter as the training set, so the underrepresented segment is underrepresented in the evaluation too. The measurement apparatus has the same blind spot as the model.

The fix is to name the segments in advance, give each its own floor, and size the evaluation set so each clears the arithmetic above. That means oversampling thin segments relative to their natural rate and weighting back in the aggregate. It costs labeling budget and it is the only thing that works. A slice with 30 items cannot be validated, only gestured at.

Symptom in the field	What people usually blame	What is actually happening	The check that separates them
Accuracy drops after launch and keeps dropping	Concept drift	Deployment widened the funnel past the training population	Compare live input distribution to training distribution on the same features
High confidence on the cases humans override	Miscalibration	No contradicting examples exist in the training region	Calibration curve computed per segment, not pooled
Offline and online metrics diverge	Instrumentation bug	Offline evaluation reuses the historical selection	Score the model on a randomized holdout that bypasses the old gate
Retraining makes it worse	Bad hyperparameters	The model is now selecting its own training data	Trace label provenance: which rows entered because the model said so
One region or product line is always wrong	Local data quality	That segment was never in the source system at volume	Coverage table against an external population reference

Techniques that help, and the assumption each one buys

No method recovers information that is not in the data. Every technique below trades an untestable assumption for an estimate, and the engineering discipline is stating which assumption you bought.

Inverse propensity weighting. Model the probability that each row was selected, then weight observed rows by the inverse of that probability. It works when selection depends only on variables you observe, and it fails silently when it does not. Two traps: propensities near zero produce enormous weights that a handful of rows then dominate, so trim or stabilize them; and the effective sample size after weighting is often a fraction of the row count. Compute it and report it rather than quoting n.

Heckman correction. The two-stage approach from econometrics fits a selection equation and carries a correction term into the outcome equation. It is identified properly only when there is an exclusion restriction, meaning a variable that drives selection and does not drive the outcome. Underwriting policy changes, branch openings, staffing schedules, and eligibility cutoffs are the usual candidates. Without one, the correction rests on a functional form assumption and can be worse than doing nothing.

Randomized exploration. Deliberately accept a small share of cases the current rule rejects, or route a small share of traffic past the filter, and label what comes back. This is the only technique that adds information rather than assumptions. It costs real money in the short run and it is what separates firms whose models keep improving from firms whose models keep confirming themselves. One to five percent of volume is usually enough, and the cost should go in the business case, not be discovered later.

Bounds instead of a point estimate. When you cannot identify the answer, bound it. Manski-style partial identification asks what the metric would be if every missing row took the worst plausible value, and what it would be at the best. If the decision is the same across the whole interval, you can act without resolving the bias. If it flips, you have learned that the analysis cannot support the decision, which is a real finding and one buyers respect.

Where the frameworks already require this

If the system touches a regulated decision, the obligation is not aspirational. SR 11-7, the Federal Reserve and OCC guidance on model risk management, has required since 2011 that validation assess "the quality and appropriateness of the data" and whether a model developed on one population is being applied to another. The document is short and worth reading even if you are nowhere near a bank, because it is the clearest statement in regulation of what a validation is supposed to do.

The NIST AI Risk Management Framework puts the same requirement under Map and Measure. Map 2.3 covers scientific integrity and test, evaluation, verification and validation practice. Measure 2.11 covers fairness and bias evaluation, which is a subgroup performance requirement in different words. ISO/IEC 42001 requires documented data provenance and quality criteria for the data behind each AI system. A coverage analysis is the artifact all three describe.

For federal work, the OMB memoranda on AI acquisition and use tightened this further in 2025 by requiring independent evaluation on agency-held data that the vendor cannot access, and by requiring an impact assessment on high-impact use cases that documents the quality and appropriateness of the data. A vendor asked to demonstrate segment coverage is being asked for something the buyer needs anyway. Producing it without being asked is how a technical team makes an evaluator's job easy, and easy is what gets picked.

What this costs and how long it takes

A coverage analysis on an existing dataset is a two to four week engagement for a small team: pull the training population, obtain an independent reference, define the segments, join, compute the gaps, write the memo. The output is a segment table, a risk register, and a recommendation per gap. It is the cheapest work on this page, because it uses data you already hold.

Building the labeled evaluation set is the expensive part. Sizing several thin segments to a few hundred items each, with expert review where the label needs judgment, runs six to twelve weeks and consumes most of the phase budget. Randomized exploration is a policy change rather than a project: it costs whatever the accepted bad cases cost, it runs continuously, and it has to be argued into the operating plan rather than a statement of work.

Set against that, finding out in production costs a rollback, a remediation, an audit response if the domain is regulated, and a customer who now asks for evidence on everything. That asymmetry is why the check comes first.

What a defensible answer looks like

When someone asks whether the model works, the answer that survives scrutiny has four parts, and each part is a document rather than a claim. First, the population the training data actually represents, stated against an external reference with the gaps quantified. Second, per-segment performance with a confidence interval on each, on an evaluation set sized so those intervals mean something. Third, the named assumption behind every correction applied, and what would have to be true for it to hold. Fourth, the monitoring that detects the live population moving away from the training population, with a threshold that triggers something specific.

None of that requires a novel method. It requires deciding to look before shipping, and a team that has seen the failure enough times to know where to look. The work is unglamorous and the finding is often unwelcome. It is also the difference between a model that holds up under audit and one that produces a very good number until the moment it matters.

Bottom line

Historical data records the output of a filter. When the filter correlated with the outcome, the model inherits the filter's judgment and reports it as its own, confidently, on exactly the cases the filter got wrong. Coverage analysis against an independent reference finds the gap. Per-segment evaluation, sized properly, measures it. Randomized exploration is the only thing that closes it. Everything else moves an assumption from one place to another, and the professional obligation is to say which assumption you moved and where you put it.

Frequently asked questions