Feature engineering that survives contact with reality

The pattern is familiar enough to name. Strong offline numbers, a launch, a first month well below what the validation set promised, then a week of arguing about regularization. Regularization is almost never the answer. In the systems we get called into, the data reaching the model at inference time is not the data it was fit on, and the difference came from a join, a default value, a timestamp, or a second implementation of the same feature written six weeks later by someone reading the first one.

Feature engineering gets taught as a creative act: a clever ratio, a rolling window, a well-chosen encoding. That part matters less than people expect. What decides whether a model earns its operating cost is unglamorous and mostly about time, identity and defaults. This is the checklist we work through, in order, with the test that catches each failure.

Leakage, and the four kinds that survive code review

Target leakage is the one everybody knows: a column populated only because the outcome already happened. A refund_amount field in a refund model, a collections_agent_id in a default model, an account status of closed in a churn model. These get caught because somebody eventually asks what the column means.

The other four survive review because each looks like ordinary data work.

Temporal leakage. The feature is computed from a window that includes data which did not exist at the moment of prediction. A thirty-day aggregate anchored on the label date instead of the decision date pulls the future into the row. This is the most common leak we find, and it comes from writing the training query in the warehouse, where every table is complete, then serving from a system where it is not.

Group leakage. One entity appears on both sides of the split. A customer with forty sessions gets thirty in train and ten in test, and the model learns the customer rather than the behavior. Random train_test_split guarantees this on any event-level data with repeated entities. The fix is GroupKFold keyed on the entity, a time-based split, or both.

Preprocessing leakage. A scaler, imputer or encoder fit on the full dataset before the split, so every fold sees statistics computed from its own holdout. It inflates results by a small amount that is hard to attribute, which is what makes it dangerous. Put every fitted transform inside a scikit-learn Pipeline or ColumnTransformer, and keep the fitted object with the model artifact rather than reconstructing it in the serving code.

Duplicate leakage. Reprocessed batches, retried writes and near-identical events under different ids put the same record on both sides of the split. Any pipeline with at-least-once delivery produces these eventually. Deduplicate on a business key before splitting, not after.

Leak	What it looks like	Where it comes from	The test that catches it
Target	One feature carries most of the importance and nobody can explain why	A column written by the process that produces the outcome	Ablate it and retrain. If performance collapses, read the column's write path.
Temporal	Offline results well above anything the model achieves on live traffic	Window anchored on label time instead of decision time	Recompute the training set through the serving path at the historical decision timestamp and compare.
Group	Cross-validation scores with unusually low variance across folds	Random splits on event rows with repeated entities	Count entities appearing in more than one fold. It should be zero.
Preprocessing	A small, consistent gap between cross-validation and a held-out set	Fitting transforms before splitting	Move every transform inside the pipeline, rerun, compare.
Duplicate	Near-perfect performance on a subset of rows	Retries, reprocessed batches, soft deletes	Hash the business key, count collisions across the split.

Point-in-time correctness is most of the job

Every training row needs its features as they stood at the moment the decision would have been made, not as they stand when you run the query. That is easy to agree with and hard to implement, because most operational schemas are built to answer "what is true now" and destroy the history you need.

The usual offender is a mutable dimension table. A customers row has plan_tier, and an upgrade overwrites it in place. Train on that column and every historical row carries today's plan, including rows from before the upgrade. The model learns that premium customers behave in a way they only began behaving after they upgraded. Nothing errors. Offline numbers improve.

Three mechanics fix this, all ordinary engineering. Capture history at the source with slowly changing dimensions, which dbt snapshots implement directly. Store the table in a format with snapshot semantics, which Apache Iceberg and Delta Lake both give you, so a query can read it as of a timestamp. Or write an append-only event log and derive state from it, which is more work up front and the least fragile.

Training and serving disagree for four reasons

Skew is not a mysterious property of production. It has a short list of causes, and on any given system you can go and check each one in an afternoon.

Two code paths. The offline feature is a SQL expression in the warehouse. The online feature is Python in a request handler. They agree the day they are written. Then someone changes a rounding rule, or a window to include today, on one side only.

Two data sources. Offline reads the warehouse, which has been deduplicated, backfilled and repaired. Online reads the operational database or a cache, which has none of that. Same column name, different content.

Two time semantics. The batch job computes a thirty-day window over complete days. The service computes one ending mid-afternoon, over a partial and still-arriving day. Counts run low at serve time for structural reasons, and the model reads low counts as signal.

Two default policies. The offline join misses and produces NULL, which the training pipeline imputes with a median. The online lookup misses and the service returns 0 because that was the simplest thing to write. The single most common production skew we find is a disagreement about what absence means.

One definition, one code path, and a parity test when you cannot have one

The structural fix is to define each feature once, in version control, as code rather than as two dialects of the same intent. Feature stores like Feast and Tecton exist mostly to enforce this, and at smaller scale a warehouse plus a disciplined library gets you a long way. What matters is that there is one definition and it has an owner.

Name features so the definition is legible without opening the code. orders_completed_30d_count gives a reader the event, the state filter, the window and the unit. orders gives them nothing, and in six months two teams will mean different things by it.

Sometimes one code path is genuinely impossible. A batch job in Spark and a low-latency service in Go will not share an implementation. When that happens you owe a parity test. Sample real production requests, compute every feature both ways, and assert equality within a tight tolerance. Run it in continuous integration on every change to either side, and nightly against yesterday's live traffic. A mismatch rate above zero is a bug with a known cost.

• Two implementations of the same feature with no automated comparison between them
• Feature names with no window and no unit, so nobody can tell 30 days from 30 events
• fillna(0) applied across a whole frame because the model would not fit otherwise
• Transforms fit on the full dataset, then reimplemented by hand in the serving code
• Joins on ingestion date because the event timestamp was inconvenient
• Derived features stored without the raw inputs, so no value can ever be recomputed or explained
• A backfill written as a separate script from the forward path, with different logic in it

Labels arrive late, and that shortens every honest evaluation window

Feature timing gets attention. Label timing gets much less, and it decides how much of your data is usable. Card disputes can land months after the transaction, an insurance claim settles on its own schedule, a subscription is not churned until a renewal date passes, and collections outcomes resolve over quarters.

Every label therefore has a maturity period, and rows younger than it are not labeled, they are provisionally labeled. Include them and the recent past looks clean, because only the fast-settling outcomes have arrived. The model learns that recent activity is safe. It is not, it is unresolved.

Two practices handle this. Record a label-available timestamp beside the event timestamp, so evaluation can filter on maturity rather than intuition. And end the test window at least one maturity period before today, accepting that the most recent evaluation is always somewhat stale. That cost is smaller than shipping on numbers computed from half-arrived labels.

Missing is not zero, and a default is a decision

There are at least three reasons a value is absent, and each calls for different handling. The attribute does not apply to this entity, it applies but has not been observed yet, or the upstream system failed and the value should exist.

The third is the one that hurts, because a failed feed presents as a legitimate value. A join that silently returns nothing becomes a zero, the zero is in range, and the model scores it. Nothing alerts. Catch it by alerting on null rate and freshness per feature rather than inferring breakage from model output, which moves last.

For numeric features where zero is meaningful, a zero-fill is a factual claim about the world. days_since_last_login filled with 0 says the customer logged in today, the opposite of what absence usually means. Prefer an explicit missingness indicator alongside an imputed value, so the model can learn that absence carries signal. XGBoost and LightGBM handle missing values natively and learn a default direction per split, usually better than anything you would impute by hand.

Whatever the policy, it has to be one object shared by both paths. Fit the imputer, serialize it with the model, and load it in the service. A default that lives in serving code as a literal will diverge from the training default, and the divergence will be invisible until somebody diffs the two.

Cardinality is where features quietly stop working

Categorical features degrade in a way continuous ones do not. They keep returning values, and the values keep meaning less.

Unseen categories. New merchants, device types, SKUs, campaign codes. The encoder needs an explicit out-of-vocabulary bucket, and that rate needs to be a live metric. A rate climbing from low single digits toward double digits is a retraining trigger, and it moves long before accuracy visibly does.

Target encoding done carelessly. Replacing a category with its mean outcome is a strong technique and a leakage engine if that mean is computed on the same rows the model trains on. Compute it out of fold, with smoothing toward the global mean for rare categories. Done in one line over the whole training frame, it is leakage wearing a respectable name.

Identifier features. Raw user_id or device_id as a category memorizes individuals. On a group-leaked split it looks excellent. On new entities, which is the production case, it contributes nothing and consumes capacity.

Growth as a capacity problem. An embedding table or hash space sized for today's cardinality is a commitment. Hashing bounds memory by trading it for collisions, and the collision rate rises as cardinality grows. Track distinct counts per feature over time and treat the trend as a planning input.

The latency budget decides what a feature is allowed to be

A feature that takes 400 milliseconds to compute is not a feature for a synchronous decision, however predictive it is. The budget belongs to the request, measured at p99 rather than the mean, because the tail is what users and downstream timeouts experience.

Work backwards. If the product allows 150 milliseconds end to end and the model takes 20, then retrieval, transformation and serialization share what is left with network overhead. Below is how we allocate a 50 millisecond retrieval budget. The numbers move per system, and writing them down at all is most of the value, because it turns an argument about elegance into an argument about milliseconds.

Four classes of feature fit that budget differently, and the class is a design decision to make before the feature is written rather than discover during load testing.

Class	Example	Freshness	Behavior when the source is down
Precomputed batch	90-day spend percentile per customer	Hours to a day	Serves the last written value, which usually works and always needs a staleness metric
Streaming window	Events in the last five minutes for this account	Seconds	Window silently truncates, so counts read low; needs a watermark lag alarm
Request-derived	Amount relative to the account's stored average	Exact	Fails only if the request itself is malformed
Third-party call	External enrichment or geolocation lookup	Exact, at a cost	Needs a timeout and a fallback that also exists in the training data

That last row is worth dwelling on. A synchronous external call needs a timeout, and a timeout needs a fallback value. If that fallback never appeared during training, the first time the vendor has a bad afternoon your model sees a distribution it has never seen, at the moment traffic is already degraded. Inject the fallback into training data at a realistic rate.

Backfills are a separate program from the pipeline

Every feature you add needs history, and generating it is not the same job as running the pipeline forward. Treating a backfill as "the same code with an older date parameter" is how leakage gets introduced after launch rather than before it.

Writes must be idempotent, keyed on entity plus feature plus event time, so a rerun converges instead of double-counting. The backfill must read historical dimension state rather than current state, the same as-of discipline as above and the easiest place to lose it, because current state is right there and faster. And the compute cost deserves an estimate before anyone commits to a date.

What we check before a feature ships

Below are the weights we use when reviewing a feature set, fixed before we look at anything so the review is not steered by whatever we find first. Adjust them to context: long label maturity moves label timing up, a strict latency endpoint moves serving cost up.

And the artifacts that travel with a feature before we consider it done:

• A definition in version control, with an owner and a name carrying window and unit
• An as-of join test proving no training row used data unavailable at its decision time
• A parity test comparing offline and online values on sampled real requests, running in CI
• Assertions on null rate, range and distinct count, enforced in the pipeline
• A documented default per failure mode, shared by both paths as one serialized object
• A p99 latency measurement taken with a cold cache, not a warm one
• An idempotent backfill that reuses the forward code path
• A lineage query that answers why this entity had this value at this timestamp

What to monitor once it is live

Monitor features, not only the model. Prediction distribution and accuracy move late, after the damage, and by then the diagnosis is a long one. Feature-level signals move first and point at an owner.

The set worth instrumenting is short. Null rate per feature. Freshness lag from event time to available time, which is the best early indicator of an upstream problem. Out-of-vocabulary rate for categoricals. Distinct-count trend. A population stability index or Kolmogorov-Smirnov statistic against a fixed reference window, per feature rather than only on the score. Mismatch rate from the shadow comparison. Retrieval p99.

Alert on the operational ones first: freshness and null rate have owners and runbooks. Compute distribution statistics from day one and page on them only once you know their normal variation, otherwise they generate noise that trains everyone to ignore the channel.

The cost of finding it late

These are the planning multipliers we use when deciding how much test work a feature deserves. They are estimates, not measurements, and they make one point: the cost of a defect is dominated by when it is found, not by how hard it is to fix.

The curve is steep because a late discovery is never only a code fix. It is a fix, a backfill, a retrain, a revalidation, and an explanation to everyone who consumed outputs while the input was broken. The engineering is the cheap part.

The order we would do the work

Step six usually produces an uncomfortable meeting, because the corrected offline number is lower than the original. It is also the first number anyone has that predicts production behavior, which makes it the more useful of the two.

Bottom line

Feature engineering that survives production is mostly about time and agreement. Time, because every value has a moment it became knowable and training has to respect it. Agreement, because the offline and online paths make claims about the same quantity and nothing forces them to match unless you build the thing that does. Get both right and ordinary features perform close to their offline promise. Get them wrong and no model selection recovers the gap, because the model was never what broke.

Common objections

Frequently asked questions