Two accuracy numbers that never match
The backtest said 8 percent error. Six weeks after go-live the weekly report says 17. Nobody changed the model, nobody changed the features, and the data scientist who built it can rerun the notebook and get 8 again. This is the most common way a forecasting project fails, and the cause is almost never the model. It is the machinery between the notebook cell and the number a planner acts on at 8am, and that machinery is built out of a small number of specific, findable mistakes.
You are probably here because
- The backtest number and the number in the weekly report are far apart, and nobody can point at what is responsible.
- Yesterday’s forecast is gone, because the daily job overwrites the same table, so real accuracy for last month cannot be computed at all.
- A planner asks what the forecast said on the third, and there is no way to answer.
- A refit went out on a schedule, and nobody can say whether the new version is better or worse than the one it replaced.
The sections below on point-in-time correctness and on storing forecasts as facts address all four, because all four come from one root cause: nothing in the pipeline records what the model saw and what it said at the moment it ran.
This article is about that machinery. It assumes you have already decided what to predict, at what granularity, and how far ahead. Those are the expensive decisions and they belong to a different conversation. What follows is everything after: the schemas, the evaluation loop, the retrain policy, the monitoring you can run before you know whether you were right, and the contract with whoever consumes the number.
Four structural differences separate a backtest from a running system, and each one has an engineering fix.
The backtest reads one table, assembled after the fact. Production reads six or eight systems in whatever state they happen to be at 05:40, with a warehouse job that may or may not have finished.
The backtest sees the final version of history. Production sees what had arrived by run time, which is a different and worse dataset.
The backtest fits once. Production refits on a cadence, and every refit is a small unreviewed deployment.
The backtest scores against actuals. The business scores you on whether the plan built from the forecast was better than the plan they had before.
Backtest-to-Production Gap — Triage Order
Our triage order when a live number diverges from a backtest. A priority ranking, not a measured distribution.
Point-in-time correctness is most of the fix
Operational data gets rewritten. Invoices land five days late and are backdated to the transaction. Returns post against the original order. Someone recategorizes a product line and the warehouse applies the new category to all of history, including the two years your model trained on. An ERP close adjusts last month after the fact. None of this is anybody's fault, and all of it means the table you trained on is not the table that existed on the day you would have had to predict.
The consequence is subtle and expensive. A model trained on restated history has been shown a cleaner, more complete, more internally consistent world than the one it will operate in. Its error at inference is higher, and the size of the increase is roughly the size of the restatement. On revenue and demand data we have seen the effect run in the mid single digits of error percentage, which is exactly the size of the gap that gets blamed on the model.
There is a direct test. Pick a date two weeks in the past, reconstruct the feature vector your pipeline would have produced on that morning, and diff it against the vector you actually served. If your pipeline does not store the served vector, that is the first thing to build; it is a hash and a few columns and it makes every later argument decidable.
The warehouse-side fix is well trodden. Facts get an ingestion timestamp separate from their event timestamp, so you can filter to what had arrived. Dimensions get slowly-changing type-2 treatment with valid_from and valid_to, so a product's category on the day of the sale is recoverable. dbt snapshots do the second, and table formats with time travel, Delta Lake and Apache Iceberg among them, give you the first at the storage layer. Feature platforms such as Feast and Tecton exist mostly to make the point-in-time join correct by default, which tells you how often teams get it wrong by hand.
If none of that exists yet, do not start with the model. A forecasting system on a warehouse with no vintage is a system whose accuracy cannot be measured, and one whose accuracy cannot be measured cannot be improved.
Store forecasts as facts, not as a current-state table
The most common structural mistake we find is a serving table with one row per series and horizon that the daily job overwrites. It is convenient for the dashboard and it destroys the ability to answer any interesting question. You cannot compute real production accuracy, because yesterday's prediction is gone. You cannot attribute a bad month to a model version. You cannot tell a stakeholder what the system said on the third, which is the question they will ask.
Write forecasts to an append-only table instead, keyed on the run. A rerun is a new row, never an update. Downstream consumers read a view that selects the latest run per target, so nothing about their query changes.
| Column | What it holds | Why it earns its place |
|---|---|---|
| series_id | SKU, location, account, node, or the composite key | The grain the model predicts, which is not always the grain a decision is made at |
| target_ts | The period being predicted | Join key to actuals when they eventually arrive |
| horizon | Steps ahead, derived from target minus origin | Accuracy is a function of horizon; reporting a single blended number hides the failure |
| run_ts | When the batch executed | Makes reruns additive instead of destructive and gives you revision history for free |
| origin_ts | The data cut-off the run was allowed to see | Distinct from run_ts whenever a source is late; this is the column that proves no leakage |
| model_version | Artifact hash or semantic version | Error attribution and the shadow comparison below both depend on it |
| yhat, q10, q50, q90 | Point forecast and quantiles | Inventory and staffing decisions are asymmetric; a point forecast cannot express that |
| feature_hash | Digest of the served feature vector | Turns "the pipeline must have changed" from an argument into a lookup |
The storage objection comes up every time and it does not survive arithmetic. Fifty thousand series with a ninety-step daily horizon is 4.5 million rows a day, about 1.6 billion a year. Those are eight narrow numeric and key columns; in a partitioned columnar format they compress to roughly 20 to 60 bytes a row, so the annual footprint is tens of gigabytes. That is a rounding error against the cost of not being able to prove what your system predicted.
Backtesting that matches the serving path
A single train/test split is not an evaluation of a forecasting system. It is one sample from a distribution with high variance, and which side of the mean you landed on is luck. Rolling-origin evaluation is the working standard: step an origin forward through history, forecast the full horizon from each origin, and score across all of them. Report the distribution, not just the mean, because a system with a good average and three catastrophic origins is a system that will lose someone's trust in a specific month.
Two details decide whether the backtest tells the truth.
Refit inside the loop. If the model is fit once on all history and then evaluated at earlier origins, it has seen the future. This is the single most common leak in forecasting code and it is invisible in the output, because the numbers just look good. Fitting at every origin is expensive, so a common compromise is refitting every fourth or eighth origin and stating the cadence in the report. That is defensible. Fitting once and not saying so is not.
Put a gap between the training cut-off and the origin. If invoices land five days late, then at origin t the model has complete data only through t minus five. A backtest that trains through t is scoring a system with five days of information it will never have. The optimism is not small, and it grows with the freshness of the signal you are relying on.
Budget the compute honestly. Three years of weekly origins is 156 fits. A pooled gradient-boosted model over a large panel might take four minutes a fit, so a full rolling backtest with refits at every origin is around ten hours. That is a real number to plan around, and it is the reason most teams evaluate on a quarterly schedule rather than per commit. Cache aggressively, keep the feature build separate from the fit, and make the backtest reproducible from a config file so a disagreement about results is resolvable.
The retraining decision
Three retrain policies are defensible: a fixed cadence, a triggered refit, or a frozen model with a scheduled human review. What is not defensible is the fourth option most systems drift into, which is a refit on a cron that ships straight to serving with nobody looking at it.
The rule we hold to is that a refit is a deployment. It gets a version, it gets written to the forecast table alongside the incumbent, and it runs in shadow for two to four weeks before it replaces anything. Because both versions write rows keyed on model_version, the comparison is a query rather than a project. When the challenger wins on the same targets over the same origins, promote it and keep the row history. When it loses, you have the evidence, which is more useful than the promotion would have been.
Retrain Triggers — How Often Each Justifies a Refit
The last row is the one that causes trouble. A single bad week is usually variance, and refitting on it is how a stable system becomes a twitchy one.
Latency and cost are decided by the shape of the batch
Architecture here follows from arithmetic more than from taste. Fitting a per-series statistical model, an ETS or ARIMA or a Prophet-style decomposition, takes somewhere between 0.2 and 2 seconds depending on history length and seasonality. Across fifty thousand series that is 3 to 28 CPU-hours per run, which parallelizes onto a large box in minutes and is entirely workable. What it costs you is elsewhere: the run time scales linearly with the catalog, and you now version and monitor fifty thousand artifacts instead of one.
A global model, one learner trained across the pooled panel with lag, rolling-window and calendar features, trains in minutes, predicts a full horizon set in seconds, produces a single artifact, and borrows strength across series so that a new SKU with six weeks of history inherits behavior from its neighbors. That is the practical reason gradient-boosted trees on lag and calendar features became the production default, and it is also what the top accuracy entries in the M5 competition looked like. The M4 competition before it was won by a hybrid of exponential smoothing and a recurrent network, and across both, simple statistical combinations finished close enough to the leaders that anything you build should be scored against them.
Deep sequence models earn their place under specific conditions: many related series, and covariates whose future values you know, such as prices, promotions and holidays. N-BEATS, DeepAR and temporal fusion architectures all handle that setting well. The operational cost is a GPU in the batch path and a much wider surface for the training-serving mismatch described above, because more feature engineering happens inside the model. Take that trade when the covariates are real. Do not take it for a catalog of two hundred series.
Real-time serving is claimed far more often than it is needed. If a page or an API needs a forecast, precompute it in the batch and serve from a key-value store; the read is a millisecond and the failure modes are ones you already understand. Reserve on-demand inference for cases where the input genuinely arrives at request time and materially changes the answer.
Send it over and we will tell you what we would change.
Email your forecast table schema and the backtest script that produced your go-live accuracy number to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comMonitoring before the truth arrives
Forecasting has a monitoring problem that classification does not: the label is delayed by exactly the horizon. A thirteen-week-ahead forecast cannot be scored for thirteen weeks. A monitoring plan built only on error metrics is therefore blind for a full quarter, which is more than enough time to lose the room.
Four classes of signal are available immediately, and together they catch most of what actually breaks.
Input health. Freshness per source, row counts against an expected band, null rate on driver columns, and unit and timezone assertions. Most bad forecasts are bad inputs, and the input check fires hours earlier than anything downstream.
Feature distribution. Compare the served features against the training window on the twenty features that carry the most weight. A population stability index or a plain quantile diff is enough. Alerting on all four hundred features produces noise nobody reads.
Forecast distribution. Total predicted volume against the same period last year, share by category, and the count of series predicted at zero. A join that silently went to the wrong grain shows up here as a total that moved 30 percent overnight.
Revision churn. The mean absolute difference between this run's forecast for a target and the previous run's forecast for the same target. This is the most useful early signal in forecasting and almost nobody instruments it. Set the threshold by horizon: near horizons should barely move day to day, far horizons legitimately wander. A broken feature, a bad backfill or a partial load moves churn immediately, long before any actual arrives to prove it.
Signal Value During the Label-Delay Window
Relative usefulness while the long-horizon labels are still unavailable. All six are cheap; the top two are close to free.
Patterns that produce a forecast nobody trusts
- Overwriting yesterday's forecast, so real production accuracy can never be computed after the fact.
- Training on tables that get restated, then attributing the live error gap to the model.
- Reporting MAPE on series that contain zeros, where the metric is undefined or explodes.
- One train/test split reported as the accuracy of the system.
- A scheduled refit that reaches serving with no shadow period and no version in the output.
- Reconciling the hierarchy in the dashboard but not in the forecast table, so two different numbers exist for the same thing.
- Alerting on job failure rather than on the numbers the job produced.
- Shipping point forecasts only, then asking planners to set safety stock from them.
- Scoring at the model's grain when every decision is made two levels up.
Reconciliation, and the two-number problem
The moment a forecast is consumed at more than one level, coherence becomes an engineering requirement. The sum of SKU forecasts will not equal the regional forecast, and the regional forecast will not equal the total, unless something makes them agree. Bottom-up, top-down and middle-out all work and all lose information. Optimal reconciliation methods, of which minimum-trace is the best known, use the error covariance across levels to produce coherent forecasts that are usually more accurate than any single-level approach. Implementations are available in the R fable ecosystem and in Python through scikit-hts and Nixtla's hierarchicalforecast.
The engineering rule matters more than the method choice: reconcile once, inside the pipeline, and write only reconciled numbers to the serving table. If a BI tool reconciles on its own, you have built a second forecasting system that nobody maintains, and the first time the two disagree in a meeting, the whole system loses credibility regardless of which one was right.
The consumer contract
Write down who consumes the forecast, at what grain, on what cadence, and what action changes when the number changes. If nobody can answer the last one, stop and fix that before writing more code. A forecast with no default action is a dashboard, and dashboards do not pay for the pipeline underneath them.
Expect overrides and instrument them. Planners will adjust the system's output, and that is fine; they know about the customer conversation that is not in your data. What is not fine is losing the override. Store the original, the override, the person and a reason code. Override data is the cheapest labeled feedback the system will ever get, and it almost always names a feature you have not built yet: a promotion calendar, a competitor opening, a customer's own published plan. Two quarters of override reasons is a feature roadmap written by the people who know the business.
Then measure the plan, not just the forecast. If planners override a third of the output, the accuracy of the raw model is a partial answer at best. Track both, and track the case where the override made things worse, which is the conversation that earns the system its authority.
Serve quantiles for anything with an asymmetric cost. Being one unit short of a component that halts a line is not the mirror image of holding one unit too many. That is a newsvendor problem, the right service level follows from the two costs, and it needs a distribution rather than a mean. Evaluate quantiles with pinball loss and check calibration as horizons mature: if your stated 90th percentile is exceeded 20 percent of the time, the interval is decorative.
Metrics that survive a room full of stakeholders
Metric choice is where forecasting arguments go to die, usually because two people are quoting different metrics at different aggregations. Pick the metric from the decision, report it at the grain the decision is made at, and publish a skill score against a naive baseline so the number means something to someone who does not do this for a living.
| Metric | Good for | Where it breaks |
|---|---|---|
| MAPE | Familiar, easy to explain to a business audience | Undefined at zero, unbounded near zero, and it penalizes over-forecasting more than under-forecasting, which quietly biases the model |
| WAPE | Volume or revenue-weighted accuracy across a catalog | Hides poor accuracy on the long tail; report it beside a per-series distribution |
| MASE | Comparing across series with different scales; well behaved with zeros | Needs a stated seasonal period, and the number is not intuitive without explanation |
| RMSE | When large misses cost disproportionately more than small ones | Dominated by outliers, and it optimizes toward the mean when the decision wants a quantile |
| Pinball loss | Scoring the quantiles you actually serve | Not interpretable to stakeholders; pair it with a coverage table |
| Skill vs seasonal naive | The one number that answers "is this worth running" | Nothing, which is why it belongs in every report you publish |
Report accuracy by horizon, always. A blended number across a ninety-day horizon set averages an easy next-week problem with a hard next-quarter one and tells you nothing about either. Two curves, error against horizon for the model and for the baseline, communicate more in one glance than a table of aggregates.
The go-live checklist
- Every source has an ingestion timestamp and every dimension has validity dates
- The forecast table is append-only and carries run, origin, horizon and model version
- The served feature vector is hashed and stored with the forecast
- Rolling-origin backtest with refits in the loop and a gap equal to real data latency
- A seasonal-naive baseline runs in production beside the model, permanently
- Freshness, distribution and revision-churn alerts route to a named on-call owner
- Refits shadow the incumbent for a stated period before promotion
- Quantiles are served and interval coverage is measured as horizons mature
- Overrides are captured with a reason code and reviewed quarterly
- A written rollback: pin the last good model version and rerun, in one command
When the answer is not a forecast
Part of doing this work honestly is knowing when to say the project should not run. Four cases come up repeatedly.
Intermittent demand. When most periods are zero, a point forecast of 0.3 units is not actionable and standard accuracy metrics stop meaning anything. Croston's method and its variants exist for this, but the better answer is often a stocking policy driven by a service level rather than a forecast at all.
Not enough history. Learning an annual season needs several annual cycles. With eighteen months of data, a model that appears to find a season has found noise. Pool the series into a global model, borrow from a related category, or accept a simpler method and say so.
A regime break. A pricing change, an acquisition, a channel closure. History before the break is not evidence about after it, and no amount of feature engineering makes it so. State the break, hold the model for the rebuild period, and use judgment with a wide interval in the meantime.
The decision does not move. The most valuable question to ask in week one is what the decision-maker does differently at 900 versus 1,100. If the answer is nothing, buffer policy dominates and the forecast will not change the outcome. Saying that in week one costs a conversation. Discovering it in month six costs the budget.
Bottom line
Production forecasting is a data-engineering discipline with a model inside it. The parts that decide whether the system earns its place are vintage-correct training data, an append-only forecast record, an evaluation loop shaped like the serving path, a retrain policy with a shadow period, monitoring that works while the labels are still in the future, and a written contract with whoever acts on the number. Get those right and the choice between a boosted tree and a sequence model becomes what it should be, a tuning decision worth a few points. Get them wrong and no model recovers the gap, because the gap was never in the model.
Frequently asked questions
Usually because the model trained on restated history it will not have at inference, or on features that are not yet available at the forecast origin. Test it by reconstructing a past run's feature vector and diffing it against what was actually served. The remaining causes, in order, are unreviewed refits, upstream schema changes, and reconciliation applied in only one path.
Retrain when a seasonal cycle completes, when a driver's distribution shifts and holds across two windows, or when skill against the seasonal baseline declines for several periods. Do not retrain on one bad week. Whatever the cadence, treat a refit as a deployment: version it, run it in shadow beside the incumbent for two to four weeks, and promote on evidence.
Series, target period, horizon, run timestamp, data-cut-off timestamp, model version, the point forecast, the quantiles you serve, and a hash of the feature vector used. Append only, never update. Consumers read a view that selects the latest run per target, so the interface stays simple while the history stays intact.
Watch input freshness and row counts, feature distributions against the training window, forecast totals against the same period last year, and run-over-run revision churn. Churn is the strongest early signal and the least commonly instrumented: a broken feature or partial load moves it hours before it moves any accuracy metric.
Pick from the decision. Use WAPE weighted by volume or revenue for catalog demand, MASE when comparing across series of different scales or with zeros, pinball loss for the quantiles you serve, and always publish a skill score against a seasonal-naive baseline. Report by horizon rather than as one blended figure, and report at the grain the decision is made at.