Time-series forecasting in practice: demand, capacity, and failure prediction on operational data

The part of forecasting that decides the outcome

Forecasting on operational data is an engineering problem with a statistics core. Most teams invert that. They spend months on model architecture and about a week on the framing, then discover that the number they trained a network to predict is not the number anyone needed. A team that gets the framing right can ship a gradient-boosted model in two weeks and beat a research group that spent six months on architecture, because the framing decides the ceiling and the model only decides how close you get to it.

The framing is four questions. What quantity are we predicting? At what granularity and level of aggregation? How far ahead, and with how much lead time before the decision it feeds? And what evidence will convince an operator or an auditor that the forecast beats what they do now? Each has a wrong answer that is cheap to choose and expensive to undo a year later.

Forecast accuracy is also bounded by the process, not by the model. Series driven by human decisions, weather, and funding cycles carry an irreducible error floor. A good project finds that floor early, then spends its effort on the interval, the lead time, and the decision workflow, which still have room to move long after the point forecast has stopped improving.

Three problem shapes that look alike and are not

Demand. How much of something will be consumed, requested, or arrive. Order volume, call volume, benefit claims, fuel draw, spare-part requisitions. These series usually have real seasonality, calendar effects, and a long tail of slow-moving items where the count is zero most days. Demand forecasts feed inventory, staffing, and budget decisions, so the cost of being under is almost never the same as the cost of being over. That asymmetry belongs in the loss function, not in a footnote.

Capacity. How much throughput a system will have, or how close it will run to a limit. Grid load, network utilization, GPU cluster occupancy, runway availability, cloud spend against a ceiling. These series are often better behaved than demand series, with strong daily and weekly cycles and heavy weather dependence. The useful output is rarely the mean. It is the probability of crossing a threshold in the next N hours, which is a distribution question from the start.

Failure and time-to-event. When will this asset degrade past a limit, and how much warning can we give. This is not a standard forecasting problem at all. The label is an event, the positive class is rare, the history is right-censored because most assets have not failed yet, and the metric that matters is precision at an operationally useful lead time. Survival models, Weibull hazard fits, and classification over a labeled prediction window all beat a naive regression on remaining useful life. The public NASA C-MAPSS turbofan degradation datasets, released through the NASA Prognostics Data Repository, remain the standard sandbox for testing an approach before touching real fleet data.

The decisions that are expensive to reverse

Granularity and hierarchy level. Forecast at the level the decision is made, then aggregate up, unless the bottom level is too sparse to carry signal. Moving from weekly to daily after launch means rebuilding the feature store, the backtest, and every consumer. Decide once, with the operator in the room.

Horizon and lead time. A 24-hour-ahead forecast and a 90-day-ahead forecast are different products with different feature sets and different achievable accuracy. The horizon that matters is rarely the one the customer names first. Ask what the decision is and how long it takes to act on it. If a part takes 45 days to procure, a 7-day forecast is decoration.

Point estimate or distribution. Committing to a point forecast and adding intervals later is a rewrite, because the training objective changes. If any downstream decision involves a safety stock, a staffing buffer, or a threshold alarm, build quantiles from day one. Quantile regression, quantile gradient boosting, and conformal wrappers all give you that without a research program.

The backtest protocol. The rolling-origin evaluation described by Tashman in the International Journal of Forecasting in 2000 is the standard: repeatedly cut the history at a point in time, train on everything before it, predict forward, and roll the cut. A random train/test split on time-series data is not a weaker test. It is an invalid one. Write the protocol down before the first model, and change it only with a written reason.

Data work comes first, and it is most of the work

Three data problems account for most of the difference between a system that works and one that quietly does not.

The first is feature availability at prediction time. Every feature has to be knowable when the forecast is issued, from information that existed then. Joining tomorrow's actual temperature into a model that runs before tomorrow happens is the most common way a project produces a spectacular backtest and a useless product. The fix is to train on forecast weather rather than observed weather, and to build the feature store so every row carries the timestamp at which its value became available.

The second is revisions. Many operational and economic series are restated after publication. If the training data holds today's revised values but the production system will see first-print values, the model has been trained on a world it will never inhabit. The Federal Reserve Bank of St. Louis maintains ALFRED, an archive of the original vintages of the series in FRED, precisely so that this can be done correctly on economic inputs. Internal systems rarely keep vintages by default, and adding that after the fact is painful.

The third is alignment and calendars. Time zones, daylight saving, fiscal-year boundaries, federal holidays, and pay periods all show up as structure the model will otherwise call noise. The federal fiscal year ending 30 September produces one of the strongest annual signals in any series touching obligations or procurement, and it costs one column to encode.

Model classes, and what the evidence actually says

The forecasting competitions are the closest thing this field has to a referee, and they are worth reading before choosing an architecture. The M4 competition in 2018 ran 100,000 series and was won by a hybrid of exponential smoothing and a recurrent network from Slawek Smyl. The M5 competition in 2020 used 42,840 hierarchical Walmart series over 1,941 days, and the top of the accuracy leaderboard was dominated by gradient-boosted trees, mostly LightGBM. Pure deep learning has posted real wins since, including N-BEATS at ICLR 2020, but the practical lesson holds: on tabular operational data with many related series, gradient boosting over lag and calendar features is a strong default.

Approach	Where it earns its keep	What it costs you
Seasonal naive	Mandatory baseline. Sometimes the shipped answer on stable, strongly periodic series.	No covariates, no intervals worth trusting.
ETS / ARIMA	Few series, long clean history, need for interpretable components and analytic intervals.	Does not scale to thousands of series or many exogenous drivers.
Croston / SBA	Intermittent demand where most periods are zero. Syntetos and Boylan's 2005 correction fixes Croston's known bias.	Point forecast of a rate, not a usable distribution.
Gradient boosting on lags	Many related series, rich covariates, hierarchy. The workhorse for demand and capacity.	Feature engineering is the job; extrapolation past the training range is poor.
Global neural models	Long horizons, shared structure across series, native probabilistic output (DeepAR, TFT, N-BEATS).	Training cost, tuning surface, harder to explain to an evaluator.
Pretrained forecasting models	Cold start and zero-shot baselines on new series (Chronos, TimesFM, Moirai).	Rarely beats a tuned local model once you have two years of your own history.

One published result deserves its own sentence, because it saves projects. Zeng and colleagues showed at AAAI 2023 that a one-layer linear model, DLinear, matched or beat several transformer architectures on the standard long-horizon benchmarks. That does not mean transformers are useless for time series. It means the burden of proof sits with the complicated model, every time, and the linear baseline has to be run before anyone buys GPUs.

What "good" means numerically

Report scaled error, not percentage error. Mean absolute percentage error breaks down when actuals approach zero, which is exactly where intermittent demand lives, and its symmetric variant is not symmetric in the way the name promises. Mean absolute scaled error, introduced by Hyndman and Koehler in 2006, divides your absolute error by the in-sample absolute error of the naive forecast. MASE below 1 beats the benchmark. MASE around 0.9 is a real but modest gain. MASE near 0.6 on an operational series is a strong result and worth a careful check for leakage before anyone celebrates.

For distributions, use pinball loss at the quantiles you will ship, or continuous ranked probability score for the whole predictive distribution. The M5 uncertainty track scored exactly this way, with a weighted scaled pinball loss. Then check empirical coverage separately: if your nominal 90 percent interval contains the actual 72 percent of the time, the model is overconfident and every buffer built on it is undersized. Conformal methods give distribution-free coverage under exchangeability, and adaptive variants such as the online approach of Gibbs and Candès address the fact that time series violate that assumption.

For failure prediction, ROC-AUC flatters rare-event models and should not be the headline. Use precision-recall. The arithmetic is unforgiving: at a 0.5 percent failure base rate across 10,000 assets, a model with 90 percent recall and a 10 percent false-positive rate produces 45 true alerts and 995 false ones, for a precision of about 4 percent. That may still be worth deploying if an inspection is cheap and a failure is not, but the number has to be on the table before anyone promises a maintenance crew.

Finally, test whether the gap between two models is real. The Diebold-Mariano test, published in 1995, compares predictive accuracy between two forecasts on the same series and is a two-line addition to any evaluation script. Shipping a model that is 1.5 percent better on one holdout is how teams lose a year.

The failure modes we see most

• Target leakage through time. A feature that only becomes available after the forecast is issued. The backtest looks superb and production never reproduces it.
• Baseline never run. No seasonal naive number in the report, so nobody can say whether the model added anything at all.
• Aggregation illusion. Accuracy is reported at the national or monthly roll-up while the decision is made per site and per day, where error is several times larger.
• Metric mismatch. Squared-error training on a business problem where stockouts cost ten times what overstock costs. The loss function should carry the asymmetry.
• Regime breaks treated as noise. A funding change or a facility closure shifts the process, and models retrained blindly across the break stay wrong for months.
• Intervals bolted on late. Quantiles produced by a rule of thumb over a point model, with no coverage check, then used to size real buffers.
• Silent feature drift. An upstream schema change fills a lag column with nulls, the model imputes, and error degrades slowly enough that nobody notices for a quarter.
• No decision hookup. The forecast is accurate, arrives after the planning meeting, and is therefore never used.

Production: latency, cost, and monitoring

Most operational forecasting is a batch job, and treating it as one removes a great deal of complexity. Compute forecasts on a schedule, write them to a store keyed by entity and horizon, and serve reads in single-digit milliseconds. Real-time inference is only needed when the forecast is conditioned on inputs that arrive with the request. Ask that early: an unnecessary online serving path can double the build.

Cost is usually modest and usually misunderstood. The M5 data is roughly 59 million rows at the bottom level, and a gradient-boosted global model over that fits on one well-provisioned machine. The expensive parts are unpruned hyperparameter sweeps, retraining faster than the data changes, and storing every intermediate feature matrix. Retraining weekly when the process shifts quarterly is pure burn.

Monitoring a forecaster differs from monitoring a classifier, because ground truth arrives on a delay equal to the horizon. Build the scoring job that joins predictions to actuals as they land, and track rolling MASE by cohort, interval coverage against nominal, and input drift. In federal deployments that scoring history is also the artifact that answers documentation questions later, whether the frame is the NIST AI Risk Management Framework (NIST AI 100-1) or the minimum practices for high-impact AI in OMB Memorandum M-25-21. Log the model version, training window, feature set, and code commit with every batch, and that record writes itself.

One production note specific to federally funded work. If the model is developed under an SBIR or STTR award, data rights on the delivered software and technical data run through DFARS 252.227-7018, with the noncommercial clauses at 252.227-7013 and 252.227-7014 behind it. Decide before delivery which artifacts are the model, which are the pipeline, and which are the customer's data, because the markings follow those lines.

When the simpler method is the right answer

How we run this work

Our team starts with the decision, not the dataset. We write down the target, granularity, horizon, lead time, and loss asymmetry in one page, get it signed off by the person who will act on the output, and only then touch data. We build the rolling-origin backtest and the seasonal naive baseline before the first real model, so every later result has a reference point. We ship quantiles by default and check coverage every cycle. We instrument the pipeline so the evidence a reviewer or a source-selection board will ask for is a query rather than a scramble.

We work across the modeling range, from state-space and intermittent-demand methods through gradient-boosted global models to probabilistic neural architectures, on AWS, Azure, and Google Cloud including the government regions. The choice among them is an evidence question answered on the customer's data, and we are glad to be the ones who report that the simple method won.

Frequently asked questions