Streaming versus batch for model serving

Three questions wearing one coat

"Should we serve this model in a stream or in a batch job?" is a question that cannot be answered as asked, because it bundles three independent decisions that usually have different answers on the same system. When the prediction is computed. How the request for it arrives. How fresh the inputs to it are. Teams collapse those into one word, pick the word, and then spend two quarters discovering that the word committed them to things they never chose.

Computation timing. Ahead of the request on a schedule, or at the moment of the request. This is the axis people mean when they say batch, and it is the one with the clearest cost consequence.

Request arrival. A synchronous call with a person or a caller waiting on the other end, an event landing on a log with nobody waiting, or a scheduled sweep over a population. Arrival pattern sets your tail-latency obligation, and it is independent of when the number was computed.

Input freshness. How old the newest feature is at decision time. This is the axis that actually determines whether the model works, and it is the one nobody measures before the argument starts.

Separating them lets you build the thing that is usually correct: predictions precomputed on a schedule, delivered over an event stream, using features that are seconds old for two of the twelve feature groups and a day old for the rest. That system is not "streaming" or "batch." It is a set of four defensible choices, and you can only make them once you stop treating the question as binary.

Start with the staleness budget, not the latency budget

The staleness budget is the maximum age of the newest input at the moment the prediction is used, past which the prediction gets materially worse. The latency budget is how long the caller waits for a response. They are different numbers and they trade against each other, and conflating them is the most common error in this whole area.

A 40 ms response computed from features refreshed at 2 a.m. is fast and stale. A prediction that takes three seconds to reach a downstream consumer but incorporates an event that happened 400 ms ago is slow and fresh. Which one is wrong depends entirely on what the model is doing. A next-best-offer model on a monthly billing cycle does not care about the last four minutes. A card-not-present fraud score cares about almost nothing else.

Two ways to get the staleness budget without anyone raising their voice. The first is a freshness-weighted importance split: take permutation importance on a holdout set, not tree gain, which is biased toward high-cardinality splits, then bucket every feature by how often the underlying value actually changes, and sum the importance mass per bucket. If eighty percent of the mass sits in features that move monthly, streaming the remaining twenty percent buys a small metric change and costs you a stateful job you now operate forever. If a third of the mass is in counts over the last few minutes, batch cannot serve the model at all and the meeting is over in ten minutes.

The second is cheaper and more convincing to people who do not trust importance measures. Re-score the same holdout with features artificially aged by one hour, six hours, twenty-four hours, seven days. Plot your primary metric against feature age. That curve is the staleness budget, it usually has a visible knee, and it turns an opinion into a chart that a product owner can read. Run it before the architecture discussion, not after.

Four serving shapes, not two

Once the axes are separate, four shapes fall out. Most production systems run two or three of them at once, which is fine as long as somebody wrote down which is which.

Batch scoring. Score a population on a schedule, write predictions into a key-value store or a table, serve them by lookup. The model is not in the request path at all, so serving latency is your store's latency and has nothing to do with model size. You can run a model here that would never survive an online budget.

Synchronous online inference. The model runs inside the request. Every millisecond of the forward pass is a millisecond a caller waits, and capacity has to be sized for peak concurrency rather than average throughput.

Streaming inference. A long-lived consumer reads events off a log, computes, and writes results to another topic or to a store. Nobody is blocked on it, so the latency that matters is event-to-availability rather than request-to-response. It is continuous and asynchronous, which are two different properties that the word "real-time" hides.

Near-line hybrid. Precompute the expensive, slow-moving part on a schedule. Compute the cheap, fast-moving part inside the request. This is where most systems that work well end up, and it is almost never what anybody proposes in the first design review.

Shape	What sets latency	What drives cost	Dominant failure	Backfill story
Batch scoring	Key-value read, plus how stale the last run left you	Population size, not traffic	Run fails at 3 a.m. and yesterday's predictions serve silently	Trivial. Rerun the job.
Synchronous online	Feature fetch plus forward pass plus network	Peak concurrency, which you pay for at the mean	Tail latency under fan-out; cold start on scale-out	None. There is no history unless you logged it.
Streaming inference	Event time to result availability, plus allowed lateness	Standing state and broker capacity, mostly fixed	Windowed state divergence and unbounded state growth	Replay from the log, if retention outlives the window
Near-line hybrid	Online leg only, over precomputed candidates	Both, but each on a smaller base	Two code paths drifting apart without a parity test	Rerun the batch leg; replay the online leg from logged vectors

The economics: coverage versus peak

Batch pays for coverage. You compute a prediction for every entity in the population whether or not anyone asks for it. Cost scales with the size of the population and is almost independent of traffic. Reuse is what makes it cheap: a prediction read fifty times before it expires cost one fiftieth of a forward pass per read.

Online pays for peak. Cost scales with requests, and capacity has to be provisioned for peak concurrency while it idles at the mean. In a business-hours product the peak-to-mean ratio commonly sits somewhere between three and ten, so a fixed fleet runs at ten to thirty percent utilization unless it autoscales well. Model servers autoscale poorly compared to stateless web services because a new replica has to pull and load weights before it takes traffic, which pushes real scale-out time into tens of seconds or minutes. That gap is exactly when your traffic spike happens.

The crossover is arithmetic, and it is worth doing on a whiteboard before anyone opens a design doc. Let N be the population you would score in batch, R the requests you would serve online over the same refresh interval, c_b the marginal cost of one prediction inside a batch job and c_o the marginal cost of one prediction served online. Batch total is N × c_b. Online total is R × c_o. Batch wins when R / N > c_b / c_o. Because batching amortizes kernel launches and keeps accelerators busy, c_b is often a third of c_o or better, so the threshold sits near 0.33: if the average entity is read more than once every three refresh intervals, precomputing everything is cheaper than computing on demand.

Then look at the distribution, because the average is lying to you. Read traffic over entities is almost always skewed hard. When a small fraction of entities takes the large majority of reads, batch spends most of its budget on rows nobody touched, and the correct answer is neither pure shape. It is a cache: compute lazily on first request, store the result with a time-to-live equal to the staleness budget, serve every subsequent read from the store. A cache is a batch of one, computed on demand, and in a skewed workload it beats both options on cost and on freshness at the same time.

One cost sits outside all of this. A stateful streaming job has a floor: brokers or a managed log, a state backend, checkpoint storage, replay tooling, and a person who can be woken up when consumer lag climbs. That floor is roughly constant whether the job handles a thousand events a second or fifty thousand, and it is what makes streaming expensive for small workloads and cheap for large ones.

Latency: the model is rarely the slow part

Open up a synchronous prediction and the forward pass is usually not what is spending your budget. The sequence is network and TLS, authentication and authorization, feature retrieval, the model itself, business rules and post-processing, then serialization back out. Feature retrieval dominates in most online paths we open, and it dominates for a boring reason: the code fetches twelve feature groups one after another because that is how it reads well.

Twelve sequential lookups at eight milliseconds each is ninety-six milliseconds spent before the model has seen anything. Collapsing them into a single multi-get, or into a small number of parallel fetches with a shared deadline, is routinely the highest-return change available in an online serving path, and it requires no change to the model. Do that before anyone proposes quantization.

The tail behaves worse than intuition suggests. With fan-out to several independent dependencies, the request is as slow as its slowest leg, so the probability of hitting a slow leg compounds with the number of legs. A dependency with a clean p99 of 20 ms, called ten times in parallel, does not give you a 20 ms p99. The mitigations are known: per-leg deadlines derived from the remaining request budget rather than fixed constants, hedged requests for read-only legs, and a fallback that is defined in advance.

That fallback is a modeling decision that gets made by infrastructure defaults. When a feature group times out you serve a default value, a stale value, or an error, and all three change the prediction. Choose deliberately, record which path ran in the prediction log, and evaluate the model on the degraded path as well as the healthy one. Almost nobody measures the quality of the fallback path, and the fallback path runs on exactly the days when accuracy matters most.

Streaming is a state problem, not a throughput problem

Log throughput is almost never the constraint. A single partition on any modern log handles far more than a feature pipeline produces, and if it does not, you add partitions. What actually makes streaming hard is windowed state, and there are four specific places it goes wrong.

Event time versus processing time. Your training features were almost certainly built with a SQL window over a timestamp column, which is event time. If the streaming job aggregates by arrival, which is processing time, the serving feature has a different definition than the training feature. Monitoring will not catch it, because both values look completely reasonable. The model just performs worse than it did offline and nobody can say why.

Watermarks and allowed lateness. You have to choose how long to wait for late events, and that choice puts a floor under freshness. Two minutes of allowed lateness means the correct value of a windowed feature is at best two minutes old. You can emit early and revise, but then a consumer sees a value that changes underneath it, and a prediction logged against the early value will not reproduce from the revised one. Decide which of those you want and write it in the feature definition.

State size. A thirty-day distinct-count per user across ten million users is not a configuration change. It is a state backend with a real memory and disk profile, checkpoints that take real minutes, and a recovery time you should measure before you need it. Approximate structures such as HyperLogLog or count-min sketches exist because exact distinct counts over long windows are expensive, and the accuracy trade is usually acceptable in a feature but should be a stated decision rather than a discovery.

Rebuilds. Change the definition of a thirty-day window and you need thirty days of history to repopulate it. Log retention has to exceed the longest window plus the longest outage you intend to survive, or the rebuild path is a batch backfill that you must therefore keep working and keep tested. Teams find this out during an incident, which is the worst possible time to discover that retention is seven days.

Where streaming serving actually breaks

• Aggregating on processing time while the training set used event time. The single most expensive line in a streaming feature pipeline, and it survives code review because both versions look correct.
• Log retention shorter than the longest feature window. Discovered on the day you need to rebuild, when the data required to rebuild no longer exists.
• Unbounded key space in a keyed state store. No time-to-live on per-key state, so the job runs fine for six weeks and then starts failing checkpoints for reasons nobody connects to a session identifier.
• Alerting on job liveness instead of consumer lag and feature freshness. A running job that is forty minutes behind pages nobody, and forty minutes is well outside the staleness budget you wrote down.
• Non-idempotent writes to the feature store. Every replay inflates counters, which turns recovery from an operation into an incident.
• A separate backfill implementation from the streaming path. Two code paths computing the same feature, one of which is only ever exercised during emergencies.
• Serving predictions with no record of the input vector. When someone asks why a specific decision was made, the honest answer is that you cannot reconstruct it.

Training-serving skew is created at this boundary

Nearly every model that scores well offline and disappoints online is losing to skew, and the boundary between batch training and online or streaming serving is where skew is manufactured. Three fixes, in order of how reliably they work.

One implementation, two execution modes. Write each transformation once, in a library that the batch job and the serving path both call. This is the only fix that eliminates skew by construction rather than by testing for it. It constrains your choice of tooling, and that constraint is worth accepting.

Two implementations plus a parity test. Sometimes the online path has to be a different language or a different engine. Then run both implementations over the same recorded inputs in continuous integration on every change and assert agreement inside a stated tolerance. A parity test that runs on merge is a real guard. A parity check somebody performs by hand before a release is not.

Log the served feature vector and train on it. Record the exact vector used at serving time, with the feature version and the code version alongside it, then build training sets from those logs instead of recomputing history from source tables. This removes recomputation skew entirely and it makes incident replay possible, which is worth as much. The storage arithmetic is friendlier than people assume: two hundred float32 features is eight hundred bytes plus keys, so ten million predictions a day is under ten gigabytes a day before compression. Sample if you have to, and keep a full unsampled slice so replay stays available.

Where the hybrid lives

The near-line pattern is worth naming explicitly because it is the answer more often than either pure shape. Split the work by how fast the inputs move. Precompute anything expensive and slow-moving: embeddings, an approximate-nearest-neighbor index, long-horizon user profiles, candidate sets. Compute online only what has to be fresh: session context, the last few interactions, an eligibility filter, a re-ranker over a few hundred candidates rather than the whole catalog.

The rule is mechanical. Any input that cannot change between refreshes belongs in the batch leg. Any input that can change inside the request belongs in the online leg. Everything in between belongs in a streaming aggregate with a refresh interval you can state in seconds, and if nobody can state it in seconds, that feature has not been designed yet.

There is a second hybrid split that is worth as much and gets used less: precompute the head, compute the tail lazily. When a small share of entities produces most reads, run the scheduled job over that head only and serve everything else on demand with a cache. Same model, same transformation code, two triggers. It cuts batch spend by the share of the population you stopped scoring and it removes the cold-start hole that pure caching leaves for your busiest entities.

The rubric

Score each criterion 0 to 10, where 10 argues for computing at request or event time and 0 argues for a scheduled batch. Multiply by the weight, sum, divide by ten. The result lands between 0 and 100.

Criterion	Weight	Scores 10 (compute at event time) when	Scores 0 (precompute) when
Staleness budget	28	Top features change in seconds or minutes and the metric curve drops sharply with age	Nothing in the vector moves faster than daily
Arrival pattern	22	A caller blocks on the answer, or the triggering event is unpredictable	Consumption is scheduled, or the entity set is known in advance
Read reuse	18	Most entities are never read; traffic is sparse against a large population	Each entity is read repeatedly inside one refresh interval
Reproducibility	14	You already log served vectors, versions and the code that produced them	Reconstruction depends on rerunning a job over source tables
Operating capacity	11	Someone owns consumer lag, checkpoint health and replay, on call	The team can support a scheduled job and not much more
Time to first version	7	No table exists to score against; the data only arrives as events	A working batch version can ship this sprint

65 and above: compute at request or event time, and budget for the state and replay work honestly.
35 to 64: near-line hybrid. Precompute the slow layer, compute the fast layer in the request. Most systems belong here.
34 and below: scheduled batch scoring with a cache in front of it, and spend the saved engineering time on the model.

Three archetypes to calibrate against. A weekly marketing ranking over a known audience scores in the teens: the audience is enumerable, nothing moves inside a week, and precomputing is both cheaper and easier to audit. A card-not-present fraud score scores in the eighties: the decisive features are velocity counts over the last few minutes, a caller is blocked, and the entity set is not known ahead of time. A support-ticket triage classifier lands in the middle forties, because the text arrives as an event and has to be scored on arrival, while the account features that matter to routing were fine when they were computed last night.

A two-week way to decide

Two weeks is enough because every expensive unknown here is measurable inside it. Whether freshness moves the metric is measurable. Whether your reads are skewed is a query against logs you already have. Whether your team can operate a stateful job is answered by having them build one small one and rebuild it from the log on purpose. Everything else is preference, and preference is what turns this into a six-month argument that still ends in the wrong shape.

Own these whichever shape you pick

• A written staleness budget per feature group, in seconds, next to the feature definition
• One transformation implementation, or two with a parity test that runs on every merge
• Served feature vectors logged with feature version and code version
• Idempotent writes keyed on entity, window and version, so replay is safe
• Alerts on feature freshness and consumer lag, not only on job success
• A defined and evaluated fallback path for every dependency that can time out
• A rebuild path that has been executed at least once outside an incident
• An offline evaluation set that can score both the healthy and the degraded path

Bottom line

Streaming versus batch is not a philosophy and it is not a platform preference. It is a staleness budget, a read distribution, and a latency budget, and those three numbers can be measured in under two weeks by people who already work on the system. Precompute what does not move. Compute at request time what does. Put the boundary where the metric curve tells you to put it, log enough to reconstruct any prediction you served, and keep the transformation code in one place so the model you evaluated is the model you deployed.

Frequently asked questions