Capacity planning for bursty inference

The average request rate is not a capacity number

A team tells us they serve forty inference requests per second and asks how many GPUs they need. There is no answer to that question. Forty per second averaged across a day is a completely different machine from forty per second that shows up as four hundred for ninety seconds at 9:05 and nearly nothing overnight, and the second shape is what almost every production workload actually looks like. Capacity planning for inference is not arithmetic on a mean. It is a decision about what happens in the minutes between a burst arriving and new capacity being able to serve it.

The first thing to fix is the measurement. Most teams read their traffic off a dashboard that rolls up to one-minute or five-minute buckets, and that averaging destroys exactly the signal capacity planning needs. Work it out: a burst that runs at ten times baseline for twenty seconds, inside a five-minute window, averages to (20 × 10 + 280 × 1) / 300 = 1.6 times baseline. The event that took the service down appears on the chart as a modest bump. Nobody investigates a 1.6× bump.

So before any sizing work, pull raw request timestamps for a representative week and re-bucket them at one second. Then compute three numbers that matter more than the mean: the ratio of the 99th-percentile one-second rate to the median one-second rate, the longest continuous stretch above 2× median, and the fastest ramp measured as the largest increase between two adjacent seconds. Those three describe the workload you are buying hardware for. Peak-to-mean ratios between 3× and 10× are ordinary for interactive products. We have seen higher on workloads driven by a customer's own batch schedule, where an entire day's volume lands inside a fifteen-minute window because somebody's cron job fires at the top of the hour.

Two equations do most of the work

The first is Little's Law. In a stable system, the number of requests in flight equals the arrival rate multiplied by the time each request spends in the system. Concurrency is not something you configure. It is something the traffic and your latency jointly decide, and your job is to have enough seats for it.

Forty requests per second at a three-second mean end-to-end time gives 120 requests in flight. If a replica can hold thirty concurrent sequences before it starts queueing, that is four replicas at the average, and the average is not what you are provisioning for. At a 5× burst the same arithmetic wants 600 seats. That is the whole capacity conversation in two lines, and it is worth doing on a whiteboard before anyone opens a cloud console.

The second equation explains why the last twenty percent of a machine is not for sale. For a single-server queue with random arrivals, mean waiting time in the queue is service time multiplied by ρ / (1 − ρ), where ρ is utilization. Real inference servers are not textbook queues, but the shape of that curve is the shape every serving system follows, and it is brutal near the top.

Two consequences follow. The finance argument for running a fleet at 90 percent utilization is an argument for multiplying queue delay by nine, and it is usually made by someone reading a monthly average rather than a one-second one. And during a burst, instantaneous utilization goes above 1.0, at which point the queue does not settle at a higher number. It grows for as long as the burst lasts. Every plan needs an explicit answer for what happens to the requests that arrive while ρ is greater than one, because the default answer is that all of them time out, including the ones you could have served.

What one accelerator actually gives you

Transformer inference splits into two phases with different bottlenecks, and conflating them produces capacity numbers that are wrong in both directions. Prefill processes the whole prompt at once and is compute-bound, so it scales with the arithmetic throughput of the card. Decode emits one token per step per sequence and is memory-bandwidth-bound, because every step reads the model weights out of high-bandwidth memory regardless of how many tokens it produces.

That second fact sets a ceiling you cannot optimize past. Take a 70-billion-parameter model in 16-bit precision: roughly 140 GB of weights, tensor-parallel across two 80 GB cards with about 3.35 TB/s of memory bandwidth each. Each card reads its 70 GB share every decode step, so the step rate cannot exceed roughly 3,350 / 70, or about 48 steps per second. At batch size one that is 48 tokens per second per sequence as a hard upper bound, before any real-world overhead. No amount of tuning gets past it, which is why batching is not an optimization in serving. It is the entire economic model. The same weight read serves every sequence in the batch, so throughput scales with batch size while per-sequence speed stays roughly flat until the batch gets large enough to become compute-bound again.

What limits the batch is the key-value cache. Every active sequence holds cached keys and values for every token it has seen, and the per-token cost is 2 × layers × kv_heads × head_dim × bytes_per_element. For a common 70B-class configuration with 80 layers, 8 grouped key-value heads and a head dimension of 128, in 16-bit precision, that works out to 327,680 bytes per token, or 320 KiB. A sequence sitting at 4,000 tokens of context is holding 1.25 GB of cache. On two 80 GB cards with 140 GB of weights resident, roughly 20 GB remains, which is about sixteen concurrent sequences at that context length.

Sixteen. That is the number that should drive the fleet plan, and it is the number nobody has when they ask how many GPUs they need. It also explains why context length is a capacity decision rather than a product decision. Doubling the allowed context roughly halves concurrency per replica at the same memory. If a product manager asks for 32k context on an endpoint sized for 4k, that is a request to quadruple the fleet, and it should be priced that way in the conversation.

Continuous batching, as implemented in vLLM, TensorRT-LLM, SGLang and TGI, is what makes those seats usable. Instead of running a fixed batch to completion, the scheduler admits new sequences into the batch as others finish, so a short request does not wait behind a long one. Paged attention allocates the key-value cache in fixed blocks rather than contiguous reservations, which removes the fragmentation that otherwise wastes a third of the memory. If a serving stack does not do both, the concurrency ceiling above is optimistic by a wide margin.

The clock nobody measures

Ask a team how long it takes to add a replica and you will usually get the autoscaler's cooldown setting. That is not the number. The real interval runs from the moment the scaling signal crosses its threshold to the moment the new replica is serving production traffic, and on a large model it is dominated by things that have nothing to do with the autoscaler.

The weight-load line is where the time goes and where the fixes are. Pulling 140 GB from object storage at a realistic 1 to 2 GB/s is 70 to 140 seconds on a good day and considerably worse when several replicas pull at once and contend for the same bucket throughput. Baking weights into the machine image, staging them on local NVMe, or keeping a warm read-through cache on the node removes most of it. Loading in a lower precision removes more, at a quality cost you should measure rather than assume.

Once you have that number honestly, the arithmetic is unforgiving. If a burst ramps to 5× in twenty seconds and a replica takes six minutes, autoscaling contributes nothing to the first burst. It contributes to the second one, and only if the burst is long enough to still be running when the replica arrives. Every capacity plan is really a plan for those six minutes, and if the plan does not say what happens during them, it is not a plan.

Three shapes of burst, three different answers

Bursts are not one phenomenon. They differ in what causes them, how much warning you get, and which mitigation is even applicable. Sorting your traffic into these buckets is fifteen minutes of work and it eliminates most of the wrong answers.

Shape	What causes it	Warning you get	The right response
Diurnal and weekly Smooth, repeats, 2–4×	Working hours across the time zones your users live in	Weeks. It is on last month's chart.	Scheduled scaling, not reactive. Scale up before the ramp, on a clock.
Scheduled external Sharp, aligned to the hour	A customer's batch job, a nightly export, an integration polling on a cron	Full, if you ask. None, if you do not.	Rate-limit per tenant, offer an async job API, negotiate the schedule.
Exogenous spike Fast ramp, unpredictable size	A launch, a press mention, one large customer switching on a feature	Minutes at best, usually none	Standing headroom plus overflow capacity you rent by the token.
Self-inflicted Amplifying, follows a degradation	Retry storms, thundering herd on cache expiry, a redeploy that drops warm replicas	None. It is caused by your own response.	Retry budgets with jitter, request hedging limits, staged deploys.

The last row deserves attention because it is the one teams create themselves. When latency rises past a client timeout, clients retry. If each retries three times, offered load triples at the exact moment the service is already saturated, and the system cannot recover on its own even after the original cause is gone. The fix is a retry budget rather than a retry count: cap retries at a small fraction of total requests across the client fleet, add full jitter to the backoff, and make the server return a signal that means stop rather than a generic error that means try again. A client library that retries on a 503 with no jitter is a load amplifier wearing a reliability costume.

The response ladder

Every mitigation for a burst has a time constant, and the useful way to organize them is by how fast they take effect. Fast and cheap ones absorb the first thirty seconds. Slow ones handle the tail. A plan needs coverage at every rung, because a gap in the ladder is a gap in the outage.

Mechanism	Takes effect in	Capacity it adds	Where it fails
Batch headroom Free KV-cache seats on running replicas	Immediate	Whatever you deliberately left unallocated, typically 20–35%	Gone the moment you optimize utilization upward
Bounded queue with admission control	Immediate	Smooths bursts shorter than the queue's time budget	Useless without a deadline; an unbounded queue makes things worse
Graceful degradation Shorter outputs, smaller model, cached answers	Seconds	2–5× effective throughput, at reduced quality	Requires the product to have a defined degraded mode
Warm standby replicas Loaded, healthy, holding little traffic	15–60 seconds	Exactly what you are paying to keep idle	You pay for it every hour of every day
Token-billed overflow endpoint Spill to a managed API above a threshold	Seconds, if wired in advance	Effectively unbounded	Different model behavior; needs its own evaluation and data terms
New nodes from the cloud	4–15 minutes	Unbounded until you hit a quota or a regional shortage	Arrives after short bursts are over; check quotas before you need them

The overflow rung is the one most teams have not built and the one that changes the economics most. If your serving layer speaks a stable interface internally, adding a second backend that routes to a managed token-billed endpoint when queue depth crosses a threshold is a few days of work. It converts the tail of your traffic distribution from a hardware problem into a line item that scales with use. The engineering discipline it requires is real: the overflow path needs the same prompt handling, the same safety filters, the same logging, and its own quality evaluation, because a spillover that silently produces worse answers during your busiest hour is a bad trade.

Admission control is the part people skip

Under overload, a service has to decide what it will not do. Teams that never make that decision explicitly end up making it by accident, and the accidental version is the worst one: an unbounded in-memory queue that accepts every request, holds them past every client's timeout, and produces a service that is fully occupied producing responses nobody is still waiting for.

Four mechanics make this behave. Bound the queue by time, not by length. Every request carries a deadline derived from the client's timeout; when a request reaches the head of the queue with insufficient remaining budget, drop it before spending a GPU on it. Reject early and clearly. A fast 429 with a Retry-After header that the client library actually honors is a better outcome than a 30-second wait ending in a timeout, and it is the only response that reduces offered load. Give priority classes real meaning. Interactive user traffic, background enrichment, and internal evaluation jobs should not share one queue; when capacity is short, the background work is what stops. Enforce per-tenant limits with token buckets so one customer's bulk import cannot consume the whole fleet, and size the bucket in tokens rather than requests, because a request is not a unit of work when one prompt can be a hundred times larger than another.

That last point generalizes. Rate limiting on request count is nearly meaningless for inference. Two requests per second can be a trivial load or can saturate a replica, depending on prompt length and how many tokens each one generates. Meter and limit on input tokens and a bounded max output, and cap the max output per tier. An endpoint that allows unbounded generation length has no capacity model at all, because a single client can hold a seat for as long as it likes.

Stop autoscaling on GPU utilization

GPU utilization as reported by the standard tooling is the fraction of time in a sampling window during which at least one kernel was resident on the device. It says nothing about whether the card is doing useful work. A decode loop at batch size one, reading weights out of memory and leaving most of the arithmetic units idle, reports high utilization while the replica is nearly empty and could accept thirty more sequences. Scaling on that signal adds hardware to a fleet that has plenty of seats, and the reverse happens too.

Scale on the signals that describe saturation of the resource that actually binds. Key-value cache occupancy is the best single indicator for a decode-heavy workload, because it is what runs out. Queue depth measured in waiting requests, and the age of the oldest waiting request both lead the latency curve rather than trailing it. Time to first token at p95 is the number your users feel, and it degrades before end-to-end latency does. Preemption or recompute rate, where the scheduler evicts a sequence and later reruns its prefill, is a direct sign the replica is oversubscribed. Modern serving stacks export all of these, and event-driven autoscalers such as KEDA can consume them directly.

One more piece of the same problem: set the scale-up and scale-down asymmetrically. Scale up aggressively on a short window, because the cost of a replica you did not need is one hour of compute. Scale down slowly on a long window, because the cost of removing a replica just before the next burst is the entire cold-start budget spent during an incident. A stabilization window of ten to fifteen minutes on scale-down is not wasteful. It is the price of not thrashing.

Load-test open loop, or do not bother

Most load tests are run with a fixed pool of virtual users, each sending a request, waiting for the response, then sending the next. That is a closed-loop generator, and it has a property that makes it useless for capacity work: when the service slows down, the generator slows down with it. Offered load automatically backs off exactly when you needed to know what happens if it does not. Gil Tene named this coordinated omission, and it is why so many systems pass their load test and fail in production.

Generate load at a constant arrival rate instead, ideally with Poisson-distributed inter-arrival times so the burstiness is realistic rather than metronomic. Tools that support this properly include k6 with a constant-arrival-rate executor, Vegeta, and wrk2; most serving frameworks also ship a benchmark script with a Poisson request-rate option. Then run the ladder: hold each arrival rate for several minutes, record p50, p95 and p99 for time to first token and end to end, and increase until p95 crosses your target. The rate at which it crosses is the replica's real capacity, and it is typically 40 to 60 percent of the throughput number in the marketing benchmark, because that benchmark was measured at a batch size no interactive product can tolerate.

Test the burst directly too. A steady-state ladder tells you the knee; it does not tell you whether a step change from 1× to 5× in fifteen seconds recovers. Run that step, leave it up for two minutes, and watch three things: whether the queue drains after the burst ends, whether any request is served after its client has already given up, and whether the autoscaler did anything before the burst was over. The third answer is usually no, which is the finding that justifies the standing headroom you were about to cut.

Own the base, rent the peak

Once the shape of the traffic is known, the cost structure follows from it. Committed or reserved capacity is the cheapest per hour and the least flexible, so it should carry the load that is present essentially all the time. On-demand capacity costs more per hour and can be released, so it carries the predictable daily ramp. Interruptible capacity is cheaper still and can be taken away with two minutes of notice on some clouds and thirty seconds on others, which makes it right for work that can checkpoint and wrong for anything a user is waiting on. Token-billed managed endpoints cost the most per unit of work and nothing at all when idle, which is exactly the profile you want for the top of a spike.

The crossover between owning and renting is arithmetic you can do in an afternoon, and it is worth doing with today's prices rather than remembered ones, because accelerator pricing has moved substantially in both directions over the past two years. Take your measured sustained tokens per second per replica at your latency target, multiply by 3,600 to get tokens per hour, and divide the fully loaded hourly cost of the replica by that number. Fully loaded means the accelerator, the host, storage, network egress, and the fraction of an engineer's time the fleet consumes. That gives cost per million tokens on owned hardware. Compare it to the list price of a managed endpoint for a model of similar capability. Owned capacity generally wins when utilization stays high, and loses badly when it does not, which is precisely why the base and the peak belong on different tiers.

One caution on interruptible capacity for serving. It is genuinely useful for offline batch inference, embedding generation, and evaluation runs. Putting user-facing traffic on it requires you to handle a reclaim notice by draining in-flight sequences within the notice window, and on a long generation that window may not be enough. If you do it, do it as a fraction of the fleet with a fast failover path, and test the reclaim behavior deliberately rather than waiting to discover it.

Headroom is a number you choose on purpose

The point of all the measurement above is to make headroom an explicit decision instead of an accident. Pick a target: steady-state utilization of the binding resource at 55 to 65 percent, which keeps you on the flat part of the queueing curve and leaves roughly a 1.5× instantaneous burst absorbed with no scaling at all. Then write down, in one sentence each, what covers a 2× burst, a 5× burst, and a 20× burst. If the answer to the last one is not some form of "we shed the low-priority tier and serve the rest," the plan is incomplete, because no amount of provisioned hardware covers an arbitrary spike and pretending otherwise is how budgets get spent on capacity that sits idle for eleven months.

A two-week sizing exercise

Two weeks is enough because every expensive unknown in this problem is measurable inside it. What a replica does at your latency target is measurable. How long a cold start takes is measurable. Whether the queue drains after a burst is measurable. What is left after those three is pricing, and pricing is a spreadsheet. The reason capacity planning drags on for months is almost never that the questions are hard. It is that nobody has been assigned the load generator.

The mistakes we see most

• Sizing on average requests per second, then discovering the peak on a Tuesday morning with customers watching
• Autoscaling on GPU utilization, which correlates poorly with whether a replica can accept another sequence
• Closed-loop load tests that back off when the service slows and therefore never reproduce overload
• An unbounded queue with no deadlines, converting a partial degradation into a complete one
• Client retries with no budget and no jitter, which multiply offered load at the worst possible moment
• Scale-out targets set without the cold-start number, so the autoscaler reacts after the burst has ended
• Interactive and batch traffic sharing one pool, letting an offline job evict the sequences a user is waiting on
• Unbounded max output length, which lets a single client hold a cache seat indefinitely

What a capacity plan should contain

• Traffic characterized at one-second resolution, with peak-to-mean and fastest ramp written down
• A latency SLO stated as percentiles for time to first token and end-to-end, agreed with the product owner
• Measured sustained throughput and maximum concurrency per replica at that SLO, not from a vendor benchmark
• A cold-start budget broken into stages, measured on a real scale-out
• An explicit headroom target and the utilization the fleet is allowed to run at
• Admission control with deadlines, priority classes, and per-tenant token limits
• A named answer for a 2×, a 5× and a 20× burst, one sentence each
• A cost model per tier, with the owned-versus-rented crossover computed at current prices
• Quota headroom confirmed with your cloud provider before you need it, per region and per instance family

Bottom line

Bursty inference is a queueing problem wearing a hardware costume. The mean tells you almost nothing, the utilization curve punishes you for running hot, key-value cache memory sets the concurrency ceiling, and cold-start time decides whether autoscaling is a mitigation or a footnote. Measure those four things and the fleet size stops being an argument. Then spend the engineering effort where it actually pays: on admission control that fails a few requests fast instead of all of them slowly, and on an overflow path that turns the top of the distribution into a variable cost.

Frequently asked questions