The invoice is one line and it explains nothing
Accelerated compute arrives on the bill as one figure. Nobody in the room can say what a single request costs, which requests cost the most, or whether the last three deploys made it better or worse. So the conversation turns to the only visible variable, the price of a GPU-hour, and the team spends two months negotiating a rate on hardware that is idle two thirds of the time. That is the wrong end of the problem.
Cost per inference is division. You need two numbers: the fully loaded hourly price of the machine, and the useful work it completes per hour. Almost every team has the first. Very few have the second, and the gap is where the money hides. Once the second number exists you can watch a change move it and price that change in dollars instead of adjectives.
Cost per million output tokens, from two numbers you can measure today
Take the hourly price of the instance, divide by 3,600 for dollars per second, then divide by the output tokens per second the replica sustains under real concurrency. Multiply by a million. A card at $10 per hour sustaining 2,000 aggregate output tokens per second costs about $1.39 per million output tokens. The identical card sustaining 200 tokens per second, because requests are served one at a time, costs $13.90. Ten times the unit cost, invisible on the invoice.
You are probably here because
- The compute line went up again this month and nobody can say which change did it.
- Nobody can tell you what one request costs, so every conversation turns into an argument about the GPU hourly rate.
- Concurrency stalls at a handful of simultaneous users, and the faster card you moved to did not shift it.
All three usually come from one root cause, which is that nobody is measuring how much useful work the hardware completes per hour; the next two sections give you that number and the section on the KV cache explains the concurrency wall.
Measure cost per successful task, not cost per token
Tokens are the unit the hardware understands, not the unit your business pays for. One user-visible answer in a retrieval system is often an embedding call, a reranking pass, a generation, and sometimes a second generation after the first fails validation. An agent loop that plans, calls three tools and writes a summary is five or more generations, each carrying the conversation forward as input. The token count on the last call tells you almost nothing about the interaction.
Instrument at the task boundary. Give every user-visible interaction a trace identifier, attach it to every model call underneath, and sum input tokens, output tokens and wall time across the tree. Then divide the day's spend by the day's successful tasks. That is the dashboard number, because it moves when retries rise, when a prompt template grows, when an agent loops, and when someone quietly raises the context window. Cost per token hides all four.
Two details are usually missed. Abandoned requests still consumed GPU time, so they belong in the numerator and not the denominator. And a generation cancelled mid-stream consumed everything up to the cancellation, which is why an aggressive front-end timeout can double the bill without producing one more answer.
Prefill and decode are two different machines
Serving a transformer has two phases with opposite performance characteristics, and treating them as one thing is the root of most bad capacity decisions.
Prefill processes the prompt. Every token in the input is available at once, so the work is a set of large matrix multiplications that keep the tensor cores busy. Prefill is compute bound. It scales roughly with input length until the context gets long enough for the quadratic attention term to dominate, and it is the phase that determines time to first token.
Decode generates the answer one token at a time. Each step depends on the last, so there is no parallelism inside a sequence. To produce one token the GPU reads every weight in the model out of high-bandwidth memory, does a little arithmetic with it, and discards it. Decode is bound by memory bandwidth, not floating-point throughput, and extra compute on the die will not fix it.
That has a consequence you can compute before buying anything. A 70-billion-parameter model in bfloat16 is 140 GB of weights. Split across two 80 GB accelerators, each card reads its own 70 GB every decode step. At 3.35 TB/s per card the floor is roughly 21 milliseconds per token, about 48 tokens per second on one stream. Renting a faster card does not beat that unless it has more bandwidth. You beat it by reading fewer bytes per token, which is quantization, or by amortizing the read across many sequences, which is batching.
Memory Bandwidth, TB/s — the ceiling on decode
Vendor-published peak bandwidth. Single-stream decode speed tracks this column far more closely than tensor-core throughput.
The KV cache is your real capacity limit
Weights are the number everyone quotes. The key-value cache decides how many users fit on the card. Every generated token leaves a key and a value vector in every layer, resident for the life of the request. The size is fixed by architecture: two tensors, times layers, times key-value heads, times head dimension, times bytes per element, per token.
For a 70B-class model with 80 layers, 8 grouped key-value heads and head dimension 128, that is 320 KiB per token in half precision. An 8,000-token conversation holds about 2.6 GB. Thirty-two of those need roughly 84 GB, more than an 80 GB card has left after the weights. That is why a deployment refuses to go past a handful of concurrent users, and why the fix is almost never a faster GPU.
Grouped-query attention is why those numbers are tolerable at all. An older architecture with 40 full attention heads at the same head dimension holds around 800 KiB per token. If you are still serving a multi-head model at long context, that is a cost decision you may not know you made.
Memory Footprint, GB — 70B-class model on 80 GB cards
Computed from parameter count and the standard KV formula at 80 layers, 8 KV heads, head dim 128. Run it for your own model before sizing a fleet.
Two moves follow. Quantize the cache itself, usually to 8-bit, which roughly halves it and is far less quality-sensitive than quantizing weights. And use a serving stack with paged cache allocation, so memory is handed out in fixed blocks rather than one contiguous worst-case reservation per request. The PagedAttention work published with vLLM at SOSP in 2023 exists because that reservation wasted most of the cache.
Utilization is most of the answer
A GPU that is waiting costs exactly what a GPU that is working costs. If traffic is bursty, if requests are served one at a time, if replicas are provisioned for a peak that happens twice a day, most of the bill is for idle silicon. No quantization scheme recovers that, and a cheaper hourly rate recovers a fraction.
The trap is that the obvious metric is wrong. The GPU utilization figure from nvidia-smi is the share of sampling intervals in which at least one kernel was executing, so one tiny kernel running throughout reads as 100 percent. Teams autoscale on it, see it pinned high, conclude the cards are saturated, and add replicas. Read the DCGM profiling fields instead: streaming-multiprocessor activity, tensor-pipe activity, memory activity. Those say whether the machine is working or merely occupied.
Scale on queue depth and admission wait, not on a utilization percentage. Queue depth says directly that demand exceeds capacity, responds immediately, and maps to the latency objective you promised. Utilization is a lagging proxy.
Where to Spend Engineering Attention First
Attention weights summing to 100 for a typical untuned deployment. Re-rank once you have measured your own, then work top down.
Continuous batching is the largest software lever
Static batching collects a fixed number of requests, runs them together, and waits for the longest to finish before starting the next group. Because generation lengths vary by an order of magnitude, most of the batch idles waiting on one long answer, and a request arriving a millisecond after the batch closes waits for all of it.
Continuous batching schedules at the token step. A finished sequence leaves the batch immediately and a queued request takes its slot on the next iteration. Iteration-level scheduling came out of the Orca work at OSDI in 2022 and is now standard in vLLM, TensorRT-LLM and Hugging Face TGI. The vLLM paper reports two to four times the throughput of the systems that preceded it at comparable latency, with larger gains on longer sequences. If your stack does not do this, changing stacks is the highest-return week available to you.
Two refinements follow. Chunked prefill splits a long prompt into pieces and interleaves them with decode steps, so one large prompt stops stalling every in-flight generation. Prefill-decode disaggregation runs the two phases on separate pools sized independently, the right shape when traffic has long prompts and short answers, or the reverse, because the phases stop competing for the same silicon.
Precision: what it buys and what it costs
Quantization cuts the bytes read per decode step and frees memory that becomes KV cache, raising the batch you can hold. Both effects push the same way. Gains are largest where you are memory bound, at low-to-moderate batch decode, and shrink as batch rises and the work turns compute bound.
| Format | Bytes / param | Where it wins | What to watch |
|---|---|---|---|
| BF16 / FP16 | 2 | The reference. Use it to define quality before changing anything else | Twice the memory traffic of FP8 for identical output |
| FP8 | 1 | Hopper-class and newer hardware with native support. Usually the best quality-per-byte trade available | Needs calibration; activation outliers in some layers still want higher precision |
| INT8 weight and activation | 1 | Broad hardware support including older data-center parts | Outlier channels; per-channel or per-group scales are not optional |
| INT4 weight-only | 0.5 | Single-stream latency and fitting a large model on fewer cards | Gain collapses at large batch, where you become compute bound again |
| KV cache at 8-bit | n/a | Roughly doubles concurrent sequences at a given memory budget | Generally more forgiving than weight quantization; still measure at long context |
The discipline that makes this safe is boring and non-negotiable. Change one precision setting at a time, re-run a fixed set of several hundred real requests with agreed answers, and record the quality number and the cost number for that configuration. Teams that flip four flags in one deploy and get a complaint two weeks later cannot attribute it, so they revert everything and lose the gains with the regression.
Two levers that remove work instead of speeding it up
Prefix caching. If a large block of your prompt is identical across requests, and in most production systems it is, that prefill work is being repeated on every call. A system prompt, a tool schema, a few-shot block and a document set that repeats across a conversation are all shared prefixes. Caching the computed keys and values removes prefill entirely for the shared portion. On a workload with a 3,000-token system prompt and 200-token user turns, that is most of the input processing gone. The requirement is that the prefix be byte-identical and at the front, so anything variable, a timestamp or a user identifier, goes at the end of the prompt. That ordering decision is worth real money.
Speculative decoding. A small draft model proposes several tokens, the large model verifies them in one forward pass, and accepted tokens are kept. Verification is a single pass over a short sequence instead of several sequential passes, so bandwidth cost per accepted token falls. The original work reported roughly two to three times faster generation on the tasks it evaluated, and the result depends entirely on acceptance rate, which depends on how well the draft model matches the target on your distribution. It helps most at low batch, where the card is bandwidth starved, and least at high batch. Measure acceptance on your own traffic first, because a poorly matched draft model makes things slower.
Send it over and we will tell you what we would change.
Email your serving config (engine, max batch size, max model length, tensor-parallel degree), the instance type you run it on, and one day of token counts with p95 latency to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comRight-sizing the accelerator
The most common hardware error we find is a model that does not need a flagship card sitting on one. An 8B model in half precision is 16 GB of weights. On an 80 GB flagship it uses a fifth of the memory and a fraction of the compute, and you pay for both. It belongs on a mid-tier card, or several copies belong on one flagship partitioned into isolated instances, which the data-center parts support at up to seven per card.
Work the sizing in this order. Compute weight footprint at your chosen precision. Compute the KV cache for your target concurrency at your real context length, from the formula rather than a guess. Add both, add ten to fifteen percent for activations and fragmentation, and that is the memory requirement. Then pick the cheapest card that clears it with the bandwidth to hit your latency target. Tensor parallelism comes after that, and it is a cost rather than a feature: it adds a collective communication step to every layer.
| Lever | Mechanism | Effort | Returns nothing when |
|---|---|---|---|
| Continuous batching | Schedules per token step, so finished sequences leave and queued ones join mid-flight | Days, mostly configuration or a stack change | Concurrency is genuinely one request at a time |
| Prefix caching | Skips prefill for a shared prompt prefix already resident in cache | Days, plus a prompt-ordering change | Every prompt differs from the first token |
| Weight quantization | Fewer bytes read per decode step, more memory left for KV | Days to weeks including evaluation | You are already compute bound at large batch |
| Speculative decoding | Drafts several tokens, verifies them in one pass | Weeks, and needs a matched draft model | Acceptance is low, or the batch is already full |
| Model routing | Sends the easy majority of requests to a smaller model | Weeks, needs a router and a quality gate | Task difficulty is uniform across traffic |
| Capacity terms | Buys the same GPU-hour for less money | Days to one purchasing cycle | The fleet is idle, in which case you are optimizing waste |
Buying the hours: three tiers, not one
Once utilization is real, purchase terms are worth attention. Split the fleet into three tiers and buy each differently.
The floor. Capacity that runs every hour of every day belongs on a one-year or three-year commitment, where discounts typically run from about thirty percent to near sixty depending on term, region and the flexibility you give up. Size it to the trough of the demand curve, never the average, because unused commitment is the most expensive capacity there is.
The peak. On-demand covers the gap between the floor and the daily high. Expensive per hour, cheap in aggregate when the floor is sized right. This is where autoscaling on queue depth earns its keep.
The interruptible. Preemptible and spot capacity trades availability for price, commonly sixty to ninety percent below on-demand. It is the wrong home for synchronous user traffic, where a preemption becomes a timeout, a retry and a duplicate charge. It is the right home for everything offline: batch scoring, embedding a corpus, evaluation runs, index rebuilds, fine-tuning. Hosted model APIs offer the same trade, pricing an asynchronous batch tier at roughly half the synchronous rate for the identical model.
Do not run the large model on easy work
Traffic is never uniform in difficulty. In most production systems a large share of requests are classification, extraction, short lookups and formatting, and a smaller share are the hard reasoning cases the big model was chosen for. Sending everything to the biggest model is the most expensive default there is.
The cheapest version of this fix is not a router. It is noticing which calls should not be model calls at all. Parsing a well-formed document with a stable schema is a parser's job. Matching an identifier against a table is a database's job. Deterministic code is orders of magnitude cheaper than a forward pass and needs no evaluation suite to stay correct next quarter.
Where a model is genuinely needed, a cascade works: run the small model first, check the output against a cheap validator such as a schema check or a grounding test, and escalate only the failures. The economics hold at a mediocre pass rate, because an escalated request costs both models, and if the small one handles seven in ten you are well ahead. Build the quality gate before the router. A router without a gate is a quality regression with a cost saving attached.
Cold starts and the price of holding capacity warm
Scale-to-zero sounds like the answer to idle cost until you price the restart. A 70B model at half precision is 140 GB that has to move from object storage onto the card. At one gigabyte per second of effective throughput that is over two minutes before the first token, and shared network storage under contention is often slower. Meanwhile the autoscaler, seeing a queue and no capacity, starts more replicas that all pull the same 140 GB across the same link.
The fixes are known and worth doing before you need them. Cache weights on local NVMe so a restart is a local read. Use a memory-mappable format so the loader is not deserializing the file into host memory first. Keep a small warm pool sized to your worst realistic burst and accept that it costs money, because the alternative is two minutes to first token on the request that mattered. Set cooldowns long enough that a burst does not stampede simultaneous cold loads.
Instrument this before changing anything
- Cost per 1,000 requests and cost per successful task, computed nightly from billing and request counts
- Input and output token counts on every call, stored against a trace identifier spanning the whole task
- Time to first token and inter-token latency at p50, p95 and p99, split by route and prompt-length bucket
- Achieved output tokens per second per replica, aggregated across in-flight sequences, not measured on one stream
- Batch occupancy: sequences the scheduler is actually running against what the memory would allow
- KV cache utilization, plus preemption and recompute events, the early warning that concurrency is collapsing
- Streaming-multiprocessor and memory activity from DCGM, never the nvidia-smi utilization percentage
- Queue depth and admission wait, which is the correct autoscaling signal
Cuts that do not work
- Shopping for a cheaper GPU-hour before measuring achieved throughput on the hardware you already rent
- Turning on four optimizations in one deploy, then unable to attribute the quality complaint that follows
- Autoscaling on nvidia-smi utilization, which reports that a kernel ran, not that the card was busy
- Setting maximum batch size from a memory formula and never checking what the scheduler admits under real traffic
- Moving synchronous user traffic onto preemptible capacity and paying the discount back in retries
- Capping output tokens to save money, producing truncated answers that users resubmit at full cost
- Buying committed capacity before the utilization work, which locks in the waste for a year at a discount
A two-week pass that produces a number
Inference Cost Pass
Two weeks is enough because every unknown here is measurable inside it. Whether the batch scheduler is filling is measurable. Whether quantization moves your quality number is measurable on a replay set. Whether the fleet is idle is measurable in an afternoon. What is not measurable is an argument about which GPU is better value, which is where these projects usually start.
Bottom line
Cost per inference is arithmetic over two quantities: what the machine costs per hour, and how much useful work it completes in that hour. The second is the one nobody measures, and it varies by an order of magnitude between a tuned deployment and an untuned one on identical hardware. Fix utilization first, remove repeated work second, change precision third, right-size the hardware fourth, negotiate the hourly rate last. Put cost per successful task on a dashboard the team sees daily, because a number nobody watches drifts upward.
Frequently asked questions
Divide the hourly instance price by 3,600 for dollars per second, then divide by the throughput the replica sustains under real concurrency, measured across all in-flight requests rather than one stream. Express it per thousand requests or per million output tokens. Then divide daily spend by daily successful tasks, which is the figure that reflects retries and multi-step work.
Input tokens are processed in parallel in one compute-bound pass, so the hardware is efficient. Output tokens are generated one at a time, and each step reads the entire model out of memory for a single token. Decode is therefore bandwidth bound at very low arithmetic intensity, which is why providers price output several times higher than input.
It can, and how much depends on format, calibration and task. FP8 and well-calibrated INT8 usually sit close to the half-precision reference; aggressive 4-bit weight-only quantization is more visible on reasoning-heavy work. Quantizing the KV cache to 8 bits is the most forgiving change available. The only way to know is a fixed evaluation set run before and after, one change at a time.
For offline work, yes. Batch scoring, corpus embedding, evaluation runs and index rebuilds all tolerate preemption and the discount is large. For synchronous user traffic, no: a preemption mid-generation becomes a timeout and a retry, and you pay for the discarded work plus the replacement. Hosted APIs offer the same trade through an asynchronous batch tier.
When you can show, on a labeled set drawn from your own traffic, that the smaller model meets the quality bar on a defined slice of requests. Route that slice to it, keep the large model for the rest, and let a validator decide escalation. Switching wholesale without that evidence trades a cost problem for a quality problem, which is more expensive.