A budget is a set of numbers you agreed to before you built anything
Almost every conversational system we are asked to speed up has the same history. It was fast in the demo with three documents and one user. Retrieval was added, then a reranker, then a safety check, then a second retrieval pass for follow-up questions, and each addition cost two hundred milliseconds that nobody objected to individually. The product now takes four seconds to say anything and the team is arguing about GPUs. The GPU is rarely the problem. The absence of a budget is.
A latency budget is a table. One row per component in the path, a p50 and a p95 allocation for each, and a total that you have committed to. It is written before implementation, it is checked in with the code, and it is the thing an engineer points at when they propose adding a reranker. Without it, every individual addition is defensible and the sum is not.
The first thing the table forces is a decision about which number you are budgeting. Total latency and perceived latency are different quantities, and for anything a person waits on, perceived latency is the one that matters. A response that begins in four hundred milliseconds and finishes in six seconds feels faster than one that appears whole at two and a half. That is not a trick; it is how waiting works. The corollary is that time to first token is the headline number for interactive systems and total time is a secondary constraint.
The human thresholds worth designing against
Three response-time bands have been stable in the interface literature for decades and they transfer cleanly. Under about a hundred milliseconds, a response feels instantaneous and caused by the user. Up to about a second, thought is uninterrupted, though the delay is noticed. Past about ten seconds, attention leaves and the user goes somewhere else, which is why a long operation needs a progress indication rather than a faster spinner.
Voice is stricter and it is stricter for a reason unrelated to computers. The gap between speaker turns in ordinary human conversation clusters around a fifth of a second. People are extremely well calibrated to it. A voice agent with a full second of silence before it starts speaking does not read as slow; it reads as broken or inattentive, and users start talking over it, which then breaks turn-taking properly. For a voice product, treat roughly eight hundred milliseconds from end of user speech to first audio as the outer edge of acceptable, and understand that a good share of that budget is gone before the model is even called.
You are probably here because
- Your median looks fine and users still call the product slow
- Somebody proposed a bigger instance and you suspect that is not it
- Latency doubled after a feature that should have cost nothing
- A voice agent talks over people, or waits so long they repeat themselves
The percentile section covers the first, the component table the second and third, and the voice notes the fourth.
Where a turn actually goes
Below is the component list for a retrieval-backed chat turn, with the ranges we typically measure. Treat these as starting allocations to argue with, not as facts about your system — the point of the table is that every row has an owner and a number.
| Component | Typical p50 | Typical p95 | What moves it |
|---|---|---|---|
| Client to edge, TLS, auth | 20–80 ms | 150–600 ms | Mobile networks, cold connections, distance to the nearest edge |
| Input guard / moderation | 30–150 ms | 200–500 ms | Whether it is a model call and whether it blocks the main path |
| Embedding the query | 15–60 ms | 80–250 ms | Model size, batching, whether it is a network hop |
| Vector search | 10–80 ms | 50–400 ms | Index type, recall target, filters, cold shards |
| Reranking | 80–300 ms | 300–900 ms | Candidate count — the single biggest knob in the pipeline |
| Admission wait / queueing | 0–100 ms | 300 ms–5 s | Utilization. Non-linear and usually the surprise |
| Prefill (to first token) | 200–900 ms | 0.6–3 s | Input length, prefix cache hit, model size |
| Decode (per token) | 8–40 ms | 20–80 ms | Model size, batch pressure, memory bandwidth |
Two observations from that table do most of the work. First, the levers that people reach for — a bigger GPU, a faster model — touch two rows. Second, the row with the widest spread between p50 and p95 is queueing, and it is the one nobody budgets.
Queueing is where the budget dies
Take the simplest queueing model available. If arrivals and service are random and the server is busy a fraction of the time, the expected wait scales as that fraction divided by one minus that fraction, multiplied by the service time. At fifty percent utilization the wait equals one service time. At eighty percent it is four. At ninety percent it is nine. At ninety-five it is nineteen.
Real inference servers are not that model — continuous batching means several requests share a forward pass, and the curve is gentler for a while and then worse when memory for the key-value cache runs out. But the shape is right, and the shape is the lesson: latency does not degrade linearly with load, it degrades suddenly. A system comfortable at seventy percent utilization is not seven-eighths as comfortable at eighty.
This is why capacity and latency are one conversation, not two. If your budget assumes a two-hundred-millisecond queue wait, you have implicitly committed to running below a particular utilization, and that commitment costs money in idle capacity. Decide it deliberately. Capacity planning for bursty inference covers the sizing side; the point here is that the number belongs in the latency table with an owner next to it.
The tail is the experience, not the edge case
Averages are useless here and p99 is not a corner case. A conversation is a sequence of turns, and a user experiences the worst of them. If one turn in a hundred is slow, an eight-turn conversation has roughly a one-in-thirteen chance of containing a slow turn — about eight percent of conversations. Twenty turns, and it is nearly one in five. Users do not average their experience across turns; they remember the bad one and describe the product by it.
Which means the budget is written in percentiles, per turn, and validated at the conversation level. Publish a p50 and a p95 target for time to first token, a p95 for total turn time, and measure the distribution rather than a single aggregate. And measure it where the user is: server-side percentiles exclude the client network, the cold TLS handshake, and the rendering, which on a phone on a poor connection can be a second of the experience your dashboards never saw.
Share of a typical retrieval-backed turn, before optimization
Share of time to first token, p95, on the systems we have profiled. Yours will differ — the value of the exercise is finding out where yours differs.
Prefill and decode are two different machines
Serving a token has two phases with almost opposite characteristics, and conflating them produces optimizations that do nothing.
Prefill processes the whole input to produce the first token. It is compute-heavy and it scales with input length, so it is the phase that punishes a long system prompt, a large retrieved context, or a conversation history you never truncate. Doubling the prompt roughly doubles the time to first token. This is the phase a prefix cache attacks, and on a long stable prompt the improvement is large enough to change a product decision.
Decode emits one token at a time and is limited mostly by memory bandwidth rather than raw compute. Its cost is per output token and is close to constant, which is why the reliable way to shorten a long response is to ask for a shorter one. Telling the model to answer in three sentences is not a prompt-engineering nicety; on a five-hundred-token answer at twenty milliseconds a token it is worth several seconds.
So the two headline levers are different. If time to first token is your problem, shorten the input, cache the prefix, and cut retrieved context. If total time is your problem, shorten the output and stream so the user does not wait for it.
The levers, ranked by what they return
| Lever | What it moves | Realistic gain | What it costs |
|---|---|---|---|
| Stream the response | Perceived latency only | Often the largest single improvement in felt speed | Client complexity; see the streaming UX article |
| Cut retrieved context | Prefill, and cost | 20–50% of TTFT when context was generous | Recall risk — must be measured, not assumed |
| Prefix caching | Prefill on repeat structure | Large on long stable prompts; nothing on short ones | Prompt ordering discipline, cache warming |
| Shorter outputs | Total time, linearly | Proportional to tokens removed | Sometimes a worse answer. Test it |
| Smaller model | Prefill and decode | 2–5× on both, task permitting | Quality, on some slices. Route rather than switch |
| Parallelize the pipeline | Wall clock | The sum of anything not on the critical path | Concurrency bugs, harder tracing |
| Run below 70% utilization | The p95 and p99 tail | Frequently the biggest tail win available | Idle capacity, paid for monthly |
The one at the bottom is the one teams resist and it is often the correct answer. Buying headroom is unglamorous, it appears on a cloud bill rather than in a design document, and it fixes tails that no amount of code will.
Put the deadline in the request and let every stage read it
Pass an absolute deadline through the whole path rather than a per-call timeout. Each stage checks the remaining time and adapts: skip the reranker under three hundred milliseconds left, drop retrieved passages from ten to four, fall back to a cached answer, return what exists. Without a propagated deadline, a slow stage is followed by stages that spend their full allotment on a request that is already late, and the user waits for work nobody will read. This one change usually does more for the p99 than any model swap.
Send us a trace and we will tell you where the seconds went.
A handful of spans from a slow turn, your p50 and p95 for time to first token, and the pipeline in one paragraph. Email contact@precisionfederal.com. You get back a component budget with the three lines we would attack first, in writing, in one business day. No charge and no meeting.
contact@precisionfederal.comAgent turns multiply the budget
A single-shot answer has one model call. An agent turn that searches, reads a document and then answers has three sequential calls plus two tool round trips, and the sequence is the problem: the latencies add, and so do the tails. Three calls each with a one-in-a-hundred slow case give you roughly a one-in-thirty-four slow turn.
Three things help and all three are structural. Budget per step and enforce a step cap, so a loop cannot consume forty seconds discovering it is stuck. Run independent tool calls concurrently rather than in a chain, which is free wall-clock time in most agent frameworks and is left on the table constantly. And show the user the machinery — a status line naming the current step converts dead air into visible progress, and does more for tolerance than shaving a second would.
Degrade rather than fail
A budget is only real if something happens when it is exceeded. Decide in advance what gets dropped and in what order, and write it as a list the on-call engineer can read at three in the morning.
- Reranker off when the remaining deadline falls below its p95
- Retrieval depth reduced from ten passages to four, measured for recall loss beforehand
- Second retrieval pass skipped on follow-up turns
- Route to the smaller model when queue depth crosses a threshold, and record it on the response
- Return partial output with a clear marker rather than discarding a stream at the deadline
- Shed load at admission with an honest wait estimate, rather than accepting work you cannot serve
The last one is the hardest to get agreement on and the most valuable. A queue that accepts everything under load converts a capacity problem into a total outage, because every request is now slow enough to be retried, and retries are new load. Rejecting ten percent of requests quickly is a better product than serving all of them in thirty seconds.
How to measure it so the numbers mean something
Instrument the client, not only the server. Record time to first token separately from total time. Tag every measurement with the model, the prompt version, the cache-hit status and whether a degradation path fired, or you will be unable to explain a shift. Keep a small synthetic probe that runs a fixed prompt on a schedule, because it separates "our system got slower" from "our traffic got harder", and those have different fixes.
Then look at the distribution rather than the aggregate. A bimodal histogram — a fast mode and a slow mode with nothing in between — almost always means two code paths, usually cache hit against cache miss or warm shard against cold. That is a much more actionable finding than a p95 that drifted up, and an average hides it completely.
The mistakes we are called in to fix
- Optimizing total latency for a streaming product, where only time to first token is felt
- A budget with no queueing line, in a system that runs at eighty-five percent utilization
- Server-side percentiles only, hiding a second of mobile network and cold TLS
- A blocking safety check on the critical path that could have run concurrently
- Reranking fifty candidates because that was the tutorial default
- Timeouts per call instead of one propagated deadline, so late requests still pay full price at every stage
- Conversation history never truncated, so time to first token grows all session
- Sequential agent tool calls that had no dependency on each other
A one-week latency audit
Latency Audit
Day two is the one that pays. In most audits the histogram answers the question before any optimization starts: a fast mode and a slow mode, separated by a cache miss, a cold shard, or a long-input slice nobody had noticed was a quarter of traffic. Teams that skip straight to tuning spend the week improving the mode that was already fine.
Common objections
Our users tell us they do not mind waiting.
They report that and their behavior usually disagrees. Abandonment during the wait, the stop-button rate and the share of sessions with a single turn are the honest measurements, and they move with latency more reliably than survey answers do. Ask what people say, then check what they did.
Is it worth optimizing before we have traffic?
Writing the budget is worth it immediately, because it is free and it changes design decisions. Optimizing is not: at low traffic the queueing term is zero and you would be tuning the part of the system that scales fine. Write the table, take the structural decisions that are expensive to reverse — streaming, propagated deadlines, prompt ordering for cache reuse — and leave the rest.
Can we hide the wait with a better loading animation?
Up to a point, and the point arrives sooner than designers expect. A specific status line beats a generic spinner because it tells the user something is happening and roughly what. Neither survives past a few seconds of nothing, and neither helps if the wait is unpredictable, since users tolerate a slow response far better than an inconsistent one.
What if the model provider is the slow part?
Measure it separately before believing it, because provider time and your own queueing look identical from the outside. If it really is upstream, the levers left are input length, prompt caching, output length, model choice and region — all of which are yours. Publish the provider component as its own budget row so the conversation stays factual.
Bottom line
Write the table before you write the code. One row per component, a p50 and a p95 per row, a named owner, and a total the product has agreed to. Budget time to first token as the headline for anything interactive and treat total time as a separate constraint. Expect queueing to be the line you underestimated and the tail to be the thing users describe. Then make the budget enforceable by propagating a deadline and deciding, in advance and in writing, what gets dropped when the clock runs out. A system that degrades on purpose is faster in the ways people notice than one that is theoretically fast and occasionally takes nine seconds.
Frequently asked questions
Under about eight hundred milliseconds at p50 and under two seconds at p95 keeps a text interface feeling responsive when the answer streams. For voice, the bar is roughly eight hundred milliseconds at p95 from the end of user speech to first audio, because people are finely tuned to conversational turn gaps and read a longer pause as a fault.
Usually queueing. Wait time rises non-linearly with utilization, so a server that is comfortable at seventy percent can be four or five times worse at ninety, and that shows up almost entirely in the tail. The other common causes are cold caches, cold index shards, and a small share of requests with much longer inputs than the rest.
It makes nothing faster and it makes the product feel substantially faster, which is usually what you were asked to fix. It also lets a user judge the answer early and cancel a bad one, which saves both their time and your tokens. Streaming is not a substitute for a real time-to-first-token target, because a stream that starts in three seconds still feels broken.
Both, for different symptoms. Input length drives the time to the first token because prefill processes the whole prompt. Output length drives total time, at a roughly constant cost per token. If users complain about the wait before anything appears, cut input and cache the stable prefix; if they complain about how long the whole thing takes, cut output length.
Sometimes, and it is the first suggestion far more often than it is the right one. It touches prefill and decode, which in the systems we profile account for roughly a third of the time to first token. Retrieval depth, reranker candidate counts, propagated deadlines and running with more headroom generally return more, and cost less.