The gap is not the model
A working prototype against a frontier model is a genuinely small amount of code. Twenty lines, an API key, a prompt that took two days to get right. It is impressive, it demos well, and it tells you almost nothing about the six weeks that follow. Nearly every problem we are called in on is at the edges: the request layer, the retry behavior, the cache, the tool loop, the way cost scales with a customer nobody modelled. The model is the part that already works.
This is written for the engineer who has the prototype and is being asked when it ships. It is deliberately specific about the boring parts, because the boring parts are where the incidents come from. None of it is hard. All of it is easy to defer, and deferring it is what turns a two-week feature into a quarter.
You are probably here because
- The feature works, and you cannot say what would tell you it stopped working
- Your bill tripled in a month and nobody can attribute it to a customer or a code path
- Something in the stack retried, and one request was billed and executed twice
- A prompt change went out on a Friday and support noticed before your dashboards did
All four are the same shape of problem: an inference call is a network call with none of the properties your other network calls have.
Budget the request layer before you tune the prompt
An inference call has a latency distribution unlike anything else in your stack. A short answer with a warm cache comes back in a few hundred milliseconds. The same code path with a long input and a reasoning-heavy task can run past a minute. The spread between median and p99 is routinely one to two orders of magnitude, and no instance size fixes it, because it is the shape of the work rather than a queueing artifact.
Three consequences follow immediately. First, pick your client timeout deliberately rather than accepting a default; SDK defaults are generous on purpose and a ten-second default somewhere in your stack will silently cut off your longest and most valuable requests. Second, stream anything that can produce a long output. Streaming does not reduce total latency by a millisecond, but it moves your exposure from one long-held connection to a series of small writes, and it removes the class of failure where a large generation dies against an idle timeout you do not control. Third, if the work can outlive a connection, model it as a job with an id and a status, not a request that happens to be slow.
The threshold we use is about ten seconds at p99. Under it, a synchronous call is simpler for everyone and the simplicity is worth real integration time. Above it you are fighting middleboxes: managed load balancers commonly drop idle connections around sixty seconds, and a customer's proxy is often tighter. An endpoint with an honest ninety-second p99 is not flaky. It is exceeding a timeout somebody configured years ago and never revisited.
Where the engineering time actually goes — our observed split
Rough allocation across the LLM features we have taken from prototype to production. Yours will differ; the ordering rarely does.
The retries are already happening, so make them safe
Every SDK in your stack retries. The official clients retry connection errors, 429s and 5xx by default, usually twice. Your service mesh retries. Your frontend has a person watching a spinner who will refresh. On a normal write path this is fine, because the second write lands on the same row. Here the retry starts a second generation, produces a different answer, and bills you for both.
The fix is the one the payments industry settled on and it transfers without modification. Generate a key on the client, store it at admission alongside a hash of the request body, and put a unique index on the key. The winning insert does the work and saves the response; a losing insert returns the stored response when the body hash matches and rejects the call when it does not, because a reused key with new content is a client bug and quietly serving it hides one. Write the key inside the same transaction that creates the work, or two retries fifty milliseconds apart both start and the index protects nothing.
Retry classification matters as much as the mechanism. A 429 means back off and read the retry hint; a 400 means fix the request and never retry it; a timeout at your client is genuinely ambiguous, because the work may have completed on the server, and that ambiguity is precisely what the key resolves. Add jitter. A synchronized retry across a thousand clients is how a recoverable blip becomes an outage.
Every stop reason is a code path you own
Reading only the text of a response is the most common defect we find in otherwise careful code. The response tells you why generation ended, and the answer changes what you should do. Generation that ended naturally is a complete answer. Generation that hit the output cap is a truncated one, and treating it as complete is how half a JSON object reaches a parser. A response that ends because the model wants to call a tool is not an answer at all. And a request the model declines is a distinct outcome with its own category, which belongs in a metric and often in a human queue rather than in a retry loop.
Handle the cap explicitly. Either raise the ceiling for that route, ask for a shorter output, or split the work — but decide, rather than shipping a parser that fails on the twelve percent of inputs that run long. In our experience truncation is the single most expensive silent failure in this category, because the output looks plausible, nothing goes red, and the defect surfaces in a customer's data weeks later.
Prompt caching is an architecture decision, not an optimization
Caching gets filed under performance work and deferred. That is backwards, because caching constrains how you assemble context, and assembly is hard to change once features depend on it. The mechanism is a prefix match: the cached portion is matched from the start of the request forward, and a single byte changed anywhere inside it invalidates everything after that point. The request renders in a fixed order — tool definitions, then the system prompt, then the messages — so anything volatile early in that sequence poisons the whole prefix.
That gives you one design rule with real teeth. Stable content goes first: the frozen instruction block, a deterministically ordered tool list, a document set that does not change per request. Volatile content goes last: the user's question, the timestamp, the request id. The failure we see most often is a system prompt that interpolates the current time or a per-user greeting into its first paragraph, which produces a zero percent cache hit rate that nobody notices because nothing is broken, only expensive.
Two practical constraints are worth knowing up front. There is a minimum size below which a prefix will not cache at all, in the neighborhood of a thousand tokens, so short prompts get nothing. And the number of cache breakpoints per request is small — four — which means you are choosing a handful of boundaries deliberately rather than sprinkling them. Verify with the usage numbers on the response rather than by reasoning about it: if cache reads are zero across repeated identical requests, something in your prefix is moving, and the usual suspects are an unsorted JSON serialization, a set iteration order, or a clock.
Check the cache read counter in a test, not in a dashboard
Write one test that issues the same request twice and asserts that the second reports a non-zero cache read. It takes twenty minutes and it catches the entire class of silent prefix invalidation, including the ones introduced six months later by someone adding a field to a tool definition. Cost regressions are the only kind of regression that produces no error, no alert and no user complaint — the invoice is the only signal, and it arrives thirty days late.
The tool loop is where agents actually fail
Tool use looks like a solved problem in a tutorial: define a function, the model calls it, you return the result. Production adds four requirements that the tutorial has no reason to mention.
Return every result in one message. When a model requests several tools at once, execute them concurrently and return all of the results together. Splitting them across separate turns is accepted by the API and quietly teaches the model to stop making parallel calls, which shows up as a latency regression nobody can explain.
A failed tool still needs a result. Return the failure explicitly, flagged as an error, with a message the model can act on. Dropping the result leaves a dangling call and the loop degrades in strange ways. Returning a stack trace wastes tokens and tells the model nothing it can use.
Cap the loop and report the count. Every agent needs a maximum number of iterations and a caller-visible count, because otherwise a completed answer and an exhausted budget look identical from the outside. The most expensive bug of this kind we have seen was a tool that returned a subtly wrong error string, which the model retried in a loop until the request deadline, on every single request, for eleven days.
Tool descriptions are prompt text. Rewriting a description for clarity changes behavior. It belongs under code review and under whatever gate you use for prompt changes, not in a docstring cleanup commit.
Evaluation is the release gate, and there is no substitute
This is the part teams skip and the part that decides whether the feature survives. You cannot regression-test a system whose output changes run to run using assertions on exact strings, so you need a set of real inputs with known-good outputs and a scoring function you trust more than you trust your own reading of a sample.
Start smaller than you think. Thirty to fifty examples drawn from real traffic, chosen because they are hard or because they broke something, beat a thousand synthetic ones. Add every production defect to the set as its own case, which is the habit that compounds: after six months the set is a map of every way your feature has actually failed. Score with whatever is cheapest and defensible — exact match on extracted fields, a numeric tolerance, a rubric applied by a second model where the criterion is genuinely subjective, and human review on a sampled slice to keep the automated scorer honest.
Then wire it to something. An eval that runs when someone remembers is not a gate. Ours runs on every change to a prompt, a tool definition, a model selection or a retrieval component, and it publishes a per-case diff rather than a single number, because an aggregate that moves from 0.86 to 0.85 hides the two cases that went from right to catastrophically wrong.
Send us the feature and we will tell you what we would change.
Email the code that assembles your prompt, your retry configuration and a week of latency and token numbers to contact@precisionfederal.com. You get back a short written note naming the three things we would change first and why. One business day, no charge, no meeting.
contact@precisionfederal.comWhat to log, and what to keep
The support ticket you will get is "it gave a weird answer yesterday." Everything below exists to turn that into a query.
| Field | Why you will want it | Retention |
|---|---|---|
| Request id from the provider | The one identifier that lets a vendor support conversation be about a specific call | Long. It is tiny |
| Model identifier and version | Distinguishes a behavior change from a code change; the first question in every regression | Long |
| Prompt version or content hash | Ties an output to the exact instructions that produced it | Long |
| Token usage, split input / output / cached | Cost attribution per feature and per customer; cache hit rate | Long, aggregated |
| Stop reason and latency | Truncation rate and decline rate as first-class metrics, not log spelunking | Long |
| Full input and output text | The only way to reproduce a defect — and your largest privacy surface | Short, sampled, with a policy |
Note the split in that last row. Everything above it is metadata that is cheap to keep and answers most questions. The text itself is what you need for the hardest ten percent of investigations and it is also the thing that will appear in a privacy review. Decide the retention window and the redaction rules on day one, sample rather than storing everything, and write it down. Retrofitting a retention policy onto a year of stored customer text is a genuinely unpleasant project.
The cost model, and where the surprise comes from
Two requests to the same endpoint can differ by three orders of magnitude in tokens. Any limit that counts requests, any quota that counts calls, and any capacity plan built on queries per second is measuring something only loosely related to what you pay. Count tokens.
Output tokens are the expensive half — for current frontier models the output rate is several times the input rate — so verbosity is a cost decision, not a style one. Ask for the shortest form that is still correct, and prefer structured output over prose when a program is the consumer. On the input side, caching moves the economics substantially: a cached read is billed at a small fraction of the base input rate, while writing the cache costs a modest premium over it. A stable, heavily reused prefix is therefore close to free to re-read and worth designing around, and a prefix reused twice is not worth caching at all.
The third lever is the batch endpoint, which trades latency for roughly half the cost on work that does not need to be interactive. Nightly enrichment, backfills, evaluation runs and offline classification all belong there. Results come back unordered and keyed by an id you assign, which is a small amount of plumbing and the most reliable cost reduction available to most teams.
Set a budget guard in code. Not a billing alert — a check in the request path that knows a per-tenant ceiling and refuses cleanly when it is hit. The failure mode this prevents is a single customer, or a single loop, consuming a month of budget in an afternoon, and it is common enough that we now treat its absence as a finding.
The failures we get called about
- Reading response text without checking why generation stopped, so truncated output ships as complete
- A volatile value early in the system prompt, holding the cache hit rate at zero for months
- No idempotency, plus a default-retrying SDK, producing duplicated work and duplicated cost
- An unbounded tool loop with no iteration count in the response, so exhaustion looks like an answer
- Prompt changes deployed outside the release process, with no eval gate and no record of what changed
- Cost tracked in aggregate only, so no one can name the feature or tenant responsible for a jump
- Tool results returned across several messages, silently suppressing parallel tool calls
- Retries with no jitter, converting a brief rate limit into a self-sustaining thundering herd
A two-week hardening pass
From working prototype to something you can page someone about
The last two days are the ones that get cut and the ones that pay. Force a truncated response and watch what your parser does. Send the same idempotency key twice. Return a malformed tool result and see whether the loop terminates. Every defect found in that window would otherwise have been found by a customer.
Common objections
We are pre-product-market-fit. Is this premature?
Most of it is. Two things are not, because retrofitting them is expensive: the eval set and the shape of your context assembly. Both are cheap now and both become entangled with every feature you build on top of them. Idempotency, budget guards and logging can wait until you have users, though logging is the one people most regret deferring, because you cannot investigate an incident with data you did not collect.
Can we skip evals if we have human review in the loop?
Human review catches individual bad outputs. It does not catch a five-point drop in quality across a class of inputs, because reviewers adapt to what they are seeing and have no baseline to compare against. The two are complementary: reviewers handle the tail, the eval set tells you whether the distribution moved. If you only get one, take the eval set — it is the one that runs before the change reaches anyone.
How much of this changes if we switch providers or models?
Less than you would expect. Timeouts, retry classification, idempotency, stop-reason handling, logging, cost attribution and the eval set are all provider-independent, and they are the bulk of the work. What changes is prompt tuning, the exact cache semantics, and the tool-calling wire format. Keep provider-specific code behind a thin adapter and do not build an elaborate abstraction layer for a switch that most teams never make.
Our latency is already too high. What is the first thing to try?
Measure the split between time-to-first-token and total generation time before changing anything. If the wait is in the first token, the answer is usually caching and a shorter input. If it is in generation, the answer is a shorter output, a smaller model for that route, or streaming so the user stops waiting for the whole thing. Those are different fixes and teams frequently apply the wrong one for a month.
Before you call it production
- Every stop reason has an explicit branch, including truncation and decline
- Timeouts set deliberately at every hop, not inherited from defaults
- Retries categorized by kind, jittered, and made safe with an idempotency key
- A test asserts a non-zero cache read on a repeated request
- Tool loops capped, instrumented, and errors returned as tool results
- An eval set from real traffic gates every prompt, model and tool change
- Request id, model version, prompt hash and token split logged on every call
- Cost attributable to a feature and a tenant, with a guard in the request path
- A written retention and redaction policy for stored prompts and outputs
- A documented rollback for a prompt or model change, tested once
Bottom line
The distance between a prototype and a production LLM feature is not model quality and it is rarely prompt quality. It is that an inference call is a slow, non-deterministic, wildly variable-cost network call, and every piece of infrastructure around it was designed for calls that are none of those things. Fix the request layer, design the context assembly around a stable prefix, gate changes on real evaluation data, and make cost attributable. That work is unglamorous, it is measured in days rather than months, and it is the difference between a feature that survives its first thousand users and one that gets quietly turned off.
Frequently asked questions
Thirty to fifty real cases is enough to catch the regressions that matter, and it is small enough to build in a day. Choose them because they are hard, ambiguous, or because they already broke something. Grow the set by adding every production defect as a new case rather than by generating synthetic examples, which tend to cluster in the easy part of the distribution.
Something early in the request is changing between calls. The usual causes are a timestamp or session id in the system prompt, a tool list serialized from an unordered collection, and JSON serialized with non-deterministic key order. Because the cache matches a prefix from the start of the request forward, a single differing byte in the first block invalidates everything after it. Also check the prefix is long enough — below roughly a thousand tokens nothing caches.
For anything not on a user's critical path, yes. It runs asynchronously in exchange for roughly half the cost, which makes it the right home for backfills, nightly enrichment, offline classification and evaluation runs. The plumbing is modest: submit with your own identifier per item, poll until the batch completes, and key results by that identifier because they come back in arbitrary order.
Enforce a per-tenant token ceiling in the request path and return a clean, documented error when it is hit. Billing alerts tell you after the money is spent; an admission check stops it. Count tokens rather than requests, since two calls to the same endpoint can differ by a factor of a thousand in cost.
Code that reads the response text without checking why generation stopped. When output hits the token cap the text is truncated but perfectly well-formed prose, so it passes review, reaches a parser or a customer, and fails somewhere far from the cause. One branch on the stop reason removes the entire class.