Caching for LLM applications

Five caches, one word

When somebody says "we should cache the model calls", they could mean any of five things with wildly different risk profiles. Two are close to free and safe. One is a large win that most teams get partly right by accident and could get fully right with an hour of prompt reordering. One is safe but small. And one — semantic caching — is a mechanism for returning an answer to a question the user did not ask, which is a category of bug your monitoring will not catch because a served response looks like a success.

Cache	What it stores	What it saves	Risk if wrong
Prefix / prompt cache	Provider-side attention state for a repeated prompt prefix	Input token cost and a large share of time to first token	Low — a miss is only slower
Exact-match response	The full response for an identical request	The whole call	Moderate — staleness, and cross-user leakage if the key is wrong
Semantic / similarity	Responses keyed by embedding proximity	The whole call, more often	High — serves a wrong answer that looks right
Embedding cache	Vectors for text you have already embedded	Re-embedding cost at corpus scale	Low — deterministic per model version
Pipeline stage cache	Parsed documents, retrieval results, tool responses	Latency and third-party call cost	Depends entirely on the freshness of the underlying source

This article is about the specific behavior of each. The general discipline — staleness budgets, invalidation, stampede control, the honest way to measure — is in caching strategies that do not lie, and everything there still applies.

The prefix cache is a prompt-ordering problem

Providers cache the computed state of a prompt prefix so that a later request beginning with the identical bytes can skip recomputing it. The saving is real on both axes: a substantial discount on the cached input tokens, and a large cut in time to first token, because prefill is the phase that scales with input length.

The catch is that it is a prefix match. Everything before the first differing byte is reusable; everything after it is not. Which means the single most effective thing you can do is reorder your prompt so the stable parts come first.

The order that works, from most to least stable: system instructions, tool definitions, few-shot examples, any long document or corpus the whole conversation refers to, then conversation history, then the current user turn. The order we routinely find in real code puts a timestamp, a request ID, a personalized greeting or a randomized example ordering somewhere near the top — and a single moved byte in position forty invalidates a four-thousand-token prefix on every request. That is not a small tuning issue; it is the difference between paying a discount rate on most of your input tokens and paying full rate on all of them.

Four operational details decide whether the win holds. Cache lifetime is short — typically minutes rather than hours — so a low-traffic endpoint may never hit it, and a warming request on a schedule can be worth more than it costs. There is usually a minimum prefix length, below which nothing is cached at all, so short prompts get no benefit and the optimization is wasted effort there. Only input is discounted; output tokens cost the same, so a workload dominated by long generations will see less than the headline number. And the cache is per model and per provider, so any traffic split fragments it — the interaction with multi-model routing is the term most often missing from a routing business case.

Exact-match response caching: safe, small, and mostly about the key

Storing the full response for a byte-identical request is the easiest cache to reason about and the hardest to get a good hit rate from. In open-ended consumer traffic, exact repeats are rare — low single digits is a normal hit rate, and anything above twenty percent usually means a head of common questions worth handling deliberately. In internal tooling with templated inputs, the same mechanism can hit most of the time, because the inputs really are identical.

Everything here rides on the key. A cache key that omits an input is not a cache; it is a mechanism for serving one user's answer to another. The key must contain, at minimum:

• The full prompt bundle — system text, tools, few-shots, output schema, not just the user message
• The model identifier including the snapshot, not a moving alias
• Decoding parameters — temperature, top-p, max tokens, stop sequences
• The prompt bundle version or content hash, so a prompt change cannot serve old behavior
• The retrieval corpus version if documents are in the prompt
• The tenant, and the requesting user’s permission set
• Locale, formatting mode, and anything else the output visibly depends on

The permission entry is the one that ends projects. If two users of the same tenant can see different documents, and your retrieval-backed answer is keyed on the question alone, then the cache is a channel for showing one of them the other's data. It will not look like a breach in any log; it will look like a cache hit. Hash the effective permission set into the key, or partition the cache by it, and test the negative case explicitly with two accounts that should see different things.

Semantic caching: where the wrong answers come from

The pitch is appealing. Embed the incoming question, find a previously answered question within some cosine distance, return the stored answer. The hit rate is much better than exact match. The mechanism is also, in its naive form, a bet that vector similarity implies answer equivalence, and it does not.

Consider pairs that sit very close in embedding space and have opposite correct answers. "Can I cancel after thirty days" and "can I cancel before thirty days". "Is the fee waived for annual plans" and "is the fee waived for monthly plans". "What did revenue do in Q2" and "what did revenue do in Q3". Negation, quantity, entity, and time are exactly the dimensions general-purpose embeddings compress hardest, because for retrieval purposes those documents really are about the same topic. Topical similarity is what the embedding was trained to capture. Answer equivalence is a different relation and nobody trained for it.

So the failure is not random noise around the threshold. It is systematic, concentrated on the questions where being wrong matters most, and invisible in your metrics because a served cache hit records as a success. If you measure only hit rate, a semantic cache always looks like it is working.

If you build one anyway — and there are places it is genuinely right — constrain it hard:

• Restrict it to a curated intent set with known-stable answers, rather than open traffic
• Exclude anything containing a number, a date, a negation or a named entity from cache lookup
• Set the threshold from a labeled set of paraphrase and near-miss pairs you built yourself, never from a default
• Verify the hit with a cheap check that the stored answer addresses this question, and treat the verification cost as part of the saving
• Measure the wrong-answer rate by sampling hits and grading them, and publish that number next to the hit rate
• Never cache across tenants, and never cache anything a user could have personalized

Our honest position: for most products, a well-keyed exact-match cache over a curated head of common questions delivers most of the benefit with none of the exposure, and the engineering time that would have gone into tuning a similarity threshold is better spent on the prefix cache.

Cache the deterministic stages, not the model

The best-return caching in a language-model pipeline is usually not on the model call at all. It is on the expensive deterministic work around it, where a hit is exactly equal to a miss and there is no correctness question to argue about.

Embeddings. For a fixed model version, embedding is deterministic. Key on a hash of the text plus the model identifier and you never pay twice. At corpus scale this is the difference between a re-index costing hours of compute and costing minutes, and it makes experimenting with chunk sizes affordable.

Document parsing. Optical character recognition, table extraction and layout analysis over a PDF are slow and expensive and the input never changes. Key on the file's content hash, not its name or path, because the same document arrives repeatedly under different names.

Retrieval results. For a fixed query, index version and filter set, the retrieved passage list is reproducible. Caching it saves the search and often the reranker, which is frequently the largest latency line in the pipeline. Include the index version in the key so a rebuild invalidates cleanly.

Tool calls. This one is per tool and requires judgment. A currency rate is cacheable for minutes, a filed document for effectively forever, an inventory count for seconds if at all. Write the freshness requirement per tool as a number in the code next to the call, not in a document. A tool cache with one global time-to-live is a bug waiting for the one tool whose data moves fastest.

Every cache extends your rollback window

This is the interaction teams discover during an incident. You ship a prompt fix at ten in the morning. The bad behavior keeps appearing until two in the afternoon, and no deploy explains it. The cache is serving pre-fix answers for its whole time-to-live, and it will keep doing so for every key that was warm.

There are two ways to handle it and one of them is clearly better. You can purge on every prompt or model change, which is a coordination problem, tends to be forgotten, and produces a cold-cache latency spike at the worst moment. Or you can put the prompt bundle hash and the model snapshot into the key, in which case a change is instantly a full miss on the new version, a rollback is instantly a full hit on the old one, and nothing needs purging or remembering. The second costs you a cold period after each release and buys correctness by construction. Take it. The bundle-hash concept and what belongs inside it is covered in prompt versioning and rollback.

Measuring a cache honestly

Hit rate is a vanity metric on its own, because a cache with a ninety percent hit rate serving stale or mismatched answers is worse than no cache. Report five numbers together:

• Hit rate per cache, separately — prefix, exact, semantic, stage — never aggregated into one figure
• Dollars saved per month, computed from real token prices, not a percentage
• Latency delta at p50 and p95 between hits and misses, so the latency case stands on its own
• Age distribution of served hits, which is your actual staleness rather than your configured time-to-live
• Wrong-answer rate on any similarity-based cache, from graded samples, published beside the hit rate

Then add one operational habit: run evaluations with the cache bypassed. An eval suite that hits a warm cache is measuring your cache, not your system, and it will report that a regression did not happen because it never called the model. Provide an explicit bypass flag and default your eval harness to it. This is the same discipline as evaluation infrastructure for production models and it is broken more often than any other part of a harness.

The stampede case is worse here than elsewhere

When a popular cache entry expires and a hundred requests miss it simultaneously, an ordinary service issues a hundred database queries and recovers. An inference stack issues a hundred generations, each taking seconds, against a fixed pool of accelerators that is already sized for steady state. The queue backs up, latency climbs across every unrelated request sharing the pool, and the recovery is slow because in-flight generations cannot be cancelled cheaply.

The mitigations are standard and they matter more here: take a lock so one request regenerates and the rest wait for it, add jitter to expiry times so keys do not expire together, and serve stale content while revalidating in the background where the staleness budget permits. Refreshing shortly before expiry rather than after is worth the extra work on a small number of hot keys.

The mistakes we are called in to fix

• A timestamp or request ID at the top of the system prompt, destroying prefix reuse on every call
• A cache key without the model snapshot, so a provider model change served answers from the old one
• A cache key without the permission set, so one user saw another user’s document summary
• A semantic threshold copied from a tutorial, never validated against negation or numeric pairs
• Hit rate reported with no wrong-answer rate, making a similarity cache look free
• Evaluations running against a warm cache, reporting a regression that had not been tested
• One global time-to-live across every tool, so the fastest-moving data was the stalest
• No prompt version in the key, so a fix took the full time-to-live to take effect

A one-week caching pass

Day one usually settles the priorities on its own. A pipeline that sends a large stable context on every call is spending most of its bill on input tokens, in which case reordering the prompt is worth more than everything else on the list combined. A chat product generating long answers is spending it on output, where caching helps much less and the honest advice is to look at output length and model choice instead.

Common objections

Bottom line

Get the prefix cache right first: it is nearly free, it improves cost and time to first token together, and it is usually a matter of moving three lines of prompt assembly. Cache the deterministic stages next, because a hit and a miss are identical and there is nothing to argue about. Use exact-match response caching where the key can be made complete, and be honest that the hit rate will be modest. Approach semantic caching as a correctness decision rather than a cost decision, restrict it hard if you use it at all, and publish its wrong-answer rate beside its hit rate. And put the prompt bundle hash and model snapshot in every key, so the day you need to roll something back, the cache rolls back with it.

Frequently asked questions