Caching strategies that do not lie to you

Every cache is a promise nobody wrote down

When you put a cache in front of something, you are making a promise: a stale answer is acceptable on this path, for roughly this long, in exchange for this much latency and this much load taken off the origin. Almost nobody writes the promise down. The team ships the cache, the dashboard turns green, and the terms of the promise get discovered later during an incident, when a customer is looking at a price that changed forty minutes ago or a revoked user is still holding a session that should have died at 9:14. The fix is not a smarter eviction policy. It is stating the promise first and then building the cache that keeps it.

There are three questions per cached read path, and they take about ten minutes to answer. How stale may this be, in seconds? What does it cost when it is wrong, and who absorbs that? How would we know? A path that cannot answer the first does not get a cache yet. A path that cannot answer the third gets one that will lie for months before anyone notices, because a stale cache throws no exceptions. It returns 200 OK, quickly, with the wrong body.

Hit rate is the wrong number to lead with

Hit rate is the first metric every cache library exposes and the least informative one you can look at. It says nothing about correctness and surprisingly little about load. A path serving 20,000 requests per second at a 95 percent hit rate still sends 1,000 requests per second to the origin, and those misses are not spread evenly through the second. They cluster hard at expiry boundaries, which is exactly where the origin is least able to absorb them.

Worse, hit rate improves when the cache is wrong. Extend every TTL by 10x and the dashboard looks better, because nothing on it degrades as the data ages. That is a measurement system pointed away from the failure mode.

Four numbers tell you what is actually happening. The age of what you served, as a histogram, not an average. The divergence rate between the cached answer and a fresh one. The p99 latency of a miss, because misses are the user-visible tail. And origin request rate at its peak relative to its mean, which is the number that predicts whether an expiry event takes the database down. Hit rate is worth tracking per key class, and it belongs near the bottom of the list.

Write the staleness budget before you pick a TTL

A TTL chosen without a budget is a number somebody typed. Five minutes appears in more codebases than any other duration, and in most of them it means only that five minutes felt safe. Start from the other end: how old can this answer be before it causes a problem someone files a ticket about? Write that number in seconds, get the product owner to agree with it, and put it in the code next to the TTL. Then derive the TTL so total observed age stays inside it.

Data class	Typical staleness budget	Who notices, and how fast	Right mechanism
Permissions, roles, session validity	Seconds, or event-driven	Security review, and the customer whose access you failed to revoke	Push invalidation on the write path, with a short TTL as a backstop
Price, availability, anything at checkout	5–60 seconds	The customer, immediately, at the worst moment	Short TTL plus invalidate-on-write, single-flight on misses
Search results, listings, feeds	1–5 minutes	Rarely anyone, unless a deletion stays visible	TTL with jitter, stale-while-revalidate
Dashboards and aggregate reporting	5–60 minutes	Nobody, if the page states the as-of time	Precomputed table or materialized view on a schedule
Configuration and reference data	Minutes to hours	Engineers, during a rollout	Versioned key, changed by deploy rather than expired
Built assets and immutable objects	Unbounded	Never, if the URL contains a content hash	Immutable caching, new content gets a new URL

The last row generalizes. Content-addressed objects need no invalidation at all, because the identity of the object changes when the object changes. Every time you can push a cached item toward that shape, the problem disappears instead of getting harder.

The key is the contract

A cache key is a claim that any two requests producing the same key deserve the same answer. Most cache defects are that claim being false. The inputs that change an answer are almost always more numerous than the ones a developer types into a key at 4 p.m.: tenant or account, the caller's permission set, locale and currency, the API or schema version, the feature-flag variant assigned to the user, the model or ruleset version, every query parameter after normalization, and the version of the code that shapes the response.

Two rules make this tractable. First, keys are produced by one function in one module, never assembled with string concatenation at call sites. Scattered concatenation is how a key acquires four different spellings and how a new query parameter gets added to a handler and forgotten in the key. Second, every key carries a namespace prefix with a schema version, something like v7:search:. When a deploy changes the response shape, you bump the prefix and the entire old generation becomes unreachable in one edit, no purge required, and the dead entries age out under eviction.

The cross-tenant version is the expensive one. A response gets computed after authorization and stored under a key derived only from the request path, so the first user's answer is served to the second, correctly according to the cache and catastrophically according to everyone else. The defense is mechanical: any value produced downstream of an authorization decision carries the authorization inputs in its key, and review treats a key change like a schema migration.

Three honest ways to invalidate, and one that only looks like one

Expiry. The value carries a TTL and dies on its own. Honest because the staleness is bounded and you can state the bound. Weak because the bound is a guess, and it is the same guess for a row that changes hourly and a row that has not changed since 2023.

Versioned keys. Delete nothing. Change the key so the old value becomes unreachable and let eviction reclaim it. This is the strongest mechanism available and it is under-used outside asset pipelines. A key like user:8123:profile:rev41, where the revision is a counter on the row, turns invalidation from a distributed delete into a local read of a number you already have.

Write-path invalidation. The code that mutates the data also deletes or overwrites the cached copy. Necessary wherever the budget is measured in seconds, and it carries a race that catches nearly everyone: a read misses, fetches from origin, and is descheduled; a write commits and deletes the key; the stale read then finishes and writes its pre-write value back. The cache holds a wrong value until its TTL expires, and no amount of correct-looking delete code prevents it. Mitigate with a short TTL backstop, a second delayed delete a few hundred milliseconds after the write, or a write-through under the lock that guards the mutation.

The one that only looks honest: manual purge. A purge endpoint or a dashboard button is a break-glass tool. As the primary invalidation design it makes correctness depend on a human noticing, and the first sign of trouble is a support ticket. Keep the button. Do not build on it.

Stampede is what a cache outage actually looks like

Caches rarely fail by returning errors. They fail by all expiring at once. A hot key dies, several hundred concurrent requests miss together, every one calls the origin, latency climbs under the pile-on, connection pools saturate, and nothing repopulates the cache because every worker is stuck on a query that used to take 80 milliseconds. From the outside it presents as a database outage. The database is fine. It is being asked to do the same work four hundred times at once.

Four mitigations, in the order we usually apply them.

Jitter every TTL. A deploy that warms ten thousand keys in one second with a fixed 300-second TTL creates a wave that re-forms every five minutes, forever. Randomize each TTL by plus or minus 10 to 20 percent. One line of code, and an entire class of periodic latency spikes disappears.

Coalesce concurrent misses. Let one request compute and make the rest wait on its result. Nginx has proxy_cache_lock, Varnish coalesces by default, and Go ships golang.org/x/sync/singleflight; in-process it is a single-flight map keyed on the cache key. Coalescing turns four hundred origin calls into one, and on a hot path nothing pays as well for the effort.

Recompute early, probabilistically. Instead of waiting for expiry, let each reader roll dice weighted by how expensive the last recompute was and how close expiry is. The published version is optimal probabilistic cache stampede prevention, from Vattani, Chierichetti and Lowenstein at VLDB 2015: refresh when now − delta × beta × ln(random) reaches the expiry time, where delta is the duration of the previous recompute and beta defaults to 1. Expensive values refresh earlier, cheap ones stay lazy, and the herd never forms.

Serve stale while you refresh. The most useful primitive on this list, and the least used.

Stale-while-revalidate, and the sibling that saves your availability

RFC 5861 defines two Cache-Control extensions that most teams never send. A header like Cache-Control: max-age=60, stale-while-revalidate=600, stale-if-error=86400 says three separate things: this is fresh for a minute; for ten minutes after that, serve the stale copy immediately and refresh in the background; and if the origin is failing, keep serving the stale copy for a day rather than serving an error. Browsers and the major CDNs implement it, and the same pattern is straightforward to implement inside an application cache by storing a soft expiry alongside a hard one.

The effect on the latency profile is larger than it sounds. Under plain expiry every miss pays full origin latency and lands on a user. Under stale-while-revalidate the user gets a cache-speed response and the refresh happens off the request path, so the tail flattens and the origin sees a trickle instead of a burst.

stale-if-error is an availability feature disguised as a caching directive. A read path with a generous window survives a dependency outage in degraded form rather than failing. That trade is right for content, listings and reference data, and wrong for balances, permissions and anything transactional. Decide it per path, next to the staleness budget.

Layered TTLs add up; they do not average out

Real systems stack four or five caches, each configured by a different person in a different year. Worst-case staleness is the sum of the layers, not the largest one, because each layer can hand a nearly expired copy to the layer above it at the last possible moment.

Layer	What it holds	Typical TTL	How you actually clear it
Browser	Whatever max-age you sent, including mistakes	0–300s	You cannot. Change the URL, or send a short max-age and rely on ETag revalidation
CDN or edge	Responses keyed by URL plus the Vary set	60s–1 day (s-maxage)	Purge by tag or URL. Propagation is fast but is not instantaneous or globally atomic
Application cache	Serialized objects, fragments, computed results	10s–1 hour	Delete on write, or bump a key version
Query or ORM cache	Result sets, often keyed by SQL text	Seconds to minutes	Usually TTL only, which is why it is the layer that surprises people
Materialized view or rollup table	Precomputed aggregates	Refresh interval, 1–60 min	Rerun the refresh. In Postgres, the concurrent form needs a unique index

Do the arithmetic once and it changes the design. Browser 60 seconds, edge 300, application 900, rollup every 15 minutes: worst-case age is about 36 minutes on a path whose budget somebody wrote as five. The repair is not to shave every layer. Pick one layer to be authoritative for freshness, keep the rest short or content-addressed, and stop treating each config file as an independent decision.

One edge-specific trap: sending Vary on Authorization or Cookie disables edge caching for logged-in traffic, because the cache stores a copy per credential. Cache an anonymous shell at the edge with a long TTL and fetch personalized fragments underneath it, rather than teaching the edge your permission model.

Eviction quietly rewrites your TTLs

The lifetime of a cached value is not its TTL. It is the smaller of the TTL and the time until eviction, and under memory pressure the second term dominates. A cache near its memory limit is not honoring your TTLs. It is honoring an eviction policy you may not have chosen.

Two Redis specifics bite in production. The default maxmemory-policy is noeviction, so a full instance rejects writes instead of making room, and a cache path that is not defensive turns those rejections into user-visible errors on an optional component. The volatile-* policies only consider keys that have a TTL, so a codebase storing some values without expiry can fill an instance with unevictable entries while the config claims LRU. Redis LRU is also approximate, sampling candidates rather than maintaining a true ordering.

Watch three counters: evicted keys per second, memory used against the limit, and hit ratio per key class over time. Sizing has a cliff rather than a slope. If trimming memory by 20 percent in a load test collapses the hit ratio, the working set barely fits and the next traffic increase will find that out for you.

If losing the cache takes you down, it is not a cache

Here is the test that settles most architecture arguments about caching. In a load test at realistic traffic, flush the cache completely and watch what happens. If the system serves the miss storm with elevated latency and recovers, you have a cache. If the origin falls over, you do not have a cache. You have a database with no durability, no backups and no replication, sitting in the critical path of every request, and you should either give it those properties and operate it accordingly or fix the origin so it can serve the cold miss rate.

When the origin cannot be fixed cheaply, stop caching the query and precompute into a real table. A rollup written by a scheduled job has durability, replicas, backups and a query planner, and it does not evaporate on restart. More work than a cache decorator, and on a path whose cold-start cost is an outage, the correct amount of work.

The related failure is the miss storm from things that do not exist. Without negative caching, every request for a deleted object, an empty result set or a denied permission is a full origin round trip, and any process walking identifiers becomes an unintentional load test. Cache the absence, with a shorter TTL than the positive case. Thirty to sixty seconds absorbs a storm without noticeably delaying when a new object becomes visible.

Five ways to build a very fast source of wrong answers

• Keys assembled at call sites. A new query parameter reaches the handler and never reaches the key. Two requests with different meanings collide, and the symptom arrives as a support ticket months later.
• One TTL for everything, with no jitter. Five minutes on every key means every key written during a deploy expires in the same second, forever, in a wave the origin has to absorb on a fixed schedule.
• Caching after authorization under an unauthenticated key. The most expensive bug in this article, and it passes every functional test written by one logged-in user.
• Treating the purge button as the invalidation design. Correctness that depends on a person remembering is correctness you do not have.
• Measuring hit rate and nothing else. The one metric that improves as the cache becomes less truthful is the one on the dashboard.

The probe that catches a lying cache

The highest-value thing missing from most caching setups costs about a day to build. Store the write timestamp inside the cached value so every read can compute the age of what it served and emit it as a histogram. Then run a background job that samples key classes at a low rate, reads through the cache, reads the same value from the origin, canonicalizes both and compares. Emit a divergence rate per key class and the age distribution of the values that diverged.

That gives you a real alert. Not "hit rate dropped below 80 percent", which wakes people up for nothing, but "the pricing key class is serving values older than its 60-second budget at the 99th percentile", which is a genuine defect with a location attached. Sampling a few keys per second per class is enough for statistical confidence within minutes and adds load the origin will not notice.

Canonicalization is where the work is. Timestamps, generated identifiers and unordered collections make two identical answers compare unequal, so each key class needs a normalization function, and writing it forces a useful argument about which fields are semantically part of the answer. Where exact comparison is impractical, hash the fields that matter and ignore the rest.

Write these down before the cache ships

• The staleness budget in seconds for each read path, and the person who agreed to it
• Every input that changes the answer, and its position in the key
• A namespace prefix carrying the schema version, bumped by deploy
• The TTL, its jitter band, and the behavior at expiry: coalesce, refresh early, or serve stale
• What happens when the cache is unreachable: fail open to the origin, or fail the request
• The write-path invalidation, and the race it does not cover
• The eviction policy, the memory headroom, and an alert on evicted keys
• The divergence probe, its sampling rate, and the threshold that pages someone

Where the effort belongs

Two ordering rules save more time than any specific technique. First, when the origin call is slow because the query is bad, fix the query before adding a cache. Otherwise the problem is still there and harder to find, because the normal path now looks fast.

Second, cache the widest object whose staleness budget you can honor. One assembled response beats twelve cached fragments on latency and on origin load, and a shorter TTL is worth paying to get it. Split into fragments only when a real freshness requirement forces it, and let each fragment carry its own budget instead of inheriting the strictest one on the page.

When we review a system, the finding is rarely that caching is missing. It is that the cache exists, works, and has never been asked what it promises: nobody can say how old the data on a screen may be, nothing measures how old it is, and the keys are missing an input that has not collided yet. Those gaps take a couple of weeks to close, and closing them turns a component nobody trusts into one you can reason about during an incident.

Bottom line

A cache trades correctness for speed. The trade is often excellent, it is never free, and the engineering is in making the terms explicit rather than in choosing a library. Write the staleness budget in seconds. Put every input that changes the answer into the key, from one function. Prefer versioned keys over deletes, jitter every TTL, coalesce concurrent misses, serve stale while you refresh, and add up the layers instead of assuming they overlap. Then measure the age of what you served and probe for divergence, because a cache that is wrong raises no alarm. It answers quickly and confidently, which is what makes it dangerous.

Frequently asked questions