The question is arithmetic, not architecture
Every few months someone declares that long context has killed retrieval, and someone else replies that retrieval is obviously still necessary. Both are describing their own corpus. The honest answer depends on four numbers: how big the body of text is, how many questions get asked against it, how much of it changes and how often, and whether different readers are allowed to see different parts. Write those four down and the decision usually makes itself in about ten minutes.
What genuinely changed is that a large class of problems moved from the second column to the first. A corpus that is a few hundred pages — a product manual, a contract set, a policy handbook, one company's filings for a year — now fits comfortably in a single request. Building a chunking strategy, an embedding pipeline, a vector store and a reranker for that is a month of work and a permanent operational surface, and for many teams it is no longer necessary. That is a real shift and it deserves to be taken seriously rather than defended against.
What did not change is that most interesting corpora are much larger than a few hundred pages, change constantly, and have readers who are not allowed to see the same things. All three of those defeat the simple approach regardless of window size.
You are probably here because
- You are about to build a retrieval stack and someone asked whether you still need one
- You have a retrieval stack and half your quality complaints trace to what it did not fetch
- Long context works beautifully in testing and your per-query cost is now the line item
- Answers are correct on small documents and get vaguer as the input grows
The cost section below settles the first and third; the accuracy and permissions sections settle the second and fourth.
The cost calculation, including the part everyone forgets
Naively, placing a whole corpus in every request looks catastrophic. A 300,000-token corpus at a typical frontier input rate is a few dollars per question, so ten thousand questions a day is an eye-watering number and the conversation ends there. That arithmetic is wrong in a way that matters, because it ignores prompt caching.
When the same prefix is sent repeatedly, a cached read is billed at a small fraction of the base input rate — an order of magnitude cheaper is the right mental model — while writing the cache costs a modest premium over base input. For a stable corpus and a question appended at the end, essentially the entire input is a cache hit after the first request. The effective cost per question drops by roughly a factor of ten, and a design that looked absurd becomes ordinary.
That single fact reshapes the decision, and it comes with a hard condition: the cached portion must be byte-identical and must sit at the front. A corpus assembled in a different order, a timestamp in the header, a per-user greeting before the documents — any of these takes the discount away without any other symptom. Cache lifetimes are also short by default, measured in minutes, so the economics depend on query volume being high enough to keep the entry warm. Sporadic traffic pays the write premium repeatedly and gets the worst of both designs.
| Situation | Long context | Retrieval | Which we would choose |
|---|---|---|---|
| 50–300 pages, stable, everyone sees everything | Cheap with caching, near-zero build cost | A month of build and permanent ops for no accuracy gain | Long context, without hesitation |
| Thousands of documents, one is relevant per question | Does not fit, or fits and buries the answer among distractors | Exactly the case it was designed for | Retrieval |
| Moderate corpus, very high query volume | Caching makes it competitive; watch the write premium | Lower marginal cost, higher fixed cost | Measure both; the crossover is real and computable |
| Corpus changes hourly | Every change invalidates the cache and the full price returns | Re-index the changed documents only | Retrieval, or a hybrid with a stable core |
| Readers have different permissions | Unsafe unless you build a per-reader corpus, which loses the cache | Filter at query time, which is the normal design | Retrieval |
| Every answer must cite a source | Possible, but citations are asserted rather than constructed | Citations fall out of the mechanism | Retrieval, or long context with a grounding check |
Latency behaves differently from cost
The two do not move together, and teams optimize one while quietly ruining the other. Time to first token grows with input length, because the input has to be processed before anything is generated. Caching helps here as well — a cached prefix skips most of that work — but a large uncached input is felt directly by a user waiting for the first word.
Retrieval adds latency of its own that is easy to forget in a design document: embedding the query, searching the index, optionally reranking, then assembling the prompt. A well-built retrieval hop is tens to low hundreds of milliseconds. A poorly built one, especially with a cross-encoder reranker over a wide candidate set, can exceed a second and turn a fast model into a slow product.
The useful comparison is therefore not "retrieval is faster." It is: a warm cached long-context request is often the fastest option available, an uncached one is the slowest, and retrieval sits in between with a floor set by your index. If interactive latency is the binding constraint, measure the first-token time under both designs before choosing, because intuition is unreliable here.
Accuracy: more context is not monotonically better
The most common surprise is that adding more material can make answers worse. Two effects drive it.
Distractors. If nine documents discuss a similar topic and one contains the answer, the presence of the nine reduces the chance of a clean answer from the one. This is the mechanism behind the vague, blended, "on the one hand" responses that appear as corpora grow, and it is why a retrieval system that returns three precise passages can beat a window containing all thirty.
Position sensitivity. Material at the very start and the very end of a long input is used more reliably than material in the middle. The effect has narrowed considerably with recent models and it has not disappeared. Practically: put the question after the documents rather than before, place the material you believe is most relevant near the boundaries, and do not assume that something buried at 60% depth will be found as reliably as something at the top.
Both effects argue for the same discipline, whichever architecture you pick. Send the smallest set of material that plausibly contains the answer. Long context is permission to stop building an elaborate retrieval pipeline; it is not permission to stop thinking about relevance.
How strongly each factor pushes toward retrieval — our weighting
The top two are close to decisive on their own. The bottom row is the one people over-weight; caching usually resolves it.
The three conditions that make retrieval mandatory
Everything above is a tradeoff. These three are not, and each one ends the discussion by itself.
Different readers may see different things. If access is scoped by team, customer, region or role, the material has to be filtered before it reaches the model. A single shared corpus in the prompt cannot enforce that, and instructing the model to withhold what it can see is not a control — it is a request. Building a per-reader corpus is technically possible and destroys the cache economics that motivated the approach.
The corpus is much larger than the window. Obvious, but the version that catches people is a corpus that fits today and doubles in eighteen months. Migrating from long context to retrieval later is a real project. If growth is plausible, design the seam now even if you do not build the index yet.
Freshness is measured in minutes. If the answer must reflect a change made ten minutes ago, a cached prefix is working against you, and invalidating it on every write removes the discount entirely. Retrieval over a freshly indexed store is the natural design.
Permissions are a design decision, not a prompt instruction
The pattern we are asked to review most often is a shared corpus in the system prompt plus an instruction to only discuss what the current user is entitled to see. It works in testing because testers ask ordinary questions. It is not an access control, it cannot be audited, and a single unexpected phrasing is the whole exposure. Filter the material before it reaches the model, every time, without exception. This is the one place in this article where there is no tradeoff to weigh.
The hybrid that usually wins
In practice the best design is rarely either pure form. It is a stable core in the cached prefix plus a small retrieved tail.
Put the material that is common to every question at the front and let it cache: the schema description, the glossary, the policy document, the product manual, the reference tables. Retrieve only the part that varies per question — the specific customer record, the relevant filings, the three passages that matter — and append it after the cache boundary along with the question. You get the discount on the bulk, freshness and permission filtering on the part that needs it, and a retrieval component that is far simpler than a full pipeline because it only has to be good at a narrow job.
This shape also degrades gracefully. When the corpus grows, more of it moves from the cached core to the retrieved tail, and no rewrite is required. When retrieval misses, the model still has the stable context and produces a partial answer rather than nothing. Design the boundary between the two early; it is the seam that lets the system change later without a migration.
Send the four numbers and we will do the arithmetic.
Corpus size, questions per day, change rate, and whether readers have different permissions — to contact@precisionfederal.com. You get back the cost comparison under both designs and which we would build, in writing. One business day, no charge, no meeting.
contact@precisionfederal.comHow to decide with a measurement instead of an opinion
This comparison is unusually cheap to run properly, which is why it is frustrating how often it is settled by argument. Two days is enough.
Build a question set of forty to sixty real questions with known answers, weighted toward the hard ones: questions whose answer lives in one obscure document, questions requiring two documents at once, questions with a plausible wrong answer sitting in a neighbouring file, and questions that should be answered "the material does not say." That last category is the one everyone omits and the one that separates the designs most clearly.
Then run both. Record answer accuracy, refusal-when-appropriate rate, time to first token, total latency, and cost per question with caching enabled and disabled. Report the numbers per question category rather than as one average, because the aggregate will hide the fact that one design is much better on multi-document questions and much worse on the ones with distractors.
Expect a split verdict, and treat that as information rather than an inconclusive result. It is what tells you which questions belong on which path, and routing by question type is a legitimate and frequently correct design.
What breaks at scale
- A timestamp or user id ahead of the corpus, silently removing the cache discount that justified the whole design
- Permissions enforced by instruction rather than by filtering the material before it is sent
- A corpus assembled in nondeterministic order, so the prefix differs on every request
- The whole corpus sent when a tenth would do, adding distractors and cost together
- Cost measured on a cached benchmark that never reflects real, sporadic traffic
- A reranker over a wide candidate set adding a second of latency nobody budgeted
- No test for "the material does not say", so the system confidently answers questions it should decline
- Chunk boundaries cutting through tables, which turns a retrieval quality problem into a data problem
A two-day bake-off
Deciding it with data
Step four says no reranker deliberately. A plain vector search with a sensible chunk size is the baseline, and if it wins you have saved yourself a component. Adding a reranker before you know whether the simple version suffices is the most common way retrieval stacks acquire latency and complexity nobody can justify later.
Common objections
Does a bigger window mean we can stop chunking entirely?
You can stop chunking for retrieval, and you should still think about document boundaries. Sending whole documents rather than fragments is a real quality improvement, because a fragment cut through a table or a numbered clause loses the structure that made it meaningful. What disappears is the tuning exercise around chunk size and overlap, which is a genuine saving.
Is retrieval better for citations, or is that just habit?
It is structurally better, because you know which passages you sent, so a citation can be verified against the set rather than trusted. In long context a citation is a claim the model makes about material it can see, and it is usually right and occasionally not. If you use long context where citations matter, check that the quoted text actually appears in the input — that assertion is cheap and catches the cases that matter.
We already built a retrieval pipeline. Should we throw it away?
Almost certainly not, but consider simplifying it. Many pipelines carry components added to compensate for a small window: aggressive chunking, elaborate query rewriting, multi-stage reranking, summarization of retrieved passages. With a large window you can often retrieve fewer, larger units and delete two or three stages. That is a cheaper and lower-risk improvement than a rewrite, and it usually improves quality as well as latency.
What about putting the corpus in a fine-tuned model instead?
Different tool, different job. Fine-tuning teaches form, style and task behavior reliably; it is a poor mechanism for storing facts you need recalled exactly, and updating a fact means retraining. If the requirement is "answer from this material, accurately, and tell me where it came from," that is a context problem, whether the context arrives whole or retrieved.
Before you commit to either
- Corpus size, query volume, change rate and permission model written down
- Cost computed with caching on and off, at realistic traffic rather than in a loop
- The cached prefix is byte-identical and assembled in a deterministic order
- Time to first token measured under both designs, warm and cold
- A question set including multi-document and unanswerable cases
- Results reported per question category, not as a single average
- Permission filtering happens before material reaches the model
- A stated growth assumption, and a seam that survives it
Bottom line
Long context genuinely removed the need for a retrieval stack on a whole class of problems, and teams still building one for a 200-page manual are spending a month they do not need to spend. It did not remove the need for relevance, and it cannot enforce permissions, keep up with minute-by-minute freshness, or hold a corpus that does not fit. The design that survives contact with a growing product is usually a cached stable core plus a small retrieved tail, chosen after two days of measurement rather than after an argument. Write down the four numbers first — they answer the question more reliably than any general claim about which approach is winning.
Frequently asked questions
It can be, and only with prompt caching. A cached read costs a small fraction of the base input rate, so a stable corpus reused across many questions is roughly an order of magnitude cheaper than the naive calculation suggests. Without caching — a corpus that changes often, a prefix that is not byte-identical, or traffic too sparse to keep the cache warm — retrieval is usually much cheaper.
No. Similar-but-irrelevant documents act as distractors and pull answers toward vague blends, and material in the middle of a very long input is used slightly less reliably than material near the beginning or end. Both effects mean the smallest set of plausibly relevant material outperforms the largest set, whichever architecture you use.
Three cases. When different readers are entitled to see different material, because filtering has to happen before the model sees anything. When the corpus is substantially larger than the window, or will be. And when answers must reflect changes made minutes ago, since a cached prefix works against freshness by design.
Stable shared material — glossary, schema, policy, manual — sits at the front of the request and caches. The per-question material is retrieved and appended after the cache boundary, together with the question itself. You keep the caching discount on the bulk of the input while retaining freshness and permission filtering where they are needed.
Build forty to sixty real questions with known answers, including some that require two documents and some that the material genuinely cannot answer. Run both designs and record accuracy, appropriate refusals, first-token latency and cost per question with caching on and off. Compare by question category rather than on the average — a split result tells you how to route, which is more useful than a winner.