The question is asked wrong
"Should we fine-tune the model or put retrieval in front of it?" almost always turns into an argument about which one is more accurate. That argument has no winner, because the two techniques do not compete on accuracy. They compete on cost, and they fail in different places. Fine-tuning changes how a model behaves. Retrieval changes what the model can see when it answers. Once you write down what each one costs in your usage pattern, the decision usually stops being interesting, which is the sign you asked the right question.
We build both. The split is rarely 100/0 in either direction, and the teams that get stuck are the ones who treated it as a binary to win before they had a single measured number. This is the arithmetic we run first.
You are probably here because
- Someone handed you a fine-tuning quote and the only line on it you can check is GPU hours.
- Your token bill grows every month with adoption, and nobody can tell you what it becomes at three times the traffic.
- The team has argued fine-tune versus retrieval for weeks, the argument is about accuracy, and there is still no number written down anywhere.
- One wrong answer takes days to fix, and it is not clear to anyone why it should.
The token arithmetic section below runs the comparison at four request volumes, and the section on what a fix costs covers the line that flips the answer most often. These usually share one root cause: the five cost lines were never put on the same page, so the decision got made on confidence instead of arithmetic.
What each one actually buys
Fine-tuning adjusts the model's output distribution. Supervised fine-tuning on input-output pairs, LoRA and QLoRA adapters that train a small set of low-rank matrices instead of the full weights, and preference tuning on ranked outputs all do versions of the same thing: they make certain outputs more likely and others less likely. That buys you exact output format on every call, house vocabulary, adherence to a tool-call schema, classification into your taxonomy rather than a generic one, consistent tone, and the ability to delete a very long instruction prompt because the behavior now lives in the weights.
Retrieval changes the contents of the context window at request time. An embedding model turns your corpus into vectors, a query pulls the relevant chunks, and the model answers from what it was handed. That buys you facts that change after training day, a corpus too large to memorize, citations pointing at a source a person can open, and a correction path that is an edit to a document rather than a training run.
The overlap that causes the confusion is real. Continued pretraining or fine-tuning on domain text does raise apparent factual accuracy inside that domain. It does it by shifting priors, not by installing a lookup table, and the failure mode is the reason so many teams get burned: a tuned model is more fluent about your domain and just as willing to invent a number in it. Fluency reads like knowledge right up until someone checks a figure.
The five lines that decide the bill
Every fine-tuning proposal we review quotes training compute. Training compute is almost never the line that decides the outcome. Five lines do, and they are worth writing on the same page before anyone opens a pricing calculator.
Label production. The examples the model trains on, at whatever quality your task requires. This is the dominant cost of a fine-tuning program and the one that is systematically underestimated, because it is measured in expert hours rather than dollars on an invoice.
Training compute. GPU-hours for each run. Real, bounded, and usually the smallest line here.
Retrain cadence. How often the tuned model goes stale, which is a property of your domain, not of your model. A support workflow over a product that ships monthly goes stale monthly.
Evaluation maintenance. The test set, the scoring script, and the person who runs it. Both approaches need this. Neither is safe to change without it.
Steady-state serving. Tokens per request multiplied by requests, plus whatever it costs to host a custom model or an index.
Where the money goes — first year of a fine-tuning program
Our planning split for a first-year adapter program on an open-weight or hosted base model. Ratios shift with label reuse, not with GPU price.
Fine-tuning: the bill in detail
Start with the dataset, because it decides everything downstream. A few hundred examples is enough to lock an output format. A task with real judgment in it wants somewhere between one and five thousand, and the returns flatten well before ten thousand for most narrow tasks. Quantity matters less than consistency: a thousand examples that agree with each other beat five thousand encoding three different people's opinions.
The cost of that dataset depends almost entirely on one thing, and it is the first question we ask on any tuning engagement. Do correct pairs already exist somewhere in your systems? Resolved tickets with the reply an expert actually sent, documents with the fields a person already keyed into an ERP, questions with an approved answer in a knowledge base. If yes, label production is an extraction and cleaning problem, and it is cheap. If no, a domain expert has to write them, and a careful expert produces on the order of twenty to forty examples an hour on a simple task and five to ten an hour on a hard one. Two thousand hard examples is roughly a person-quarter. Price that at your loaded rate before you price anything else.
Training compute is the part everyone can quote and the part that rarely matters. A LoRA run over a few thousand examples on an open-weight model in the seven to fourteen billion parameter range finishes in single-digit GPU-hours. On-demand H100 capacity runs roughly $10 to $13 per GPU-hour at the major clouds and materially less on specialist GPU providers or with committed capacity, so a single run is a few hundred dollars at worst. The honest number is not the run. It is the loop: expect five to fifteen runs before something is worth deploying, plus the same again over the following year as the data drifts and the base model gets deprecated underneath you.
Serving is where the tuned model either pays for itself or quietly does not. Hosted fine-tuning endpoints typically charge either a per-token premium over the base model or an hourly fee to keep your weights warm, and the hourly shape punishes bursty traffic badly. Self-hosting means a GPU that has to stay resident whether or not anyone is asking it anything. The design that changes this arithmetic most is adapter multiplexing: several LoRA adapters served over one shared base model, so five tuned behaviors cost roughly one model's worth of memory. If you expect more than one tuned task, build for that on day one.
Retrieval: the bill in detail
The largest line in a retrieval system is not inference and not storage. It is ingest. Getting documents into a state where retrieval works at all means deduplication, version resolution so the superseded policy does not outrank the current one, OCR for the scanned material, table extraction, chunking that respects document structure instead of cutting every 512 tokens, and metadata that lets you filter by owner, date, and permission. That work is engineering, it is proportional to how messy the corpus is, and skipping it is the single most common reason a retrieval pilot returns confident answers from the wrong document.
Embedding is the cheapest thing on this page. Commercial embedding endpoints are priced in cents per million tokens, so a ten-million-token corpus embeds for pocket change and a corpus a hundred times that size is still a rounding error against one engineer's week. Re-embedding when you change embedding models costs the same nothing. The real cost of that swap is re-tuning retrieval thresholds and re-running your evaluation set.
Vector storage prices on vectors stored and queries served. A few million chunks in a managed service is a modest recurring line, and self-hosting on existing infrastructure moves it closer to zero. Neither is where your money goes.
Where it goes is prompt tokens. Every retrieval request carries its retrieved passages into the context window, and that cost recurs on every single call forever. Three to six thousand tokens of retrieved context per request is normal for a system that returns grounded answers with citations. That is the number that grows linearly with adoption, and it is the number that makes fine-tuning look attractive at volume.
The token arithmetic, done once
Here is the comparison that actually decides most of these engagements. Take a retrieval-grounded request at about 4,500 input tokens and 400 output tokens, against the same task handled by a tuned model with the instructions baked into the weights, at about 600 input tokens and 400 output. Price both on a mid-tier hosted model in the neighborhood of $2.50 per million input tokens and $10 per million output tokens. Substitute your own price sheet, because these numbers move, but the shape does not.
| Requests / month | Retrieval prompt 4,500 in / 400 out | Tuned model 600 in / 400 out | Monthly delta | Annual delta |
|---|---|---|---|---|
| 50,000 | $763 | $275 | $488 | $5,850 |
| 500,000 | $7,625 | $2,750 | $4,875 | $58,500 |
| 5,000,000 | $76,250 | $27,500 | $48,750 | $585,000 |
| 25,000,000 | $381,250 | $137,500 | $243,750 | $2,925,000 |
Now put the fine-tuning program next to it. A first tuned model on a task where labels have to be written from scratch, including the evaluation set, the training loop, integration, and a year of upkeep, lands in a wide band that we plan at $60,000 to $200,000 depending almost entirely on how many expert hours the labels take. Set that against the annual delta and the answer falls out on its own.
At 50,000 requests a month, the tuned model saves under six thousand dollars a year and costs six figures to build. It is not close, and no amount of enthusiasm makes it close. At 500,000, payback is somewhere past a year and depends on whether the task holds still long enough to get there. At five million, the savings pay for the program in a quarter and keep paying. At twenty-five million, not tuning is the expensive decision.
Two adjustments before you trust that table. First, hosted fine-tune endpoints often carry a per-token premium of their own, which eats part of the delta, so use the tuned endpoint's real price rather than the base model's. Second, the comparison assumes the tuned model does the job as well. If the task needs current facts, the tuned model does not do the job at all and the cheaper column is imaginary.
Where fine-tuning is unambiguously cheaper
High volume on a narrow, stable task. The table above is the whole argument. Classification, extraction into a fixed schema, routing, normalization, and short structured generation at millions of calls a month.
Output format that has to be exact. Prompt engineering gets format compliance high and then hits a wall, and the last few percent of malformed outputs is where your downstream parser lives. Tuning moves that wall.
Latency budgets that cannot absorb a retrieval hop. A vector query plus four thousand extra tokens of prefill adds real milliseconds. In an interactive loop or a high-throughput pipeline, that is a design constraint, not a detail.
Distilling a big model into a small one. This is the largest cost lever available on most systems today and the most underused. A frontier model with a long prompt does the task; you capture its behavior on your inputs, tune a small open-weight model on those pairs, and serve the small one. When it works, the per-request cost drops by an order of magnitude. Two cautions: confirm the source model's terms permit training on its outputs, and evaluate the small model on your own labeled set rather than on agreement with the teacher, because agreement with a teacher that was wrong is not accuracy.
Where retrieval is unambiguously cheaper
Knowledge that changes. Pricing, policies, product documentation, contract terms, inventory, personnel. If the answer differs between March and April, weights are the wrong place to keep it, and retraining monthly to chase a document edit is an expensive way to be behind.
Anything that has to be cited. A tuned model can be taught to produce a citation-shaped string. It cannot be taught to produce one that is guaranteed to exist. Retrieval hands you the source as a byproduct of how it works.
Long-tail corpora. Two hundred thousand documents each read twice a year is a retrieval problem by construction. No training set covers a tail, and paying to memorize documents nobody asks about is waste.
No labels and no budget to make them. Retrieval needs a corpus and an evaluation set. Fine-tuning needs a corpus, an evaluation set, and a training set. If the third one does not exist, that is the decision.
The cost line nobody models: what a fix costs
Every production system produces wrong answers. The question that matters six months in is not how many, it is what one costs to correct once a user reports it. This line never appears in the build estimate and it dominates the operating experience.
In a retrieval system, most wrong answers trace to a document. Someone edited the wrong page, an obsolete version outranked the current one, a chunk boundary split a table in half, or the ranker preferred a similar-sounding passage. The fix is an edit, a re-index of one document, and a re-run of the affected evaluation cases. Minutes to hours, no deployment, no model risk, and an audit trail showing what changed.
In a tuned model, the same wrong answer is a property of the weights. There is no page to edit. You add or correct training examples, retrain, re-evaluate the entire suite because tuning changes behavior everywhere and not just where you aimed, and redeploy. Days at best, and every fix carries regression risk on tasks that were working.
Relative cost to correct one wrong answer (100 = full retrain cycle)
Engineer-hours to close one reported defect, indexed to a full retrain and re-evaluation cycle. Multiply by your defect rate to get a real annual number.
Multiply that difference by a realistic defect volume and it often exceeds the token savings that motivated the tuning. That is the calculation that flips answers most often.
Send it over and we will tell you what we would change.
Email one typical request with its retrieved context, your monthly request count, and any fine-tuning quote you have been handed to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comReading the signals
Most of the decision is legible from properties you already know about your workload. This is the table we fill in before touching a model.
| Signal in your workload | What it implies | Why |
|---|---|---|
| Answers depend on documents edited this month | Retrieval | Weights cannot be edited; documents can |
| Users must be able to open the source | Retrieval | Citation is a byproduct of retrieval and a fiction without it |
| Over ~2M requests/month on one narrow task | Tune, keep retrieval if facts are involved | Prompt compression pays back inside a quarter at that volume |
| Format compliance stuck below your parser's tolerance | Tune | Prompting plateaus on structure; tuning does not |
| Correct input-output pairs already exist in a system of record | Tune is cheap enough to test | Label production, the dominant cost, is already paid |
| No labeled examples and no expert hours available | Retrieval | There is nothing to train on and buying labels is the whole budget |
| Tail of rarely-asked questions over a large corpus | Retrieval | A training set cannot cover a tail |
Where the training data came from is a contract question, not just an engineering one
Three checks before a single example enters a training set. First, the terms of the model you are training on and the terms of any model whose outputs you captured: some providers prohibit using outputs to train competing models, and that clause survives your enthusiasm. Second, whether the data carries obligations that follow it into the weights. Personal data under GDPR brings deletion rights that a trained model cannot honor by deleting a row, and health or cardholder data brings handling obligations that apply to the training environment as much as to production. Third, who owns the resulting adapter weights if a vendor trains them for you, and whether you can export them. That last one decides what your exit costs and it is settled in the agreement, not in the architecture.
The hybrid that is usually the right answer
Written as a binary, this comparison produces bad systems. In practice the two techniques sit at different layers and the strong design uses both for what each is cheap at. Retrieval supplies the facts. A small adapter supplies the behavior.
Concretely: keep the corpus in the index so a document edit is still the correction path, and train a LoRA adapter on eight hundred to two thousand examples of how your organization answers, which questions it declines, and the exact schema it emits. That adapter replaces two or three thousand tokens of instructions and few-shot examples on every call, which is most of the token savings from the table above, while leaving the fast correction path intact. You are compressing the instructions, not memorizing the knowledge.
The order of operations matters more than the choice. Prompt first, because it is free and it tells you whether the task is even well-defined. Add retrieval second, because most failures that look like model failures are context failures. Build the evaluation set third, because without it you cannot tell whether anything you do next helps. Tune fourth, when you have a measured reason. Teams that tune first spend a quarter proving a model can imitate examples, which was never in doubt, against a prompt baseline they never tried.
What actually moves accuracy on a document-grounded task
Ordering we see repeatedly on grounded question-answering. Fine-tuning ranks low for accuracy and high for cost per request, which is why it belongs later in the sequence.
What has to exist before you spend money on either
- An evaluation set of 200 to 500 real requests with agreed correct answers, in your repository
- A measured cost per resolved request for whatever you run today, including human review time
- A twelve-month request volume forecast with a stated upper bound
- A corpus inventory: document count, change rate, and a named owner per source
- A written answer to where correct input-output pairs already exist in your systems
- An owner of measurement who did not build the model
- A rollback path that does not require a training run to execute
Mistakes we see repeatedly
- Fine-tuning to stop a model inventing numbers. Tuning changes style, not the capacity to generate. Fabrication is stopped by grounding, constrained decoding, and verification against a source.
- Provisioning GPUs before producing a thousand labels. The hardware is the easy half and the half that arrives fastest, which is exactly why it gets bought first.
- Comparing a tuned model against a weak prompt. If the baseline was never optimized, the comparison measures effort, not technique.
- Training on captured outputs without reading the terms. Some providers prohibit it, and finding out later invalidates the model, not just the paperwork.
- Treating the index as write-once. A corpus with no re-ingest schedule and no version resolution degrades quietly, and the answers stay confident the whole way down.
- Optimizing accuracy without tracking cost per resolved request. A model two points better that costs four times as much per answer is a worse system with a better slide.
- Skipping the small-model test. Teams tune the largest model they can afford and never check whether a 7B model handles the narrow task at a twentieth of the price.
A four-week way to get a real number
Decision Sprint
Four weeks works because every expensive unknown here is measurable inside it. Whether retrieval answers your questions is measurable. Whether a tuned adapter beats a good prompt is measurable. Whether labels already exist is a query against your own systems. What it costs per request at your real volume is arithmetic once you have the token counts. Everything else is preference, and preference is what turns this into a six-month debate that ends in the wrong column.
Bottom line
Fine-tuning and retrieval are not rivals and the accuracy framing hides the decision. Retrieval is a per-request subscription: low to start, grows with adoption, corrects in minutes. Fine-tuning is a capital cost with a maintenance contract: expensive to build, cheap per request, slow to correct. Put knowledge in the index and behavior in the weights, build the evaluation set before you spend on either, and let the volume decide how much prompt compression is worth buying. The teams that get this right are not the ones with a strong opinion. They are the ones who did the arithmetic on their own numbers.
Frequently asked questions
Per request, usually yes, because the instructions live in the weights instead of the prompt. In total, only above a volume threshold. Compare the annual token delta against the full program cost, which is dominated by label production rather than GPU time. Below roughly half a million requests a month on a single task, the delta rarely repays the build.
A few hundred to lock an output format, one to five thousand for a task with judgment in it. Consistency matters more than volume: a thousand examples that agree with each other outperform five thousand that encode three different opinions. Returns flatten well before ten thousand on most narrow tasks.
It shifts what the model tends to say, which raises apparent accuracy in a domain, but it does not install a lookup table and it does not make the model able to check anything. Facts that change, facts that need citation, and facts that need to be corrected quickly belong in retrieval.
That is the usual production answer. Retrieval supplies the facts and keeps the fast correction path; a small adapter supplies the format, vocabulary, and refusal behavior, which lets you delete a long instruction prompt and capture most of the token savings without memorizing anything that changes.
Ingest, not inference. Deduplication, version resolution, OCR, table extraction, structure-aware chunking, and permission metadata are the bulk of the work, and they recur as the corpus grows. After that it is prompt tokens at steady state, which scale linearly with adoption and are the line that makes tuning attractive at high volume.