When to fine-tune and when to prompt

Diagnose the gap before choosing the tool

Fine-tuning gets reached for early and often, usually because the output is wrong and weight updates feel like the serious response. But "wrong" covers four completely different failures, and only two of them respond to training. Getting this diagnosis right takes an afternoon and saves the six weeks that a mistargeted tuning project consumes before anyone admits it did not help.

A knowledge gap. The model does not know something it needs to know: your product catalogue, your internal terminology, last quarter's policy. The fix is to put the information in the context, through retrieval or a structured lookup. Training facts into weights is expensive, imprecise, impossible to update without another training run, and produces confident errors when the fact changes. This is the misdiagnosis that wastes the most time in this field.

A behaviour gap. The model knows the thing but will not do it consistently: it ignores the schema one time in twenty, drifts out of your house style, will not stop adding a preamble, does not follow your escalation rules. This is what fine-tuning is genuinely excellent at, and often the only thing that gets the last few percentage points of consistency.

A capability gap. The task requires reasoning the model cannot do at all. More examples will not create the ability. Your options are a stronger model, decomposing the task into steps each model can handle, or accepting a lower ceiling. Fine-tuning a weak model to do something it fundamentally cannot do produces confident nonsense in your house style.

A cost or latency gap. The model does the task correctly and costs too much or takes too long. This is the other case where training is exactly right: distil the behaviour of a large model into a small one on your narrow slice of the problem. You are not trying to make the small model generally smarter, only reliably good at one thing.

Symptom	Gap type	What actually fixes it
Invents plausible facts about your domain	Knowledge	Retrieval with provenance, not training
Breaks the output schema occasionally	Behaviour	Constrained decoding first, then fine-tuning
Right answer, wrong tone or structure	Behaviour	Prompt spec, then fine-tuning if it will not stick
Fails multi-step reasoning entirely	Capability	Stronger model or decomposition. Training will not help
Correct but too slow or too expensive	Cost / latency	Distil to a smaller model on your task
Good on average, fails a specific input class	Coverage	Find the class, add examples or a routed special case

The ladder, in the order we actually climb it

Each rung costs roughly an order of magnitude more engineering time than the one below and is harder to reverse. Climb in order, and stop the moment you hit the bar, because a rung you did not need is a rung you maintain forever.

Rung one: write the prompt as a specification. Most prompts that "do not work" are underspecified rather than insufficient. Name the role, state the task in one sentence, enumerate the rules as a numbered list, define the output format exactly, and say what to do when the input does not fit — that last one is missing from almost every prompt we are shown. Then test each rule individually against your eval suite. Hours of work, instantly reversible, and it resolves a surprising share of problems that arrived labelled as needing training.

Rung two: few-shot examples, ideally retrieved. Three to ten examples of the exact input-output mapping you want, covering the edge cases you care about. Retrieving examples similar to the current input generally beats a fixed set, and can approach fine-tuning quality on format and style tasks. It costs input tokens on every call, which is the honest reason to move past it eventually.

Rung three: decompose the task. One call doing extraction, judgement and formatting simultaneously is three chances to fail combined into one unauditable step. Splitting it produces steps you can test and grade separately, and each is easier. It costs more calls and more latency, and it is very frequently the intervention that actually works when people were about to train.

Rung four: retrieval. If the gap is knowledge, this is the answer regardless of what else you do. Fetch the facts, put them in the context with citations, and the model reasons over material it can see instead of material it half-remembers. Retrieval is its own engineering discipline with its own failure modes, but the alternative — training facts into weights — is worse on every axis including accuracy.

Rung five: update weights. Supervised fine-tuning, usually with a low-rank adapter, on a few hundred to a few thousand curated examples. Now you have a fork of a model, a data pipeline, a training job, an eval gate and a rollback plan. It is the right answer for a real class of problems and it is a standing commitment, not a task.

The bottom row deserves an explicit warning. Fine-tuning on a set of confident, well-formed answers teaches the model that confident, well-formed answers are what you want — including for inputs where the correct behaviour is to say it does not know. Unless your training set deliberately contains examples of abstention, refusal and "insufficient information," tuning tends to make a model more fluent about being wrong, not less wrong.

How much data, and where it comes from

The numbers are smaller than people expect and the quality bar is higher than people expect. Both facts change how you plan.

For format and style, several hundred clean examples usually move the needle and a thousand is often plenty. This is the cheapest win in fine-tuning and the most reliable.

For narrow classification, a few thousand examples with good coverage of the confusing boundary cases. Class balance matters more than volume; a thousand well-chosen examples across the confusion boundary beat ten thousand drawn from the easy middle.

For distillation, generate from the large model, filter hard, and expect to need tens of thousands of examples if the task is broad, far fewer if it is narrow. The filtering is the work: outputs the large model got wrong become errors you have baked in permanently.

Quality beats quantity by a wide margin. A hundred examples a domain expert has personally checked will outperform five thousand scraped from logs and never reviewed. Contradictory examples are worse than no examples, because the model learns the inconsistency. Budget for the review, not the collection.

The best source is almost always your own production traffic with human correction attached — real inputs, real distribution, and an expert's fix as the target. Which means the highest-value thing you can build before any tuning project is a correction interface: a way for the people who already review the output to save their edit as a labelled pair. Six months of that gives you a dataset nobody can buy.

Adapters, full fine-tuning, and what each does to serving

Three mechanisms, and the difference matters more for operations than for quality.

Low-rank adapters. Train a small number of additional parameters while the base weights stay frozen. Cheap to train, small to store, and — the part that matters — multiple adapters can be served against one loaded copy of the base model, swapped per request. If you need per-customer behaviour, this is the only approach that does not multiply your serving footprint by the number of customers. For behaviour and format work it is usually indistinguishable from full fine-tuning in quality.

Full fine-tuning. Every weight updated. More capacity for large behavioural shifts, considerably more compute, and each result is a complete separate model to store and serve. Reach for it when adapters have measurably fallen short, not by default.

Continued pretraining. Further training on a large corpus of your domain text with no labels. Occasionally right when your domain language genuinely differs from ordinary text — specialised notation, an unusual register — and it needs a lot of data and real expertise. It is rarely what a product team needs and frequently what gets proposed.

One serving consequence people discover late: a fine-tuned model may be excluded from the fastest managed inference paths, from certain caching behaviours, or from provider-side optimisations available to base models. Check what you give up before committing, because a tuned model that costs more per token than the base model it replaced has to earn that back in quality.

The maintenance bill nobody prices

A prompt change is a text edit: reviewed, deployed, reverted in a minute if it is wrong. A fine-tuned model is a fork of somebody else's artefact, and forks have a carrying cost that shows up quarters later.

Base models are deprecated. The model you tuned will eventually be retired or superseded, and your adapter does not transfer. That means re-running the pipeline, re-validating, and re-shipping — on somebody else's schedule, not yours. Plan on this happening once or twice a year and budget the engineering time as recurring rather than one-off.

Your data drifts. The behaviour you trained in was correct for last year's policy, last year's product, last year's customers. Nothing signals when it stops being correct except a metric you have to be watching.

Every retrain needs the full eval gate. Not a spot check. A tuned model can improve on your target metric and quietly regress on something adjacent that nobody thought to measure — refusal behaviour and instruction-following outside the tuned task are the usual casualties.

You lose free upgrades. This is the cost people feel most keenly in hindsight. Teams on prompts alone got better results the day a stronger base model shipped, at no engineering cost. Teams on a tuned fork got a project.

Intervention	Time to first result	Ongoing cost	What invalidates it
Prompt as specification	Hours	Input tokens; occasional revision	A model upgrade changing how it reads instructions
Few-shot, retrieved	Days	Input tokens on every call; example upkeep	Task distribution shifting away from the examples
Decomposition	1–2 weeks	More calls, more latency, more code	Requirements changing the step boundaries
Retrieval	2–6 weeks	Index maintenance, embedding refresh, chunking upkeep	Source documents changing shape
Fine-tuning with adapters	2–8 weeks incl. data	Labelling, retraining 2–4×/year, full eval each time	Base deprecation, data drift, task change

A protocol that decides it in a week

The order here matters, and step two is the one that saves the most time.

Day one: build the eval before anything else. Fifty to two hundred cases from real traffic, with correct answers written by someone who knows the domain, graded programmatically wherever possible. Without this you cannot tell whether any intervention worked, and every argument afterwards is about impressions.

Days two and three: find the prompting ceiling with the strongest model you can afford. Spec-style prompt, retrieved few-shot examples, decomposition if the task has natural seams. Measure. This number is the reference point for everything that follows, and it is frequently high enough that the project ends here.

Day four: measure the small model unaided. Same eval, same prompts, on the model you would want to tune. The gap between this and the previous number is what tuning has to close.

Day five: decide with the numbers in front of you. If the strong model with good prompting cannot reach the bar, tuning a weaker model almost certainly will not either — you have a capability or a task-definition problem, and more training data will not fix it. That single inference has ended more unnecessary tuning projects for us than any other argument. If the strong model clears the bar comfortably and only cost or latency is the issue, distillation is well-founded and you can start collecting data with confidence.

The mistakes we get called in to fix

• Fine-tuning to inject facts that change monthly, then retraining every time they change
• Tuning before writing an eval, leaving nobody able to say whether it helped
• A random train/test split on data with near-duplicates, inflating the measured gain
• Training only on confident answers, teaching the model never to say it does not know
• Skipping the prompting ceiling measurement, so the reference point never existed
• Five thousand unreviewed examples scraped from logs, with contradictions inside
• No plan for base-model deprecation, discovered when the deprecation notice arrived
• A tuned model that costs more per token than the base model it replaced

If you do tune, the sequence that works

Step six is not optional and is the one that gets cut. A tuned model in production without a documented path back to the base model is a single point of failure with no fallback, and the retrain runbook is what stops the next base-model deprecation from being a fire.

Before you start a tuning project

• The gap is named: knowledge, behaviour, capability, or cost
• An eval suite exists and was written before any intervention
• The prompting ceiling on the strongest available model is measured and recorded
• The test set is split by entity or time and has not been looked at
• Every training example has been reviewed by someone who knows the domain
• Abstentions and refusals are represented in the training data
• Adapters chosen over full fine-tuning unless a measurement said otherwise
• Serving cost of the tuned model checked against the base model
• Retraining budgeted two to four times a year as recurring work
• A rollback to the base model is one flag away and has been tested

Bottom line

Prompting and fine-tuning solve different problems, and the expensive mistake is using the second to attack the first. Knowledge gaps go to retrieval. Capability gaps go to a stronger model or a decomposed task. Behaviour gaps and cost gaps are where weight updates earn their keep, and there they are genuinely the best tool available. Climb the ladder in order, measure the prompting ceiling before you commit, and price the fine-tune as an ongoing obligation rather than a project — because that is what it is, and it is worth it exactly when the behaviour or the cost saving is worth it.

Frequently asked questions