Why fine-tuning does not stop a model from making up a number
Fine-tuning changes how a model sounds. It does not change whether the number it states was ever computed. That distinction decides whether a system survives an audit.
The most common plan we are handed reads: fine-tune the model on our data so it stops hallucinating. It is a reasonable sentence and it does not work, and the reason it does not work is worth understanding before anyone spends a quarter on it.
What fine-tuning actually changes
Fine-tuning adjusts weights so a model produces output that looks more like your examples. It learns tone, structure, vocabulary, the shape of a good answer in your domain. Those are real gains and they are worth having.
What it does not do is install a mechanism that prevents the model from emitting a token sequence that happens to be a number nobody calculated. Generation is generation. A well-tuned model produces a fabricated figure in exactly the register your experts would have used, which makes it harder to catch, not easier.
A fine-tuned model does not fabricate less. It fabricates more convincingly.
The test that separates the two
Take a question whose answer requires arithmetic over records — a total, a count, a ratio across a filtered set. Ask a fine-tuned model. Then ask whether the system can show you which rows produced the figure.
If the answer is a confident number with no row-level trace, the model computed nothing. It produced a plausible value. Sometimes that value is right, which is worse, because it builds trust that the next one will be.
What actually prevents it
The property you want is structural: the model is never the thing that states a fact. It selects an operation and arguments, a deterministic system executes that operation, and the returned value is what appears in the output.
- The model reads the question and decides what to compute
- A query or function executes against the real records
- The returned value is inserted, not generated
- Every sentence carries the identifier of the record behind it
- When no valid operation answers the question, the system says so
That last behavior is the one people underestimate. A system that declines is usable in a regulated setting. A system that always answers is not, regardless of its accuracy rate.
Where fine-tuning does earn its cost
Fine-tuning is the right tool for making the output read like your organization's work product — the sectioning, the hedging conventions, the house style for expressing confidence. It is very good at that and hard to replace.
The correct mental model is a division of labor. Architecture decides what is true. Fine-tuning decides how it reads.
What this changes about scoping a project
Projects scoped as tuning exercises budget for data preparation and training runs. Projects scoped correctly budget for the evidence layer underneath — the schema mapping, the query tools, the provenance chain, and the graded evaluation set that proves the fabrication rate.
In our experience the model work is the small share of the hours. The evidence layer is most of it, and it is also the part that makes the result sellable to anyone who has to defend it.
Frequently asked questions
Can you not just prompt the model to say when it is unsure?
You can, and it helps at the margin. It is a request, not a constraint. Under pressure — an ambiguous question, a gap in the data — a model instructed to be careful still produces an answer, because producing answers is what it does.
Does this mean fine-tuning is a waste?
No. It is the difference between a system that is correct and a system that reads like it belongs in your organization. Both matter. They are separate problems with separate solutions.
Need a system where every number can be traced?
We build models that compute rather than assert — every figure executed against the real record, every sentence carrying the source it came from, and an honest refusal when the data does not support an answer.