When an LLM is the wrong tool

The model always answers, and that is the problem

Every other tool in software tells you when it does not know. A parser throws an error on a malformed file. A database returns zero rows. A validation rule rejects the input and names the field. A language model does none of that. Ask it for the total of a column and it will give you a number. Ask it for the effective date of a contract that has no effective date and it will give you a date. The output is fluent, formatted, and confident in exactly the cases where a stricter tool would have stopped and told you something was wrong.

This is why the pilot goes so well and the rollout goes so badly. In a pilot, a person reads every output, and a person reading every output catches the wrong ones. At volume nobody reads every output, and the errors do not announce themselves — they look exactly like the correct answers. A parser that fails on 3% of inputs produces 3% obvious failures. A model that is wrong on 3% of inputs produces 3% invisible failures distributed evenly through the work.

So the question in front of a buyer is never “can a language model do this.” It nearly always can, to some standard, in a demo. The question is whether the job has a shape that a model is good at, and whether the cost of the errors it will make is one you can absorb.

Four questions that decide the tool

These are the questions we ask in a scoping conversation, roughly in this order. They take about twenty minutes and they settle most cases before anyone writes code.

1. Is there exactly one correct answer, and can it be computed? If the answer is a sum, a difference, a lookup, a match against a list, or the output of a formula somebody already wrote down, a model is the wrong tool. Not because it cannot get it right, but because it can get it right 99 times and wrong once, and you will not know which time. Arithmetic belongs in code. Code that adds a column adds the column, every time, and a test proves it.

2. Does the input already have structure the model would have to discard? A CSV, a database table, an XML feed, a fixed-layout form, an API response — these have named fields and known types. Feeding them to a model converts a reliable structure into text, asks the model to rebuild the structure it just destroyed, and charges you for the round trip. If a field is at a known position in a known format, read the field.

3. What does a wrong answer cost, and who would notice? This is the question that separates the two halves of the decision. If a wrong answer produces a slightly worse draft that a person edits anyway, the cost is near zero and a model is a good fit. If a wrong answer changes an invoice, a payment, a credit decision, a medication list, or a number that appears in a filing, then the cost is not the error — it is the discovery of the error six months later, plus everything downstream of it.

4. How many distinct inputs are there, really? A surprising number of “we need AI to handle the variety” problems turn out to have eleven variants covering 96% of volume. Count before you build. If eleven templates cover almost everything, write eleven parsers and route the remainder to a person. That system is cheaper, faster, auditable, and it tells you when it does not know.

Jobs where a plain method wins outright

These are not edge cases. In our experience they account for a large share of the work that gets proposed as an AI project.

The job	What gets proposed	What usually wins
Totals, variances, aging, any arithmetic	“Ask the model to compute it”	SQL or a few lines of code. Exact, testable, free at any volume.
Pulling fields from a fixed-layout form	Document AI over the whole page	Positional extraction with a validator. Use a model only for the layouts that vary.
Matching records to a controlled list	Semantic matching	Normalization plus fuzzy string matching, then a model only for the leftovers.
Routing by rules the business already wrote	A classifier	The rules, in code. They are already documented and they can be changed by the people who own them.
Deduplicating customers or vendors	Embeddings and a vector search	Blocking keys plus a scoring function, with a review queue for the middle band.
Forecasting a stable series	A large model with the history in context	A seasonal statistical baseline. Beat it before you replace it; often nothing does.
Scheduling, routing, allocation	“Let the model figure out the plan”	A constraint solver. It respects the constraints because it is built to; a model approximates them.
Keyword lookup in a small corpus	Retrieval over embeddings	A proper full-text index. Faster, cheaper, and it explains why a result matched.

The pattern behind the table is worth stating plainly. A language model is a good general reader of messy human text and a poor engine for anything with a defined right answer. When the job is “understand this paragraph well enough to draft a reply,” it is excellent. When the job is “produce the number,” it is the most expensive calculator ever built and the only one that occasionally makes things up.

Where a language model is genuinely the right tool

To be fair to the technology, there is a real and growing set of jobs where nothing else comes close, and the shape is consistent: the input is unstructured human language, the output tolerates variation, and a person or a downstream check absorbs the errors.

Drafting where a human edits. Replies, summaries, first-pass documentation, meeting notes, translations of internal jargon into plain language. The edit loop is the error correction, and it was going to happen anyway.

Reading documents nobody standardized. Contracts from four hundred counterparties, inspection notes typed by field crews, correspondence going back a decade. There is no schema to parse against because there was never a schema. This is the strongest case there is.

Classification with fuzzy boundaries. Sentiment, intent, topic, severity — categories that a person can apply consistently but cannot write a rule for. A model trained on a few hundred labeled examples, or prompted with them, does this well.

The long tail after a cheap method has taken the head. This hybrid is the design we recommend most often. Deterministic code handles the 85–95% of inputs that follow a known shape, at effectively zero marginal cost, and the model handles the remainder. Volume through the expensive path drops by an order of magnitude, and the parts that must be exact are exact by construction.

The cost shape is different from what buyers expect

Traditional software has a large build cost and a marginal cost near zero. Model-based software inverts part of that: the build can be shorter, and then every single call costs money forever. Buyers who have only ever bought the first shape budget for the wrong thing.

Work the arithmetic before you commit. A two-page document is roughly 1,000 to 1,500 words, which lands somewhere around 1,500 to 2,500 input tokens once you add instructions and examples. At mid-tier hosted model prices in 2026, that is a fraction of a cent per document — which sounds like nothing until you multiply. A hundred thousand documents a month, with a retry rate and a second verification pass, commonly lands in the low hundreds to low thousands of dollars monthly. That is a real number, it recurs, and it grows with your business rather than with your headcount.

Then add the two costs that never make it into the vendor estimate. The first is verification: if a person checks even 10% of outputs, that person is now a permanent line item, and at high volume they cost more than the inference. The second is the errors you do not catch, which show up as a rework project, a restated report, or a customer credit.

Latency has a similar shape. A parser returns in single-digit milliseconds. A model call is typically one to several seconds, and a chain of three calls with a retry can reach ten. If the output has to appear while a customer waits, that budget is decided before the architecture is.

Determinism is worth more than most buyers price it at

Run the same input through a parser twice and you get the same output twice. Run it through a model twice and you may not, even at the lowest temperature setting, and definitely not after the provider updates the model underneath you. Most of the time that variation is harmless. It stops being harmless the moment somebody asks why last quarter's report does not reproduce.

If your output ever has to be explained to an auditor, a regulator, a customer disputing a charge, or a court, the important property is not accuracy. It is that the same input produces the same output and that you can show the path from source to answer. That is achievable with a model in the loop, but it takes deliberate engineering — pinned model versions, stored prompts, logged inputs and outputs, and a deterministic layer around anything that becomes a number. It is not free, and it is not what you get by default.

How to test the boundary in about a week

There is a cheap experiment that settles the argument, and it is the same one every time.

Build the stupid baseline first. Rules, regular expressions, a lookup table, whatever the least clever thing is. Give it two or three days. Then measure it on a real sample — a few hundred items, drawn from actual volume rather than the clean examples someone kept.

Then measure the model on the same sample, with the same scoring. The comparison is only meaningful if the test set was fixed before either method was tuned, and if the person scoring the outputs cannot tell which system produced which.

Then compare on four axes, not one. Accuracy, cost per thousand items, latency at the required percentile, and what happens on the inputs each method gets wrong. That last one decides more cases than accuracy does. A baseline that fails loudly on 8% of inputs is often better operationally than a model that fails silently on 3%.

We have run this comparison many times and it does not always go the same way. That is the point of running it. What it reliably does is convert a two-month argument between the engineering team and the vendor into a one-week measurement that both sides accept.

The mistakes we see most

• No baseline was ever built, so nobody knows what the model is actually beating
• Accuracy measured on clean examples that somebody hand-picked months ago
• The model does the arithmetic instead of calling code that does the arithmetic
• Structured input converted to text and then reconstructed by the model, at cost
• No cost ceiling, so a retry loop or a traffic spike produces a surprising invoice
• Verification treated as temporary when it is a permanent operating cost
• Model version unpinned, so behavior changes the week the provider ships an update
• Failure mode never specified — nobody decided what the system does when it is unsure

When we tell people not to hire us

There are three situations where we say so directly, and they come up more often than you would guess from how vendors talk.

The first is when the real problem is that the data does not exist or nobody can get to it. No model fixes an access problem, and a six-month project that ends with a request for a database export was a six-month project that should have been a two-week one.

The second is when an off-the-shelf product already does 90% of it. Custom software is worth building when the thing you need is genuinely specific to how you operate. It is not worth building because the available product is mildly annoying. We would rather tell you the product name than sell you a year of engineering.

The third is when the job is a spreadsheet that one person maintains and the actual constraint is that person's time. Sometimes the answer is to hire a second person. That is not a project we can bill for, and it is still the right answer.

Before you commit to a model

• Somebody built the dumb baseline and wrote down its score
• The test set was fixed before either approach was tuned
• Every number the system outputs is computed in code, not generated
• The cost per thousand items is known, and so is the monthly total at real volume
• The latency budget is written down and measured at the 95th percentile
• The system has a defined behavior when it is not confident
• Model version, prompt and inputs are logged for every call
• Verification effort is in the operating budget, not the project budget
• Somebody named the volume of inputs that are genuinely unstructured

Bottom line

Language models are a real capability, and there is a class of work — reading messy human text, drafting things a person will edit, sorting into categories nobody can write a rule for — where nothing else is close. Outside that class they are an expensive way to get an approximate answer to a question that had an exact one. The way to tell the difference is not judgment or taste. It is a week of measurement against the least clever alternative, on a sample you did not curate, scored on cost and latency and failure mode alongside accuracy. Run that week before you commit a budget. It is the cheapest thing in the whole project.

Frequently asked questions