The gap between the price sheet and the invoice
A per-token price is a clean number and it invites a clean forecast: multiply expected calls by expected tokens, done. Then the first month's bill arrives at three or five or twenty times the estimate, and the postmortem always finds the same thing. Nobody was wrong about the price. They were wrong about how many tokens a single unit of user-visible work actually consumes, because between the price sheet and the invoice sit seven multipliers, each of them ordinary, each of them compounding.
This article is the demand side of inference economics. The supply side — what it costs to run the hardware — is a different subject with different physics. Here we assume you are paying somebody a published rate and want to know what your product will really cost to operate.
1. The input-to-output ratio, and the price asymmetry. Output tokens usually cost several times what input tokens cost, because generating them is bandwidth-bound and serialised while reading input is parallel and compute-bound. A workload with a long prompt and a two-line answer behaves nothing like one that writes a page from a short instruction. Averaging them into a single blended rate is where most forecasts go wrong on line one.
2. The prompt tax you pay on every single call. A system prompt, a tool schema block, a few-shot set, a formatting spec. Two thousand tokens of preamble sent on every request in a conversation that reaches twenty turns is forty thousand tokens of preamble alone. Nothing about it is visible in the product and all of it is billed.
3. Conversation history, which grows quadratically. If each turn resends the whole transcript, a twenty-turn conversation does not send twenty turns' worth of tokens. It sends roughly two hundred and ten turns' worth, because turn n carries n turns of history. This is the multiplier that catches chat products by surprise in their second month.
4. Retries, repairs and validation failures. Malformed structured output that needs a second pass, a guardrail rejection, a timeout retried by the SDK, a repair prompt asking the model to fix its own JSON. A ten percent repair rate is a ten percent surcharge, and the repair call is often more expensive than the original because it carries the failed attempt as context.
5. Steps, if there is a loop. An agent doing a task in twelve steps makes twelve calls, each carrying accumulated context. This is the difference between a per-call cost and a per-task cost, and it is routinely a factor of ten to fifty.
6. Tokens you pay for and never see. Reasoning or thinking tokens are billed as output and are frequently the majority of output on hard problems. Judge calls in an eval harness, router calls that classify before dispatching, summarisation passes that compact context — none of them appear in the product and all of them appear on the bill.
7. Work that produced nothing. Abandoned sessions, cancelled streams where you generated four thousand tokens nobody read, development and eval traffic sharing the production key. Between five and fifteen percent of spend on the deployments we have audited belongs to this category, and almost none of it is measured.
You are probably here because
- The bill came in several times the forecast and the per-token price was right all along
- You need a per-seat price and cannot say what a seat costs to serve
- Spend is rising faster than usage and nobody can attribute the difference
- Somebody proposed switching to a cheaper model and nobody can say what that would save
Every one of those is answerable in an afternoon once requests are tagged and joined to outcomes. Almost nobody does that before the first surprise.
Working an example all the way through
Take a document question-answering feature. Suppose the model you use is priced at three dollars per million input tokens and fifteen per million output. Here is a forecast built the usual way, and the same forecast built honestly.
| Layer | Assumption | Tokens per task | Cost per task |
|---|---|---|---|
| The naive estimate | One call: 4,000 in, 500 out | 4,000 in / 500 out | $0.020 |
| + system prompt and tool schemas | 1,800 tokens of preamble per call | 5,800 in / 500 out | $0.025 |
| + retrieval context | 8 chunks, ~700 tokens each | 11,400 in / 500 out | $0.042 |
| + follow-up turns | Mean 3.2 turns, full history resent | ~44,000 in / 1,600 out | $0.156 |
| + repairs and retries | 12% of calls need a second pass | — | $0.176 |
| + abandoned and eval traffic | 9% of spend produces no user outcome | — | $0.193 |
Two cents became nineteen. Nothing in that table is exotic, nothing is a mistake, and every line is a decision somebody made for a good reason. The point is not that the naive estimate was careless — it is that a per-token price cannot express any of this, so a forecast built on one is structurally unable to be right.
Run this table for each of your top three features. It takes an afternoon and it is the single most useful artefact in the whole cost conversation, because it turns an argument about vendors into an argument about the fourth row, which is the one you can actually change.
Caching helps, and the break-even is real arithmetic
Prompt caching lets a provider keep the computed state of a prefix and skip recomputing it on the next call that starts with the same tokens. It is the closest thing to a free lunch in this subject, and it is not free.
The shape is consistent across providers even as details differ: writing to the cache costs somewhat more than a normal input token, reading from it costs a small fraction of one, and the entry expires after a short idle window measured in minutes. So the economics are a break-even on reuse count. If a write costs 1.25× a normal input token and a read costs 0.1×, you are ahead from the second hit onward, and the saving on a stable ten-thousand-token prefix read fifty times is close to the whole prefix cost.
Three practical rules follow. Put the stable material at the front — system prompt, tool schemas, shared reference text — and the variable material at the back, because caching is prefix-based and a single changed token near the beginning invalidates everything after it. Do not put a timestamp, a request id or a user name in the preamble, which is the most common way teams accidentally guarantee a zero percent hit rate. And check that your traffic actually reuses prefixes within the idle window: a per-user prefix on a product with a hundred daily users spread over a working day is a cache-write tax with no reads to pay it back.
Cost levers, ranked by saving against the risk of taking them
Saving weighted by how likely the change is to cost you quality. Judgment from work we have done, not a benchmark.
Instrument first, or none of the rest is possible
Everything above requires data most teams do not have, and the fix is a day of work that has to happen before you need it.
Emit one cost record per model call. Input tokens, output tokens, cached-read and cached-write counts, the model and snapshot that served it, latency, and whether the call succeeded. Providers return most of this in the response; the job is to persist it rather than discard it.
Tag every call with four dimensions. Feature, tenant or account, environment, and a task id shared by every call in one unit of user-visible work. The task id is the one people skip and the one that makes cost-per-task computable at all; without it you have a pile of calls and no way to group them.
Join spend to outcome. Did the task succeed? Did the user accept the result, edit it, or abandon it? A cost number with no outcome beside it lets you cut spend thirty percent and quality with it, and see only the good half. This join is what makes every subsequent optimisation argument settleable with data instead of opinion.
Backfilling tags is far harder than adding them
Every cost investigation we have been asked to run began with a month of untagged calls and ended with an estimate rather than an answer. Provider dashboards show spend by key and by model; they cannot show you which feature, which customer, or which task. Add the four tags before the first cost surprise, because the surprise is when you will want three months of history and it will not exist.
Send us a week of logs and we will find where the money goes.
Email a week of request logs with token counts, or just your call graph and prompt sizes, to contact@precisionfederal.com. You get back a written cost-per-task breakdown by feature, the three largest multipliers in your stack, and what we would cut first without touching quality. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comForecasting with a distribution, not an average
Per-user cost is not normally distributed and treating it as if it were will misprice a product. In every usage-based system we have measured, the top one percent of users consume somewhere between twenty and forty times the median, and the top ten percent account for well over half of total spend. A mean cost per user is therefore a number that describes almost none of your users.
Three consequences for anyone setting a price.
Seat pricing is a bet on the tail. A flat monthly seat with unmetered inference behind it is fine while your heavy users are a rounding error and dangerous the moment one enterprise account automates against it. Model the p95 and p99 user, not the mean, and decide in advance what you will do when someone lands above the p99.
Publish an internal cost ceiling per task and enforce it in code. A token budget per task, a step budget per agent run, a hard stop with a legible message. Without one, a single pathological input — a document that triggers a retrieval loop, a user who pastes a book — can consume a meaningful share of a monthly budget in an afternoon.
Forecast in cost per active user per month, with a range. Give the finance conversation a p50 and a p95 rather than a point estimate, and state the assumption each depends on. A range you can defend beats a single number that will be wrong in a direction nobody anticipated.
Savings that are not savings
Cost reduction has a failure mode that looks exactly like success on the dashboard, and all three of these show up regularly.
The cheaper model that needs more attempts. A model at a fifth of the price that succeeds sixty percent of the time where the expensive one succeeds ninety-two percent is not cheaper per completed task once retries, repairs and human correction are counted — and the human correction is the expensive part. Always compare on cost per success, never cost per call.
Shrinking context to save input tokens. Cutting retrieval from eight chunks to three saves real money and raises the miss rate. The failures then arrive as retries, escalations and support tickets, which cost more than the tokens saved and land in a different budget so nobody connects them.
Batch discounts on a latency-sensitive path. An asynchronous tier at a substantial discount is excellent for evaluation runs, backfills and offline enrichment. Putting an interactive feature behind it trades a fifty percent saving for a user experience nobody will use, which is a hundred percent saving of a different kind.
The honest test for any cost change is one sentence: run the eval suite before and after, and report cost per success alongside the quality number. A saving that has not been through that comparison is a hypothesis.
The mistakes we get called in to fix
- A forecast built on one call per task for a product that makes eleven
- Full conversation history resent every turn, with no summarisation and no window
- A timestamp at the top of the system prompt, guaranteeing a zero percent cache hit rate
- Untagged calls, so cost per feature and cost per customer cannot be computed at all
- No token or step budget per task, and one pathological input that proved it
- Development and eval traffic on the production key, invisible inside the same line item
- A model downgrade shipped without an eval run, trading quality for a saving nobody measured
- Seat pricing set from the mean user in a distribution with a forty-times tail
A one-week cost audit that pays for itself
Inference Cost Audit
Day three is the one people find unexpectedly useful. Open the transcripts of your five most expensive sessions and read them. In our experience one of them is a loop, one is a user doing something you never designed for and should probably support, and one is a bug. That is a better return than any pricing negotiation.
Before you sign a usage forecast
- Cost modelled per completed task, not per call and not per token
- Every call tagged by feature, tenant, environment and task id
- Spend joined to outcome so quality regressions cannot hide inside a saving
- Input and output priced separately, never blended into one rate
- Conversation history strategy chosen deliberately, with a growth curve drawn
- Cache hit rate measured, and nothing volatile sitting in the prefix
- Repair and retry rate measured and included in the per-task number
- Per-task token and step budgets enforced in code, with a legible stop
- Per-user cost forecast as a distribution with p50 and p95, not a mean
- Every proposed saving validated against the eval suite before it is claimed
Bottom line
Cost per token is a vendor's unit, not yours. Yours is cost per completed task, and the arithmetic between them is seven ordinary multipliers that compound quietly. Instrument first, because none of this is computable afterwards. Build the per-task table for your top features and the argument stops being about which provider is cheapest and starts being about calls per task and tokens per call, which are the things you control. Then measure every saving against quality, because the cheapest possible system is one that produces nothing anybody uses.
Frequently asked questions
Almost always because a single unit of user-visible work is several model calls, each carrying preamble and accumulated history, plus repairs and retries. The per-token price is usually correct; the tokens-per-task assumption is what was wrong. Building a per-task table with every layer itemised will normally find the whole gap in an afternoon.
Yes, when a stable prefix is reused several times inside the cache's idle window. A write costs more than a normal input token and a read costs a small fraction of one, so you are ahead from roughly the second hit. It saves nothing if anything volatile sits near the front of the prompt, since caching is prefix-based and one changed token invalidates everything after it.
Only after comparing cost per success rather than cost per call. A model at a fifth of the price that needs two attempts and occasional human correction is more expensive in every currency that matters. Route by difficulty instead: send the easy majority to the small model and keep the hard tail on the capable one, which usually captures most of the saving with none of the quality loss.
Feature, tenant or account, environment, and a task id shared across every call belonging to one unit of user work — plus the token counts and the model snapshot that served it. The task id is what makes cost-per-task computable. Add all of it before you need it; backfilling from untagged logs produces an estimate, not an answer.
Model the distribution, not the mean. Heavy users commonly run twenty to forty times the median, so price against the p95 user, enforce a per-task budget in code, and decide in advance what happens when an account lands past the p99. A flat seat with unmetered inference behind it is safe only until one customer automates against it.