Cost per successful task, not per token

The denominator is the whole argument

Nearly every AI budget conversation we walk into is being conducted in the wrong units. The finance team has a monthly invoice from a model provider. The engineering team has a price per million tokens and a chart of usage. Both are accurate, and neither answers the question actually on the table, which is some version of: does this workflow cost less than it used to, and what happens to that number if volume triples? Token price cannot answer that, because token price is a numerator with no denominator attached.

The unit that answers it is cost per successful task. One invoice processed and posted. One support case closed without a human touching it. One document extracted, validated and accepted downstream. The definition changes by business; the shape does not. Total cost to produce outcomes, divided by outcomes actually produced.

The moment you compute it, several arguments end. The model comparison stops being about price per million tokens and starts being about which model produces more finished work per dollar. The build-versus-buy question gets a number. The pricing question — what can we charge, and does the margin survive a heavy user — becomes arithmetic instead of a debate. And the uncomfortable one gets asked honestly: is this workflow cheaper than the people who used to do it, or have we moved the cost somewhere it is harder to see?

Define success first, or the number is decoration

The definition of a successful task is the load-bearing decision, and it has to be made with the operations people, not by engineering alone. A useful definition has three properties. It is observable from data you already have or can cheaply capture. It matches what someone downstream would call done. And it does not quietly count outputs that a human had to redo.

Where teams go wrong is counting API calls that returned a response as successes. A response is not an outcome. In a document extraction workflow, the plausible definitions are: the model returned valid structured output; the output passed validation rules; the output was accepted by a reviewer without edits; the output was accepted after minor edits; or the item was posted to the system of record and never came back as an exception. Those five definitions can differ by a factor of two or more in the resulting cost, and only one of them is the one the business cares about.

Pick the strictest definition you can observe, and write it in a sentence any operator would recognize. Then count everything that fails it as a cost with no output, because that is exactly what it is.

Working the arithmetic all the way through

Take a document intake workflow with round illustrative numbers. Ten thousand documents a month. Each document costs, in model calls, one unit of inference. Suppose that unit works out to four cents once you have added the retrieval calls, the extraction call and the validation call together. That is forty cents per ten documents, four hundred dollars a month, and if the story stopped there this would be an easy decision.

It does not stop there. Eighteen percent of documents fail validation on the first pass and are retried with a longer prompt and a larger model, which costs roughly three times as much. Of those, a third still fail and route to a human reviewer. The reviewer takes seven minutes on a routed item. Separately, four percent of items that passed validation are later found wrong downstream and come back as exceptions, each costing a reviewer twenty minutes plus a correction in the system of record.

We have left the human lines as descriptions rather than dollars on purpose, because the loaded cost of a reviewer minute varies enormously by role, geography and whether the work is done by employees or a vendor, and inventing a number for you would be worse than useless. Put your own in. When you do, the shape that appears in most of these exercises is that the two human lines together are several times the two inference lines, and often an order of magnitude more.

That is the entire practical value of the exercise. Six hundred dollars of inference is not where the money is. Nine hundred and thirty human touches are where the money is, and every one of them exists because something upstream did not work the first time.

The retry tax, and why a cheaper model can cost more

Once the denominator is in place, an unintuitive result becomes obvious. Suppose you move to a model priced at a third of your current one, and its first-pass success rate drops from eighty-two percent to sixty-eight. Inference cost falls by two-thirds. Human touches rise by roughly three quarters. If human touches were already the larger share — and they usually are — the switch increases total cost while making the model bill look like a victory.

The reverse also happens and is easier to sell internally once you can show it. Moving to a more capable and more expensive model, or spending three times the tokens on a careful multi-step prompt with a verification pass, can pay for itself many times over if it removes even a few points of rework. The break-even is a simple relationship worth writing on a whiteboard once: the extra inference cost per task has to be smaller than the reduction in failure rate multiplied by the cost of a failure. When a failure costs twenty minutes of a specialist's time, that inequality is very forgiving toward spending more on inference.

Two cautions. First, this only works if you actually measure the failure rate under both options on your own traffic, not on a benchmark; a model that looks better in general may be worse on your specific documents. Second, the relationship inverts at the top end. Once first-pass success is already high, further inference spending buys almost nothing, and you are back to ordinary cost control — batching, caching, shorter prompts, smaller models for the easy slice.

Routing is the lever the arithmetic points at

The natural conclusion from the break-even is not one model but two. Most task streams are not uniform: some fraction is easy and some fraction is hard, and the hard fraction generates nearly all of the rework. If you can identify the hard fraction cheaply — by document type, by length, by a confidence signal, by a fast classifier — you can spend little on the easy path and a great deal on the hard path, and end up below both single-model options.

The engineering caution is that routing adds a component that can be wrong, and a misrouted hard item is worse than one that never had a fast path. Route conservatively, measure the routing decision itself as if it were a model output, and keep an escape valve where a low-confidence result on the cheap path is re-run on the expensive one. The added inference is small compared with a human touch.

The lines people leave out

A cost model that only contains inference will understate the total, sometimes badly. The lines most often missing:

Failed tasks that produced nothing. They consumed tokens and produced no outcome, so they belong entirely in the numerator and not at all in the denominator. Teams frequently divide by attempts rather than successes, which flatters the number by exactly the amount you most need to see.

The evaluation and observability tail. Trace storage grows with volume, nightly eval runs cost real inference, and a model acting as judge on every production sample is itself a meaningful spend at scale. Small per task; not small in aggregate.

Data preparation that recurs. Re-embedding a corpus after a model change, re-running extraction when a schema moves, and the storage that accumulates because nobody set a retention policy.

The engineer maintaining it. A pipeline with prompts, evals, routing and a vendor relationship is not zero-maintenance. Amortize a fraction of an engineer across the volume; it is a genuine per-task cost and it does not go away.

The cost of a wrong answer that nobody caught. Hardest to quantify and often the largest. In a regulated workflow it can dwarf everything above. If you cannot put a number on it, at least put it in the model as a named line with a question mark, so it does not silently become zero.

Instrumenting it: six fields on every task

The reason most teams cannot compute this number is not analytical difficulty. It is that the data was never captured with a task identifier on it, so tokens and human minutes live in different systems with no way to join them. Fix that once and the number becomes a query rather than a project.

Emit one record per task attempt, with: a task id that is stable across retries; a tenant or account; the task type, at whatever granularity your routing uses; the full token counts for every model call in the attempt, including the ones inside tools, with cached and uncached input counted separately; a terminal status from a small fixed set — succeeded, failed, escalated, abandoned; and a human minutes field, populated from your review tooling.

Three notes from doing this several times. Add a parent task id so a retry chain rolls up to one task, or your denominator will double-count. Record the model and prompt version on every attempt so a cost change can be attributed to a specific release rather than argued about. And capture the human minutes even crudely — a queue timestamp difference is imperfect and is vastly better than nothing, because the human line is the one that decides the answer.

With that in place, cost per successful task is a group-by. So is cost per successful task by tenant, which is the query that tells you whether your pricing survives your heaviest customer, and which surprises people more often than any other output of this exercise.

Where the number lies to you

Three failure modes worth knowing before you present it.

Averages across a skewed distribution. Task cost is usually long-tailed: most tasks are cheap and a small tail is very expensive. A mean sits in a place where few tasks actually live. Report the median and a high percentile alongside it, and forecast with the distribution rather than the average, because your worst month is made of tail.

Cherry-picked success definitions. If the definition of success drifts toward what makes the number look good — counting items that a reviewer fixed as successes, for instance — the metric stops being a measurement and becomes advocacy. Freeze the definition in writing, version it, and note the version on every chart.

Marginal versus fully loaded. The cost of processing one additional document is not the cost of running the workflow divided by document count. Both are legitimate numbers for different questions. Use marginal for pricing an incremental unit and fully loaded for deciding whether the workflow earns its place. Label which one you are showing, every time.

The mistakes we get called in to fix

• Dividing by attempts instead of successes, which hides the entire retry problem
• A model bill treated as the cost of the workflow, with human rework in a different department's budget
• No task id joining tokens to outcomes, so the number requires a manual reconstruction each time
• Switching to a cheaper model on price per million tokens alone, and discovering the rework increase a quarter later
• Reporting a mean on a long-tailed distribution and being surprised by the invoice in a heavy month
• Tool-call and embedding tokens uncounted, because they are emitted deep inside a library
• Cached and uncached input priced identically in the model, making caching work look invisible
• A success definition that quietly changed between two charts presented in the same meeting

Before you take the number to a budget conversation

• Success is defined in one sentence an operator would agree with, written down and versioned
• Failed and abandoned attempts are in the numerator and not the denominator
• Human minutes are in the model, at a loaded rate your finance team supplied
• Retries roll up to the parent task, so nothing is counted twice
• Every model call is counted, including tool calls, embeddings, and eval judges
• Median and a high percentile are shown, not only the mean
• The number is broken out by tenant and by task type, not just in total
• Marginal and fully loaded are labeled distinctly wherever the number appears

Bottom line

Token price tells you what the meter reads. Cost per successful task tells you whether the thing is worth running, and it is the only unit in which the real decisions can be made: which model, whether to route, what to charge, and whether the workflow beats the process it replaced. Building it is mostly a logging problem, and once the join exists it is a query you can run every month. The finding is usually the same, and it is worth expecting: the model is not the expensive part. The rework is. Which means the highest-return engineering work is almost never squeezing the token bill — it is removing the reasons a task fails the first time.

Frequently asked questions