Count tokens, not requests
Almost every cost surprise we are asked to investigate has the same origin: a system that measures requests. The rate limiter counts requests, the quota counts calls, the capacity plan is written in queries per second, and none of those correlate with what you pay. One document goes through at eight hundred tokens and the next at four hundred thousand, on the same code path, and the only number that tracks the invoice is the token count. Instrument that first, because every decision after this one depends on knowing where the tokens are.
When you do instrument it, the shape is usually the same. Input tokens dominate the volume and output tokens dominate the price, since output is billed at several times the input rate on current frontier models. A pipeline that sends a large context and returns two hundred tokens is input-heavy and its lever is caching. A pipeline that sends a short prompt and generates two thousand words is output-heavy and its lever is asking for less. Teams routinely apply the first fix to the second problem and wonder why nothing moved.
You are probably here because
- The bill went up and nobody can attribute the increase to a feature or a customer
- Finance wants a cost per document and all you have is a monthly total
- An overnight job is now taking eleven hours and nobody planned for that
- You suspect batching would help and are not sure which kind of batching is meant
The first two are an accounting problem and are solved by instrumentation. The last two are a design problem and are solved below.
Three different things are called batching
The word covers three unrelated techniques with different economics, and conversations go badly when two people mean two of them. Separate them before choosing.
Asynchronous batch submission. You hand the provider a set of independent requests, it processes them off the interactive path, and you collect results later. The saving is a straight discount, typically around half, in exchange for latency measured in minutes to hours. Nothing about the prompts changes.
Concurrency management. Running many requests in parallel against the same endpoint to raise throughput. This is a scheduling concern, not a cost one — it makes a backfill finish sooner without changing the price of a single token — and its constraints are your rate limits and your error handling.
Multi-item prompts. Putting ten records in one request and asking for ten answers. This genuinely reduces token spend by amortizing the instructions and shared context across items, and it is the one with real accuracy risk.
| Technique | Typical saving | What you give up | Use when |
|---|---|---|---|
| Async batch endpoint | ~50% on everything submitted | Interactivity; results arrive unordered, hours later | Any work not on a user's critical path |
| Prompt caching | Up to ~90% of the cached input portion | Prefix discipline; a modest write premium; short cache lifetime | A stable prefix reused across many requests |
| Multi-item prompts | 30–70% depending on instruction size | Accuracy on the tail; harder error isolation | Short, independent, homogeneous items |
| Model routing | Large but highly variable | A second quality bar to maintain and monitor | A clearly separable easy majority |
| Concurrency management | None on price | Nothing, if rate limits are respected | Throughput, not cost |
The batch endpoint is the least clever and most reliable saving
It requires no prompt changes, no quality tradeoff and no judgment. You submit a set of requests each carrying an identifier you choose, poll until the set completes, and read the results. The discount is roughly half, and the only real requirement is that the work can tolerate a delay.
Far more work qualifies than teams initially think. Nightly enrichment, document backfills, offline classification and tagging, evaluation runs, dataset generation, summarizing yesterday's activity, precomputing anything a user will ask for tomorrow. Any of these on the synchronous path is paying double for latency nobody uses.
Two implementation notes matter. Results come back in arbitrary order, so key them by your identifier and never by position — this is the defect we find most often in first batch integrations, and it corrupts data silently. And each item succeeds or fails independently, so handle a partial batch as the normal case rather than the exception: retry the failures, keep the successes, and never discard a completed set because one item errored.
Multi-item prompts: real savings, real risk
If your instructions are eight hundred tokens and each item is fifty, one item per request means 94% of your input is overhead. Ten items per request drops that to about 60%. The saving is genuine and can be substantial where instructions are long relative to the data.
It also introduces failure modes that single-item prompts do not have, and they are subtle enough to survive testing.
Cross-contamination. Item three influences the answer for item seven. On classification tasks this shows up as answers drifting toward whatever the earlier items were, and it is invisible unless you evaluate batched output against single-item output on the same data.
Position effects. Quality on the tenth item is often measurably lower than on the first. Test it directly by shuffling the order and comparing; if accuracy depends on position, your batch is too large.
Alignment loss. The model returns nine results for ten inputs and everything after the gap is attributed to the wrong record. Defend against this structurally: require an explicit identifier per item in the output and match on it, rather than trusting the order of an array.
Blast radius. One malformed item can spoil a whole request. With a batch of fifty, a retry costs fifty items of work.
Our default is a batch size between five and twenty, decided by measurement rather than by preference, with output identifiers required and a per-item validity check. Beyond about twenty the marginal saving flattens while the accuracy risk keeps climbing, which is a bad trade in both directions.
Order we apply the levers — value against effort and risk
Our working order across cost-reduction engagements. The top three are almost always worth doing; the bottom two need measurement first.
Caching, and the mistake that quietly removes it
Caching is covered in depth elsewhere, so here is only the part that governs a pipeline. A cached read costs a small fraction of the base input rate; writing the cache costs a modest premium above it. That asymmetry means caching is strongly positive for a prefix reused many times and mildly negative for one reused twice, so it is a design decision about your traffic pattern rather than a switch to turn on.
Because matching is on a prefix from the front of the request, the ordering rule is absolute: stable content first, volatile content last. In a pipeline the usual violation is a per-record header — a document id, a filename, a row number — placed above the instructions out of habit. It costs nothing to move it below and it is the difference between a 90% discount and none.
Verify with the usage counters rather than by reasoning. If cache reads are zero across repeated runs, something in the prefix is moving: an unordered serialization, a set iterated in hash order, a clock. Assert it in a test so that a change six months from now cannot silently reintroduce the cost.
Batch and cache interact, and the order of items matters
In a batch job, group items that share a prefix so cache entries stay warm, and process the groups together rather than interleaving them. A job that alternates between two document types with different instruction blocks pays the write premium on nearly every request; the same job sorted by type pays it twice. This is a two-line change to a job scheduler and we have seen it cut a nightly bill by more than half on its own.
Routing and cascades: the lever that needs a measurement
Sending everything to the largest model is the safe default and often the wrong one. Two patterns reduce cost, and both need evidence before they are trusted.
Static routing sends a known class of work to a smaller model. Short classifications, formatting, normalization and extraction from clean inputs frequently run at near-identical quality on a small model at a fraction of the price. The requirement is a labelled set demonstrating that quality is equivalent on your data, not a benchmark table.
Cascading tries the cheap model first and escalates when a confidence or validity check fails. It works when you have a reliable, cheap signal for "this went wrong": a schema violation, a failed cross-field check, a source span that does not appear in the input. It works badly when the escalation signal is the small model's own self-assessment, because a model that is confidently wrong will not escalate. Without a real check, a cascade adds latency and cost to the hard cases and delivers quiet errors on the ones it should have escalated.
Both need continuous monitoring rather than a one-time evaluation. Route mix drifts as your inputs drift, and a cascade that escalated 8% of traffic in March and 34% in September has become more expensive than the single-model design it replaced without anyone noticing.
Send a week of token data and we will tell you where it is going.
Email a week of per-request token counts split input, output and cached, plus a description of the pipeline, to contact@precisionfederal.com. You get back a written note on the three changes we would make first and roughly what each is worth. One business day, no charge, no meeting.
contact@precisionfederal.comPrice the unit of work, not the token
Cost per million tokens is the wrong unit for every conversation you need to have. The right unit is the business object: cost per document processed, per support ticket resolved, per contract reviewed, per customer per month. That number can be compared against what the work is worth, which is the only comparison that decides anything.
Computing it needs one field on every log line — the identifier of the unit of work — and an aggregation. It is a small piece of engineering and it changes conversations completely. It converts "our AI spend is growing" into "each processed invoice costs eleven cents against a thirty-cent budget, and the ninety-fifth percentile is forty cents because of a document class we could route differently."
Keep two numbers alongside it: the average and the tail. Average cost per unit sets your pricing; the tail sets your exposure. A pipeline averaging eleven cents with a ninety-ninth percentile of four dollars has a problem the average will never show you, and it is usually one input class behaving differently from the rest.
Guard the budget in code
A billing alert tells you money was spent. A guard prevents it. Put a check in the request path that knows a per-tenant and per-job token ceiling and refuses cleanly when it is exceeded, with an error that says which limit was hit and what to do.
The scenarios this prevents are common and expensive: a retry loop with no cap, an agent that cannot terminate, a customer who uploads a ten-thousand-page document into a per-page pipeline, a test harness accidentally pointed at production credentials over a weekend. Each of those has spent a month of budget in a day for somebody. Reject over-limit work at admission rather than partway through generation, since early rejection is both cheaper and faster to explain.
The same principle applies to capacity. Rate limits are usually expressed in tokens per minute as well as requests per minute, and a batch job saturating the token limit will throttle your interactive traffic. Give interactive and background work separate budgets, and let the background work absorb the throttling.
What we find when we audit a pipeline
- Everything on the synchronous path, including work no user has ever waited for
- A per-record identifier above the instruction block, holding the cache hit rate at zero
- Batch results keyed by array position rather than by the identifier, silently mismatching records
- Prose responses where a structured record was wanted, paying output rates for punctuation
- A cascade whose escalation signal is the model's own confidence, so wrong answers never escalate
- No per-unit cost, so nobody can say whether the feature is profitable
- Batch jobs and interactive traffic sharing one rate limit, so the overnight job throttles the product
- Retries without a cap, converting one bad input into an unbounded bill
A one-week cost pass
From an unexplained bill to a number per unit of work
The order matters. Every step after the first depends on the instrumentation from it, and teams that start at step five routinely optimize the wrong ten percent of their spend. We have not once found the expensive path to be the one people expected before they measured.
Common objections
Is a smaller model always the cheaper answer?
No, and the arithmetic is often the reverse. A smaller model that needs longer instructions, more examples, more retries and a repair loop can cost more per completed unit of work than a larger model that gets it right the first time. Compare cost per successful unit, including retries and human review time, not the per-token rate.
How much latency does the async batch endpoint really add?
Plan for hours and design so it does not matter. In practice sets often complete much sooner, but building anything that depends on a fast turnaround defeats the purpose. The right pattern is submit, persist the job with its identifier, and process results when they arrive. If a person is waiting, it does not belong there.
We are small. Is cost engineering premature?
The instrumentation is not — it takes a day and it is what makes every later decision possible, including pricing your own product. The optimizations mostly are. Log tokens with a unit-of-work tag from the start, add a per-tenant guard before you have a customer who can trigger it, and leave routing and cascades until the bill is large enough to justify maintaining a second quality bar.
Can we just cap max output tokens and be done?
That caps the bill and converts overspend into truncation, which is usually worse. A response cut off mid-structure is a defect that reaches a parser or a customer. Reduce output by asking for a shorter form — structured fields instead of prose, a summary instead of a restatement — and keep the cap as a safety limit that you monitor rather than as a cost control you rely on.
Before you call the pipeline efficient
- Tokens logged split input, output and cached on every request
- Every log line tagged with feature, tenant and unit of work
- Cost per unit published, with the average and the tail side by side
- All non-interactive work on the asynchronous batch endpoint
- Batch results keyed by identifier, with partial completion handled normally
- Prompts ordered stable-first, with a test asserting a cache hit
- Batch jobs grouped by prefix so cache entries stay warm
- Multi-item batch size chosen by measurement, with per-item identifiers required
- Per-tenant and per-job budget guards enforced at admission
- Interactive and background traffic on separate rate budgets
Bottom line
Inference cost is not mysterious, it is unmeasured. Log tokens with a unit-of-work tag and the picture resolves in an afternoon, usually into two or three input classes carrying most of the spend. Then apply the levers in order: move offline work to the batch endpoint for roughly half off with no quality tradeoff, restructure prompts so the stable part caches, cut output length, and only then consider multi-item prompts and routing, both of which trade accuracy risk for money and both of which need evidence from your own data. Finish with a guard in the request path, because the worst bills are never the average request getting slightly more expensive. They are one loop, one customer, or one weekend.
Frequently asked questions
Moving work off the interactive path onto an asynchronous batch endpoint, which is typically around half off with no change to prompts and no quality tradeoff. Most pipelines have more qualifying work than they realize: backfills, nightly enrichment, offline classification, evaluation runs and anything precomputed for tomorrow.
Between five and twenty for most tasks, decided by measurement. Score batched output against single-item output on the same data and shuffle the order to check for position effects; if accuracy depends on where an item sits, the batch is too large. Always require an identifier per item in the output and match on it rather than trusting array order.
Usually a per-record value placed above the instructions — a document id, filename or row number — which changes the prefix on every request. Move volatile content below the stable block. Also check job ordering: interleaving items with different instruction blocks pays the cache write premium repeatedly, while sorting the job by prefix group pays it once per group.
Only if you have a cheap, reliable signal for failure that is independent of the model's own confidence — a schema violation, a failed consistency check, a source span absent from the input. Escalating on self-reported confidence does not work, because the errors you most need to catch are the confident ones. And monitor the escalation rate over time; drift can quietly make a cascade more expensive than the design it replaced.
Cost per unit of work — per document, per ticket, per customer per month — with the average and the tail shown together, not a monthly total or a per-token rate. That is the number that can be compared to what the work is worth, and the tail is what tells you the exposure that the average is hiding.