Three different things get called routing
Somebody looks at the monthly bill, notices that a summarization endpoint and a legal-clause extractor are both hitting the same large model, and proposes a router. Six weeks later there is a service that classifies incoming requests, three model integrations, two eval suites nobody maintains, and a bill that went down by eleven percent. Whether that was a good trade depends entirely on the volume, and almost nobody does the arithmetic first.
Start by separating three things that share a name and share nothing else.
Static assignment is choosing a model per task at configuration time. Classification goes to the small model, drafting goes to the large one, the decision lives in a config file and never changes at runtime. This is not routing. It is the thing most teams should do instead of routing, it captures a large share of the available savings, and it takes an afternoon.
Cascade is trying the cheap model first, checking the answer, and escalating when the check fails. The decision happens after the cheap attempt, with an actual answer in hand.
Predictive routing is a classifier that reads the request and picks a model before any generation happens. This is what people usually mean, and it is the hardest of the three to make work, because predicting difficulty from a prompt is a genuinely harder problem than checking an answer.
A fourth thing, failover, is routinely bundled in and should not be. Sending traffic to a second provider when the first one is degraded is an availability mechanism. It has different triggers, different tests, and a completely different cost-benefit case. Build it separately or you will end up with a system where a provider outage silently changes your output quality and nothing alerts.
You are probably here because
- The model bill grew faster than usage and somebody asked why
- An engineer has proposed a router and nobody can say what it would save
- You already built one and cannot tell whether quality dropped
- A single feature now depends on three providers and the on-call rotation noticed
The break-even section answers the first two. The cascade and operations sections answer the last two.
The arithmetic, with numbers you can substitute
Routing savings are the product of four terms: the share of traffic that can move down, the price gap between the models, the volume, and one minus whatever you waste on escalations and on running the router itself. Every one of those has to be estimated before the build, not after.
Take an illustrative workload. Two million requests a month, averaging fifteen hundred input tokens and four hundred output tokens. Suppose the frontier model costs about a cent and a half per request at that shape and the small model about a tenth of that. All-frontier is roughly thirty thousand dollars a month. If sixty-five percent of traffic can move down without an acceptable-quality loss, the gross saving is about seventeen and a half thousand a month.
Now subtract. A cascade pays for the cheap attempt on every escalated request, which at a thirty-five percent escalation rate adds back a few hundred dollars. A predictive router built as a small model call adds a few hundred more, plus a hop of latency on every single request including the ones that were going to the large model anyway. Call the net saving sixteen thousand a month against an initial build of three to five engineer-weeks and a standing maintenance cost of roughly a day a week for eval upkeep. That pays back inside a quarter and keeps paying. Build it.
Now change one number. Same design, fifty thousand requests a month instead of two million. Gross saving drops to about four hundred and forty dollars a month. The build cost has not changed. The eval maintenance has not changed. The extra on-call surface has not changed. That project will never pay back, and the engineer-weeks would have earned more somewhere else.
Between those two figures it depends on the price gap and on how bimodal your traffic is. The spread between a provider's flagship and its small model currently runs somewhere between five and forty times per token depending on which pair you pick, and that ratio moves several times a year. A design whose payback depends on a twenty-times gap should be rechecked whenever prices move, because the gap has historically narrowed at the bottom and not at the top.
| Design | Where the decision happens | Signal quality | Latency cost | Build cost |
|---|---|---|---|---|
| Static assignment | Config, per task | Your judgment, checked offline | None | Hours |
| Cascade with a verifier | After the cheap attempt | Strong — you are checking a real answer | Escalations pay both calls in sequence | 1–2 weeks |
| Predictive router | Before generation | Weak to moderate — difficulty from a prompt | One extra hop on every request | 3–6 weeks incl. eval |
| Failover | On provider error or timeout | Not a quality decision at all | Retry latency on the failed call | Days, plus a quality alarm |
The number you need first is cost per accepted answer
Price per million tokens is the wrong denominator and it is the one everyone starts with. What you actually care about is the cost of one answer that survives whatever check stands between the model and the customer — a human reviewer, a schema validator, a retry loop, a downstream system that rejects malformed input.
A small model at a tenth the price that needs two attempts and a human correction on a fifth of its outputs is not a tenth of the cost. If a reviewer spends four minutes fixing one output in five, at a loaded rate of eighty dollars an hour you have added about a dollar per five answers, which swamps a token price measured in fractions of a cent. That arithmetic decides more routing questions than any benchmark does, and it is specific to your task, your acceptance bar, and who does the checking.
Run it per task, on your own data, with your own acceptance test. Public leaderboards cannot answer it because they do not know your acceptance bar and they measure on data that is not yours. This is the same discipline covered in when a small model beats a large one, and routing is simply the case where you decided both answers are right for different slices of traffic.
How often each precondition holds, in the codebases we are handed
Our impression from review work, not a survey. The last row is the one that matters: most teams propose a router before doing the free version.
Cascades beat classifiers more often than teams expect
A predictive router has to answer "is this request hard?" from the request alone. A cascade answers "is this answer good?" with the answer in front of it. The second question is easier, and the difference in signal quality is not small.
The useful escalation signals are cheap and mostly not model-based. A schema validator that rejects malformed structured output. A grounding check that confirms every cited span exists in the retrieved documents. A numeric sanity check on totals. A short second call asking a small model whether the answer addressed the question, which is weak on its own but useful in combination. Self-consistency across two cheap samples, where disagreement is a strong escalation trigger and agreement is only moderate evidence of correctness.
Where a cascade hurts is latency. An escalated request pays the cheap call, the check, and then the expensive call, in series. If your cheap attempt takes eight hundred milliseconds and your escalation rate is a third, your p95 has moved meaningfully even though your p50 improved. For anything interactive, budget that explicitly — see latency budgets for conversational AI for how to write the budget down before you spend it.
The mitigation that works is speculative parallelism on the slice you already know is hard: fire both calls at once, return the cheap answer if it passes, discard the expensive one if it did not turn out to be needed. You pay tokens to buy latency. That is a fine trade on twenty percent of traffic and a terrible one on all of it.
If you do build a predictive router, treat it as a model
It has an accuracy, it has an operating point, it drifts, and it needs an eval set of its own. Almost none of the routers we are handed have any of these.
Its errors are asymmetric and you must choose which one to make. Routing a hard request down produces a bad answer that reaches a customer. Routing an easy request up costs a fraction of a cent. Those are not remotely equal, so the threshold should not sit where accuracy is maximized. It should sit where the expected cost of a bad answer — support time, rework, churn, whatever you honestly believe it is — equals the marginal token cost. Write that number down even if it is a guess, because the alternative is that the threshold gets set by whoever last looked at a confusion matrix.
Router drift is the quiet one. The router was fit on last quarter's traffic. Your product shipped a feature that changed what people ask. The router keeps making confident decisions about a distribution that no longer exists, and nothing in your dashboards is red. Monitor the routing mix as a first-class metric: if the share going to the small model moves five points in a week and you did not ship anything, something changed and it was not the router.
What breaks once one feature depends on three models
The savings are visible on a dashboard. The costs are spread across six places, and they are the reason routing projects come in over estimate.
Prompts are not portable. A prompt tuned over three months against one model is a fitted artifact. Moved to another model it will usually work and it will not work identically — different verbosity, different refusal behavior, different willingness to guess at a missing field, different adherence to a format instruction. Every route needs its own prompt, its own eval, and its own version history. That doubles or triples the surface described in prompt versioning and rollback, and the bundle hash has to include the model or a rollback rolls back nothing.
Tool-calling formats differ. If your product exposes tools, expect differences in argument coercion, in behavior when a required parameter is missing, in how many calls a model is willing to chain, and in what happens on a malformed tool result. An adapter layer hides the syntax and cannot hide the behavior.
Structured output guarantees differ. One provider will hard-constrain generation to a schema, another will try. If your pipeline assumes valid JSON because it always was, the fallback path is untested code the first time it runs.
Caching fragments. Prefix caching is per model and per provider. Splitting traffic three ways splits your cache hit rate three ways, and on long system prompts that can eat a real share of the routing saving before you see it. The interaction is covered in caching for LLM applications, and it is the term most often missing from the business case.
Rate limits do not add up. Three providers with separate quotas means three separate ways to be throttled and three different retry behaviors, and your effective ceiling is set by the route you cannot shed. Model the limits per route, not in aggregate.
Attribution gets harder. Unless every response records which model served it, at which snapshot, under which prompt version, you cannot answer "did quality drop?" — and that question will be asked, usually by a customer, usually about last Tuesday.
Ship the kill switch before the router
One configuration value that forces all traffic to a single named model, deployable without a code change, tested in staging before the router ever sees production traffic. When quality complaints start and you have three models, an eval suite and a classifier in the path, the first thing you need is to remove three of those variables in under a minute. Teams build this after their first bad week. Build it in the first week instead.
Send us the numbers and we will tell you if it pays.
Monthly request volume per task, average input and output token counts, current model, and what your acceptance check is. Email contact@precisionfederal.com. You get back the break-even arithmetic on your own numbers and an honest answer about whether static assignment already captures most of it. One business day, no charge, no meeting.
contact@precisionfederal.comRouting for reasons other than money
Cost is the common motive and not the only good one.
Latency. A small model that answers in six hundred milliseconds may be the right choice for an autocomplete or an inline suggestion regardless of price, because a two-second suggestion is not a suggestion.
Context length. Routing by input size is the most reliable predictive signal that exists, because it is measured rather than predicted. Anything over the small model's window goes up, full stop.
Capability gates. Requests needing images, or a specific tool behavior, or a long chain of reasoning, route by capability. This is closer to static assignment than to routing, which is a point in its favor.
Data placement. Some inputs are not permitted to leave a particular environment. That is a policy decision that happens to look like routing, and it should be enforced before the router runs rather than inside it, so that a router bug cannot become a data incident.
A three-week way to find out
Routing Evaluation Sprint
Step four is the one worth defending against schedule pressure. In our review work, static assignment plus a single well-chosen cascade captures most of what a full predictive router would have delivered, at a fraction of the operational cost, and it can be explained to a new engineer in one sentence. If step four closes the gap, stop there and say so.
The mistakes we are called in to fix
- A router built before anyone measured spend per task, optimizing the second-largest line item
- One prompt reused across every route, so the small model is being judged on a prompt fitted to the large one
- No record of which model served a response, making every quality question unanswerable
- Escalation on model self-doubt alone, which correlates with verbosity more than with error
- Failover that silently downgrades quality during a provider incident, with no alarm on the mix
- A routing threshold set to maximize accuracy, ignoring that the two error types cost wildly different amounts
- Caching benefits counted in the business case that the traffic split then destroys
- No kill switch, so the first incident is debugged with all variables live
Before you ship a router
- Spend is broken down per task and the top two are named
- Cost per accepted answer is measured per model, on your own data
- Static assignment has been tried and its saving is recorded
- Every route has its own prompt, its own eval set and its own version
- The escalation signal checks the answer, not the model’s confidence in itself
- Every response records model, snapshot, prompt hash and route reason
- Routing mix is a monitored metric with an alert on sudden movement
- The p95 with escalations is inside the latency budget you published
- A one-value kill switch forces a single model and has been tested
- The business case is rechecked when provider prices move
What belongs on the dashboard
Four series, and none of them is the bill. Routing mix — share of requests per route, per day, which is your earliest signal that the input distribution moved. Acceptance rate per route, not in aggregate, because aggregate quality stays flat while one slice rots and the mix shifts underneath it. Escalation rate for any cascade, with the reason attached, since a rising rate usually means the cheap model got a harder slice rather than that it got worse. Cost per accepted answer per route, which is the only number that can tell you the router is still earning its keep.
Add one alarm that most teams do not have: fire when the routing mix moves more than a few points day over day with no deployment. That single alert has caught a provider quietly changing a model behind an unpinned alias, a retrieval index rebuild that lengthened every prompt past a size threshold, and a product change that altered what users typed. None of those would have shown up in a cost graph until the end of the month.
Common objections
Our provider offers routing as a feature. Why build anything?
Managed routing is a reasonable starting point and it optimizes for a generic objective, because it cannot know your acceptance test. It also gives you less visibility into why a given request went where it did, which is exactly what you need during a quality incident. Use it to get a cheap estimate of the available saving, then decide whether owning the decision is worth the weeks. If you keep it, still record the served model on every response.
Can we just let the large model decide when to hand off?
Then you have already paid for the expensive call, which is the cost you were trying to avoid. Self-assessment is also a weak signal: a model asked whether a task is hard tends to answer based on surface features of the prompt, and stated confidence correlates with fluency more reliably than with correctness. Check the answer instead of asking the model about the question.
Does fine-tuning a small model remove the need to route?
Sometimes, and it moves the problem rather than removing it. A tuned small model can lift the share of traffic it handles well, which widens the band where routing pays instead of closing it. It also adds a training pipeline, a data-collection loop and a second thing to version. The comparison worth running is tuned-small-only against route-between-two, on the same acceptance test, before committing to either.
How much saving is realistic?
For a workload with genuinely mixed difficulty and a wide price gap, cutting model spend by a third to a half is a defensible target and we have seen more. For homogeneous traffic where every request needs the same capability, the honest answer is close to nothing, and the useful outcome of the evaluation is finding that out in three weeks instead of three months.
Bottom line
Routing is a volume play. At high volume with a wide price gap and a task whose difficulty genuinely varies, it returns a lot of money for a few weeks of work and it is worth the operational weight. At low volume it is a tax on your engineering team disguised as a saving. Between the two, do the cheap version first: assign models per task in config, add one cascade where an answer can be checked automatically, and measure what that captured before anyone writes a classifier. Most of the time that is the whole project, and the honest version of this advice is that the best routing work we do ends with fewer moving parts than the team expected to build.
Frequently asked questions
As a rough line, below about five thousand dollars a month on a single task the engineering and maintenance cost will exceed the saving. Above about fifty thousand it is usually worth at least measuring. In between it depends on the price gap between your candidate models and how mixed your traffic actually is, so run the arithmetic with your own volumes rather than trusting the range.
Usually, because the decision is made with a real answer available rather than predicted from the prompt. Schema validity, grounding checks and numeric sanity checks are far stronger signals than difficulty estimation. The cost is latency on escalated requests, which matters for interactive features and not much for batch work.
You can, and you will be measuring the wrong thing. A prompt tuned against one model is fitted to it. Each route needs its own prompt version, its own eval set and its own history, and the version identifier recorded on a response has to include the model or a rollback will not restore the behavior you had.
Record model, snapshot and route reason on every response, then track the acceptance rate per route rather than in aggregate. Aggregate quality can look flat while the small-model slice degrades, because the mix moves at the same time. Alert on the routing mix itself, since a five-point shift with no deploy means the input distribution changed.
Yes, and it is the term most often left out of the business case. Prefix caches are per model and per provider, so splitting traffic splits your hit rate. On a long system prompt the lost cache discount can consume a meaningful part of the routing saving, so estimate the post-split hit rate before committing.