The question is four questions wearing one coat
“Should we run our own inference?” sounds like a build-versus-buy question with two answers. It is four independent choices, and teams get into trouble by deciding all four at once because they arrived bundled in one vendor's pitch. Separate them and most of the anxiety goes away: three usually have easy answers and only one is genuinely hard.
The model. A hosted frontier model behind someone else's API, or an open-weight model you can download, inspect, pin and modify. This is a capability and cost question, and it is answered by your evaluation set, not by a leaderboard.
The serving runtime. The thing that loads weights, batches requests, manages the key-value cache, and turns tokens out at some throughput and some tail latency. This is the layer people mean when they say “build our own stack,” and it is the one that comes with a pager.
The hardware and capacity. Rented by the second, reserved for a year, bought outright, or a blend. A finance question dressed as an engineering one, decided by how predictable your load is rather than how fast the accelerator is.
The gateway. The code of yours that every model call passes through on its way out: routing, retries, timeouts, budget caps, per-tenant accounting, structured logging, version pinning, a kill switch. Almost nobody frames this as part of the stack. It is the part you should build first and the part that makes every later decision reversible.
Once the four are separate, the conversation changes. You can run an open-weight model on a managed endpoint and own no hardware. You can buy dedicated capacity and own no runtime. You can own a runtime on rented accelerators and give it back in a month. The all-or-nothing framing is a framing problem, not a technical constraint.
You are probably here because
- Inference is now a visible line in your cost of goods and your board asked about it
- A customer told you their data cannot leave an environment they control
- Your p95 latency is fine and your p99 is embarrassing you
- Somebody built a spreadsheet showing self-hosting is cheaper and you cannot find the flaw in it
The section on cost per successful task is where to start. The section on what a GPU-hour comparison omits is where that spreadsheet usually breaks.
Start with a number you can already compute
Before any of this is decidable you need one figure, and it is not price per million tokens. It is cost per successful task: the total inference spend over some window, divided by the number of user-visible outcomes that were actually good enough to keep.
The denominator is the interesting part. It excludes the generation that failed schema validation and was retried, the three attempts your agent loop burned before taking the right tool, and the answer a reviewer threw away. Teams routinely find a meaningful share of token spend produces nothing anyone consumed — invisible in a per-token comparison, because both sides have it.
Computing it takes an afternoon if your gateway already logs a request identifier, token count, model version and outcome. If it does not, that is the first project and a small one. You do not need an outside firm for it, and doing it first will change what you decide.
Two more measurements belong beside it. Real concurrency — the distribution of simultaneous in-flight requests across a normal week, Monday spike and overnight trough included. And the latency your product needs, expressed as a percentile against a deadline rather than an average. A stack that is fast at median and falls apart at p99 will be described by your users as slow and by your dashboard as fine.
The five conditions that actually justify owning the serving layer
Owning the runtime is worth it when at least one of these is true and you can say which. If none of them is true, the honest answer is that you are optimising the wrong thing, and the cheaper win is in your prompts, your context size, your caching and your model routing.
1. Sustained, predictable, high volume. Not peak volume, sustained volume. Accelerators bill for wall-clock time whether or not anyone is asking questions. Self-hosting is a fixed-cost structure competing against a variable-cost one, and fixed costs only win at high utilisation. If your traffic has a ten-to-one daily swing and no batch workload to fill the trough, you will pay for a great deal of idle silicon.
2. A data boundary a hosted API cannot satisfy. Some customers require that their content never leaves an environment they control. A vendor's zero-retention setting does not always answer it, because the objection is sometimes to the network path rather than the retention. Note that this is a deployment location requirement, not necessarily a build your own runtime requirement — several products will run their serving stack inside your cloud account or your customer's.
3. A latency floor you cannot buy. If your product has a hard interactive budget, colocating the model with your data removes network hops and queueing you do not control. The effect is smaller than people expect for long generations, where token generation dominates, and larger for short ones, where overhead does.
4. A model nobody hosts. You fine-tuned it, you merged adapters, you quantised it a particular way, you patched the sampling. If the artefact you depend on does not exist on anyone's managed endpoint, the decision has been made for you. Be honest about whether the customisation is actually earning its keep first — a fine-tune that is not measurably beating a prompt on your eval set is a liability with a hosting bill attached.
5. Version stability you can enforce. Hosted models are deprecated and retired on the provider's schedule. If you have qualified a specific version against a regulated workflow or a contractual accuracy commitment, pinning weights on disk has genuine value — weighed against the cost of being the only team still running that version, unpatched flaws included.
What a GPU-hour comparison leaves out
The spreadsheet that shows self-hosting winning is usually correct about the accelerator price and wrong about everything around it. In rough order of how often it is the thing that flips the answer:
Utilisation. The comparison is almost always run at an assumed utilisation nobody measured. Take the number the plan assumes, then look at your traffic distribution and ask what fraction of the hours in a week would have work in flight. A plan that assumed high utilisation and delivered much less is the most common way these projects disappoint.
Redundancy. One node is a demo. Production means at least two, in two failure domains, each able to carry the load — not a small adjustment to the denominator.
Cold start. Scaling a hosted API is somebody else's problem. Scaling your own means acquiring an accelerator, pulling a large image, loading weights and warming caches. Measure it on your own hardware — minutes is a different autoscaling architecture than seconds.
The pager. A serving stack is a production system with driver versions, memory fragmentation, cache eviction behaviour and its own failure modes. Somebody is on call for it, and that person's time is the largest line item that never appears in the comparison.
Version churn. Staying current with a fast-moving serving runtime is a recurring cost. Not staying current is a slower one that arrives as a patch you cannot apply.
Evaluation. When you own the stack you own the question of whether a quantisation or a runtime upgrade quietly degraded output quality. Without an eval set you run, you will not know; with one, you have a second system to maintain.
| Layer | Own it when | Leave it alone when |
|---|---|---|
| The gateway | Nearly always. It is days of work and it makes every other decision reversible. | You have one call site, one model, and no tenants. |
| Model choice | You have an eval set and a measured reason to prefer an open-weight model. | You are choosing by benchmark table rather than by your own task. |
| Serving runtime | Sustained high utilisation, a hard data boundary, a model nobody hosts, or a latency floor you cannot buy. | Traffic is spiky, small, or still changing shape monthly. |
| Hardware and capacity | Load is predictable a quarter ahead and you can fill the troughs with batch work. | You cannot forecast next month within a factor of two. |
| Fine-tuning | A tuned model beats a prompted one on your own held-out set, repeatedly. | You have not tried a better prompt, better retrieval, or a bigger model first. |
The gateway is the piece almost everyone should own
If you take one thing from this article, take this. The highest-return infrastructure work in most AI products is not the serving layer. It is the thin layer of your own code that every model call passes through.
A gateway worth having does eight things. It routes, so a request can go to a different model without a deploy. It retries with sane backoff and a budget. It times out at a deadline your product chose rather than one a client library defaulted to. It caps spend per tenant and per feature, so a loop cannot cost a month of margin overnight. It logs request identifier, model version, token counts, latency and outcome in a queryable structure. It pins versions, so a provider's update is a change you make rather than one that happens to you. It supports shadow traffic. And it has a kill switch per feature.
None of it is difficult and all of it is boring. Its value is that it turns the self-hosting question from an architectural commitment into an experiment: with a gateway, pointing a tenth of production traffic at your own runtime for two weeks is a configuration change. Without one, it is a quarter.
How often owning this layer pays off — our read
Our judgment across the products we have worked on, not a survey. Read it as ordering, not as probability.
The middle of the range is where most teams should be
The choice is presented as a binary far more often than it is one. Between a shared public endpoint and a rack you own, there are at least four intermediate positions, and each removes a specific objection without taking the pager.
Dedicated or provisioned capacity from a hosted provider. You reserve throughput rather than paying per token. Cost becomes predictable, tail latency improves because you are not sharing a queue, and nobody new carries a pager. The trade is that you now pay for capacity whether you use it — the same economics as self-hosting with none of the engineering.
Managed endpoints for open-weight models. Somebody else runs the serving runtime; you choose the weights. That resolves model availability and version pinning without the runtime burden, and it is a good place to sit while you learn whether your traffic ever becomes predictable enough to go further.
Single-tenant deployment into your own account. Several vendors will run their stack inside infrastructure you control. This is often the right answer to condition two — the customer data boundary — and it is dramatically less work than building the same thing.
A split by workload. Interactive traffic stays on a hosted API where elasticity is worth paying for. Bulk work — overnight enrichment, backfills, document processing, evaluation runs — goes to accelerators you control at high utilisation, where the economics are strongest and nobody notices a restart. This is the most under-used option on the list and often captures most of the available saving at a fraction of the risk.
A two-week test that answers it with your numbers
If you have a gateway, the question is empirical and cheap. Run it as follows.
Days one to three. Freeze an evaluation set of a few hundred real tasks with known-good outputs, drawn from production and not written by hand. Score your current stack on it. This is now your baseline; nothing that follows means anything without it.
Days four to seven. Stand up one candidate open-weight model on a rented accelerator with an off-the-shelf serving runtime. Do not tune anything yet. Run the eval set. If quality is not close, stop — you have your answer, and it cost you four days.
Days eight to eleven. Load-test at your measured concurrency, not a number that flatters the hardware. Record throughput, p50, p95, p99 and what happens past capacity. Measure cold start from scale-up to first token.
Days twelve to fourteen. Mirror a slice of production traffic through the gateway while users are still served by the incumbent, and compare on live inputs. Then price it at the utilisation your traffic implies, with redundancy and the engineer who will carry it.
At the end you have a defensible answer for your system rather than a general one. The most common outcome is not “yes” or “no” but “yes for batch, no for interactive.”
The mistakes we see
- Comparing per-token prices instead of cost per successful task, so retries and rejects cancel out on paper and not in the bill
- Assuming a utilisation nobody measured, which is the single most common flaw in a self-hosting business case
- Sizing for one node, then discovering redundancy after the first failure
- Skipping the eval set, then having no way to tell whether a quantisation cost you accuracy
- Building the runtime before the gateway, which makes the decision expensive to reverse
- Fine-tuning before trying a better prompt or better retrieval, and inheriting a hosting problem to serve it
- Treating a customer's data-boundary requirement as a build requirement when a single-tenant deployment would have satisfied it
- Ignoring the batch split, which is often most of the saving with almost none of the risk
Where you do not need us
Plainly: if you are pre-product-market-fit, or your monthly inference spend is smaller than a fortnight of engineering time, this is not your problem yet. Spend the attention on the product.
If what you need is the gateway, a competent backend engineer on your team can build the useful version in under two weeks, and will do it better than an outsider because they know your call sites. The same goes for the eval set: it belongs to people who know what a good output looks like in your domain, and that is you.
Where an outside team earns its fee is narrower: running the two-week test as a controlled experiment while your engineers keep shipping, taking a serving runtime from working to on-call-ready, or standing up a single-tenant deployment into a customer environment when that is blocking a contract. Working out which of the four layers is actually your problem is usually a conversation, not an engagement.
Before you commit
- You can state your cost per successful task from your own logs
- You know your real concurrency distribution across a week, not your peak
- You have a latency requirement expressed as a percentile against a deadline
- An eval set of real tasks exists and your current stack has a score on it
- You can name which of the five conditions applies to you
- The business case uses a utilisation figure derived from your traffic, with redundancy included
- Cold start has been measured on the hardware you would actually use
- A named person is on call for the runtime, and their time is in the model
- You have priced dedicated capacity and single-tenant deployment as alternatives
- You have checked whether a batch-only split captures most of the benefit
Bottom line
Own the gateway now, because it is cheap and it makes everything after it reversible. Own the eval set, because without it you cannot tell whether any change helped. Own the serving runtime only when you can name the condition that requires it — sustained utilisation, a data boundary, a latency floor, an unhosted model, or version stability — and only after pricing the middle options, because dedicated capacity or a single-tenant deployment resolves most of these objections without handing anyone a pager. Owning accelerators outright is a financing decision a small number of companies with very steady load should make and most should not. Two weeks of measurement will settle an argument that has otherwise been running for a quarter.
Frequently asked questions
No threshold transfers between companies; anyone quoting one is quoting their own traffic shape. What matters is utilisation, not spend — a modest bill with flat round-the-clock load beats a large bill concentrated into four hours a day. Compute the utilisation your traffic implies, then compare fixed-cost capacity against variable-cost tokens at that number.
Usually not. That is a deployment-location requirement, and several vendors will run a single-tenant deployment inside your account or your customer's. Read the requirement closely first: sometimes the objection is to retention, which a provider setting resolves, and sometimes it is to the network path, which it does not. The two get conflated constantly.
Only your evaluation set can answer that, and it differs sharply by task. Extraction, classification and routing often transfer well; long multi-step reasoning and tool use are where the gap tends to be widest. Build the set from real production tasks and treat published benchmark tables as a way to shortlist candidates, not as evidence about your workload.
Usually the autoscaling assumption. Everything is fine until traffic exceeds the provisioned nodes; scale-up then takes minutes because weights must load, requests queue behind a cold node, and tail latency collapses. Second most common is a silent quality regression after a runtime upgrade or a quantisation change, visible only if you run an eval set on every deploy.
After, and only if a tuned model beats a well-prompted one on your own held-out set. Fine-tuning is often a way of paying hosting costs to fix a problem that better retrieval or a clearer output schema would have solved for free. If the tuned model does win, you have made the hosting decision for yourself — so know the operating cost before committing to the artefact.