Running inference without a GPU budget

The constraint usually arrives already decided. The accelerated instance family is not in your region, or it is in your region but not inside the accredited boundary, or the quota request has been open for six weeks, or the capital budget closed in March. Nobody is asking whether you would like a GPU. They are asking whether the thing still works without one. The honest answer depends almost entirely on which half of the inference workload dominates your latency budget, and most teams have never separated the two halves.

The one calculation that decides this

Transformer inference has two phases with opposite hardware appetites, and conflating them is the single most common reason a capacity plan is wrong.

Prefill processes the whole prompt at once. Every token in the prompt goes through the network in parallel, so the hardware gets to do many arithmetic operations for each byte of model weight it reads. Arithmetic intensity is high and roughly scales with prompt length. Prefill is compute-bound. This is the phase where a GPU's raw FLOP advantage is enormous and a CPU has nothing to offer.

Decode generates output one token at a time. Producing a single token at batch size one requires reading every weight in the model once and doing about two floating-point operations per weight. Arithmetic intensity is close to 1. Decode is memory-bandwidth-bound. The processor's FLOP rating barely matters. What matters is how fast it can stream the model out of memory.

That gives you a hard ceiling you can compute on a napkin. Take the size of your quantized weights in gigabytes. Divide the device's memory bandwidth by it. That is the maximum tokens per second any implementation can produce at batch size one, before any inefficiency. Take Llama 3.1 8B as a worked example. In FP16 the weights are about 16 GB. A 4-bit K-quant of that model measures roughly 69% smaller in a published evaluation of llama.cpp quantization levels, which puts it near 4.9 GB. Divide 4.9 into the bandwidth numbers on the spec sheets:

These are ceilings, not measurements. No real stack hits them. The important caveat is that the fractions differ: GPU kernels get much closer to peak bandwidth than CPU kernels do, so a CPU's practical result sits further below its own ceiling than a GPU's does. Measure on your own hardware before you commit to a number in a contract. But the ordering in that table is real, and it is not the ordering most people assume.

A server CPU is closer to a small GPU than the marketing suggests

Look at the bottom two rows. A modern twelve-channel server socket has close to twice the memory bandwidth of an NVIDIA L4. The L4 is not an exotic part. It is the GPU inside the G6 instance family, and G6 is one of the accelerated families available in AWS GovCloud. For single-stream token generation, the constraint that binds is bandwidth, and on that axis the CPU you already have in your general-purpose fleet is competitive with the entry-level datacenter GPU you are waiting on.

This does not mean the CPU wins. The L4 has 240 Tensor Cores and 72 watts of power draw, and it will annihilate the CPU on prefill and on any workload with real batching. It means the specific claim "we cannot generate tokens at an acceptable rate without a GPU" is frequently false at low concurrency, and it is testable in an afternoon.

The instruction sets closed part of the remaining gap. Intel's AVX-512 VNNI, shipped as part of Deep Learning Boost starting with the second-generation Xeon Scalable parts, fused the INT8 multiply-accumulate sequence that dominates quantized inference. Advanced Matrix Extensions went further, adding a tile-based matrix unit with BF16 and INT8 support in the fourth-generation Xeon and FP16 in the sixth generation. Intel states it remains the only vendor submitting stand-alone server-CPU results to MLPerf Inference, which is worth reading for what it implies about the rest of the market as much as for the scores themselves.

Where "no GPU budget" actually comes from

It is worth naming the real causes, because they determine whether the constraint is temporary.

Region lag. Accelerated instance families reach government and sovereign regions well after commercial ones. G6 became generally available commercially in April 2024 and arrived in AWS GovCloud (US-West) in November 2024. Seven months is a long time to hold a program schedule open. If a plan depends on a family that has not landed in your region yet, it is not a plan.

Accreditation boundary. An accredited baseline is a specific image with specific drivers. Adding a GPU means adding a kernel module, a driver stack, and a container runtime hook that were not in the assessed configuration. That is a change request against an authorization, and the schedule for it belongs to a security office rather than to engineering.

Shared pool politics. In many enterprises the accelerators exist and belong to a training team. Inference gets whatever is idle, which is nothing during a training run. A capacity plan that assumes access to a contended pool is a capacity plan with an unpriced dependency.

Two of those three are temporary and one of them is not. Design for the constraint you will still have in twelve months. Our piece on GPU capacity planning covers the other side of this decision, when the allocation does exist and the question becomes how much to reserve.

The four levers, in the order that pays

Teams reach for these in the wrong order almost every time. They start with kernel tuning, which is the third lever, and never revisit the first, which is worth more than the other three combined.

• Right-size the model — The largest available model is rarely the smallest model that passes your evaluation. Distillation has been a known quantity for years: DistilBERT retained 97% of BERT's language understanding at 40% smaller and 60% faster. For classification, extraction, ranking and routing, a purpose-trained encoder under a billion parameters usually beats a general-purpose generative model on both accuracy and latency for the specific task.
• Quantize — Weight bytes are the denominator in the ceiling calculation, so cutting them cuts latency directly. INT8 dynamic quantization through ONNX Runtime reports roughly 3x to 4x on encoder models with negligible accuracy movement. On the generative side, a published evaluation of llama.cpp quantization on Llama 3.1 8B measured a 4-bit K-quant scoring 69.15 against an FP16 baseline of 69.47 on an averaged benchmark suite, with WikiText-2 perplexity moving from 7.32 to 7.56, for a 69% reduction in size.
• Compile to the hardware you have — A PyTorch eager-mode model is a prototype, not a deployment. ONNX Runtime, OpenVINO and llama.cpp all reach the matrix and vector units that generic kernels leave idle. OpenVINO 2026 added a unified runtime scheduler that lets you tag pipeline stages by preferred device, which matters when the box has a CPU and an NPU and you want the transformer layers on one and the pre-processing on the other. vLLM's CPU backend covers x86, ARM and PowerPC, and carries an experimental small-batch kernel path that requires AMX with BF16 weights.
• Batch and cache — Batching raises arithmetic intensity, because one read of the weights serves many sequences. On CPU the benefit tops out early, since the FLOPs run out long before a GPU's would, but going from batch 1 to batch 4 is often close to free. Caching is the cheaper cousin: an exact-match or embedding-similarity cache in front of the model removes calls entirely, and removed calls have no latency at all.

One more lever applies only to generation, and it is genuinely free of quality cost. Speculative decoding runs a small draft model ahead of the target model and verifies its proposals in a single target forward pass. Because the verification step preserves the target model's output distribution, the result is mathematically identical to ordinary decoding. Reported speedups cluster in the 2x to 3x range when the draft model matches the content well enough to get a high acceptance rate. On a bandwidth-bound CPU this is a direct multiplier on the number in the ceiling table.

The prefill trap

Here is where CPU deployments die, and it is almost never the part teams test.

A forward pass costs roughly two floating-point operations per parameter per token. Prefill processes the whole prompt, so a 4,000-token prompt against an 8B model is about 64 trillion operations before the first output token appears. That number scales linearly with prompt length and is unaffected by quantizing weights for bandwidth. Decode does not care how long the prompt was. Prefill cares about nothing else.

Which means retrieval-augmented generation is the workload most likely to break a CPU plan. A chat request with a 200-token prompt is comfortable. The same architecture with eight retrieved passages stuffed into a 6,000-token context is a different machine entirely, and the time-to-first-token can move by an order of magnitude while tokens-per-second after that first token barely changes. If your latency requirement is expressed as time-to-first-token, prefill is your whole problem and the bandwidth table above is irrelevant to you.

The KV cache is the other prompt-length cost. For a model with 32 layers and 8 key-value heads at a head dimension of 128, each token of context holds 2 × 32 × 8 × 128 values for keys and values, which is 128 KiB per token in FP16. An 8,000-token context is about 1 GB of cache per concurrent session. On a GPU that competes with the weights for HBM. On a CPU it competes for the same DRAM bandwidth you were counting on for decode. Concurrency in a long-context system is bounded by cache growth long before it is bounded by cores.

What survives, workload by workload

Speech deserves a note because it is the class people misjudge most. CTranslate2-based Whisper implementations get roughly 4x over the reference PyTorch path on CPU with INT8, and community benchmarks put the large model near a real-time factor of 2.5 on a strong desktop CPU, meaning one second of audio takes about two and a half seconds to transcribe. Smaller model sizes fall well under real time. So the honest answer is not "yes" or "no" but "which model size, and does your audio arrive faster than you can drain it."

The accelerator you are not calling a GPU

When the workload genuinely does not fit on CPU, the next question is whether the blocked resource is accelerators in general or GPUs specifically. Those are often different procurement objects.

Inference silicon. AWS Inferentia2 carries 32 GB of HBM per chip at 820 GB/s, with instances scaling to twelve chips for 384 GB and 9.8 TB/s aggregate. Inf2 appears in the GovCloud accelerated-instance list. It is not a drop-in for CUDA code and the Neuron compiler has real model-coverage limits, so treat porting as scoped engineering work rather than a configuration change. For inference-only serving of a supported architecture, the economics are frequently better than the GPU it replaces.

NPUs on client and edge hardware. If the inference is happening on a workstation, a kiosk, or a fielded device, the neural accelerator is already in the silicon you bought. OpenVINO's 2026 release added ahead-of-time and on-device compilation for Intel NPUs without depending on OEM driver updates, which removes one of the more annoying fleet-management obstacles.

A managed endpoint inside the boundary. Sometimes the correct move is to stop hosting weights. Amazon Bedrock holds FedRAMP High authorization in AWS GovCloud, and additional model families were approved at FedRAMP High and DoD Impact Level 4 and 5 in GovCloud in June 2026. Azure OpenAI Service is authorized within FedRAMP High for Azure Government and at DoD IL4 and IL5. If the data classification permits it, a managed endpoint converts a capacity problem into a per-token cost line, which is a much easier conversation with a program office than a hardware requisition. It also creates a network dependency, which is exactly the wrong answer for disconnected work. What actually runs offline is a separate question with a separate answer.

Budget the latency, not the model

A requirement stated as "under two seconds" is not yet an engineering requirement. Three things have to be attached to it before it means anything.

Which percentile. A p50 target and a p99 target are different systems. Queueing behavior, not model speed, sets the tail. A service that is comfortably fast at p50 and unacceptable at p99 is usually suffering from arrival-rate variance against a small number of workers, and the fix is admission control and a queue you can observe, not a faster model.

Which segment of the path. Instrument the whole request before optimizing the model. Authentication, retrieval, the vector search, network hops, serialization, guardrail passes and logging routinely account for more of the wall clock than the forward pass does. We have seen more latency recovered from a retrieval step than from any change to the model. Optimizing an inference call that is 20% of the budget caps your possible improvement at 20%.

At what concurrency. Latency without a load figure is a benchmark, not a requirement. Little's Law is enough: the number of requests in the system equals the arrival rate times the time each spends in it. If ten requests per second each occupy the system for one second, you need ten concurrent workers to avoid a growing queue, and each of those workers needs its own KV cache. That is the calculation that turns "one GB of cache per session" into a memory sizing.

Where CPU inference genuinely does not work

Being useful here requires saying the parts that do not go your way.

Training and fine-tuning are compute-bound in the same way prefill is, and there is no clever quantization that fixes it. If the project needs adaptation of a model of any real size, it needs accelerators for that phase, even if serving afterward does not. Splitting the two phases across different hardware and different timelines is often the cleanest way to unblock the schedule.

High-concurrency generation is out. The economics invert quickly: once you are batching many sequences, the GPU amortizes each weight read across the whole batch while the CPU runs out of arithmetic, and the cost per million tokens diverges hard. Large diffusion and video models are out for interactive use for the same reason. And any workload where time-to-first-token is the contract term and prompts run to thousands of tokens is a prefill problem that CPU quantization does not touch.

The failure mode we would most warn against is building a CPU deployment that works at pilot concurrency and discovering the wall at production concurrency. Load-test at the real arrival rate with the real prompt length distribution before the design is locked. The number that matters is not tokens per second on a warm single stream. It is the p95 at the concurrency the program will actually see.

Writing it into the contract

If someone else is building this for you, the acceptance criteria are where the risk lives. Vague performance language in a statement of work becomes a dispute at delivery. Specific language becomes a test.

• Name the hardware. Instance family and size, or CPU model and memory configuration. "Commodity hardware" is not a specification.
• Name the percentile and the concurrency. "p95 under 1,200 ms at 20 concurrent sessions" is testable. "Responsive" is not.
• Separate time-to-first-token from tokens-per-second. They are governed by different physics and a single latency number hides which one failed.
• Fix the input distribution. Prompt length percentiles, audio duration, document page counts. Performance claims are meaningless without the input profile they were measured on.
• State the quality floor alongside the speed floor. Quantization trades accuracy for latency. Bind both ends or the vendor will optimize the one you wrote down.
• Require the benchmark harness as a deliverable. If you cannot re-run the measurement after handoff, you did not buy a measurement.

Common objections

Bottom line

The GPU question is answered by workload shape, not by preference. Compute the batch-1 bandwidth ceiling, measure your prefill cost at a realistic prompt length, and check where your latency budget actually goes before you spend anything on hardware. Classical ML, encoder models and embeddings almost never needed an accelerator. Short-prompt generation with a right-sized, quantized, compiled model is frequently viable and frequently faster to field than waiting for capacity. Long-context retrieval, high concurrency and large models are not, and pretending otherwise produces a pilot that passes and a production system that does not. Knowing which of those you have is a day of measurement, and it is the cheapest day in the project.

Frequently asked questions