Requests per second is the wrong unit
A rate limiter on a CRUD API counts requests, and that works because every request costs about the same. On an inference API the assumption is wrong by three orders of magnitude. One call sends forty tokens and streams back a sentence. The next sends a hundred thousand tokens of context and asks for four thousand tokens of summary. Both are one request. A limiter that counts requests will let the second client hold the whole fleet while staying inside its quota, and will reject a hundred cheap clients whose combined cost is smaller than that one call. Nearly every incident we get pulled into on this subject comes back to counting the wrong thing.
The reason is in the shape of the work. A transformer request runs in two phases with different cost curves. Prefill reads the whole prompt and builds the key/value cache for it: compute-bound, parallel across the sequence, and growing faster than linearly with input length because attention work grows with the square of the sequence while the rest of the network grows linearly. Decode then emits one token at a time: memory-bandwidth-bound, no parallelism within a request, cost close to linear in output length. A request is not one number. It is two, and they push against different hardware limits.
So a batch of short chat turns and a batch of long-document summaries produce identical request counts, identical error rates, and completely different queue depths. The dashboard stays calm while p99 latency triples, and requests per minute is not a knob that fixes it.
You are probably here because
- The requests-per-minute dashboard is flat and p99 latency has tripled anyway.
- One customer's key is holding most of the fleet and is comfortably inside its quota.
- The bill went up an order of magnitude while request counts barely moved.
- Nobody can say where the number in the limiter config came from.
The next two sections cover what to count instead and the cache arithmetic that produces the number, because all four of these usually come from one root cause: a limiter counting requests on an endpoint where two requests can differ in cost by a factor of a thousand.
Three limits, and they are not substitutes for each other
Three quantities are worth limiting on an inference API. Systems that stay up limit all three, in different places, with different failure behavior.
Concurrency. How many requests from a tenant may be in flight at once. This is the limit that protects latency for everyone else. It is a semaphore rather than a counter, and it is the only one of the three that returns capacity the moment a request finishes.
Token throughput. Input and output tokens per minute, tracked per key and per organization. This is the fairness limit. It smooths a tenant's consumption across a window so one client cannot spend its hourly allowance in eight seconds and then complain about queueing.
Spend quota. Cumulative cost over a billing period, with a hard ceiling. This protects the bill, and most teams add it only after an unpleasant invoice. Its semantics differ from the other two: it does not refill on a timer, it is monotonic within a period, and hitting it is a business event rather than a traffic event.
They are not interchangeable. A generous token limit with no concurrency cap lets a client open two thousand streams and destroy tail latency without exceeding a quota. A tight concurrency cap with no spend ceiling lets a well-behaved integration run a long-context job around the clock and produce an invoice nobody approved. A spend ceiling with no throughput limit lets one tenant burn the month's capacity in a day.
Concurrency is the limit that actually protects the system
Start with Little's Law, the only queueing theory needed here. In a stable system, requests in flight equal arrival rate multiplied by time in system. At 20 requests per second and 4 seconds each, you have 80 in flight at all times. That is the number your hardware has to hold and the number your admission control has to know.
The binding resource on a GPU server is usually not compute. It is the key/value cache, because every in-flight request holds one for its whole lifetime. Do the arithmetic by hand once. Take 32 layers, grouped-query attention with 8 key/value heads, head dimension 128. Per token per layer you store a key and a value: 2 × 8 × 128 = 2,048 values, which at 16-bit is 4 KB per layer and 128 KB per token across 32 layers. A request holding 8,000 tokens of context sits on roughly a gigabyte of GPU memory until it finishes.
On an 80 GB card with about 16 GB of weights resident, that leaves roughly 60 GB for cache, so somewhere near 60 concurrent requests at that context length. Not six hundred. When a continuous-batching server like vLLM or TGI runs out of cache blocks it stops admitting new sequences and may preempt running ones, and every client feels it at once. Your concurrency limit comes from that number, divided across tenants, with headroom. Most limits we find in config files were typed by someone who never did the division.
The number in your config should be derived, not chosen
Measure three things on your own hardware before setting any limit. Cache bytes per token for the exact model and precision you serve, which is 2 × layers × kv_heads × head_dim × bytes_per_value. Free memory after weights and activations, which gives maximum concurrent tokens. Tokens per second per stream at your target batch size, which gives the latency each admitted request will see. Concurrency limit, token limit and queue depth all fall out of those three numbers.
This is why admission control beats autoscaling here. Scaling out is the right long-run answer and the wrong short-run one. In the minutes it takes a new replica to become useful, the only thing between a burst and a fleet-wide latency collapse is your willingness to say no early. A fast 429 beats a 90-second wait followed by a gateway timeout, and it beats a timeout the client retries three times.
Picking the algorithm
The algorithm matters less than the unit, but it still matters. Six patterns are worth knowing, and the last one is the one people forget exists.
| Algorithm | State per key | Burst behavior | Use it for |
|---|---|---|---|
| Fixed window counter | One integer and a window id | Allows 2× the limit across a window boundary | Nothing customer-facing. The boundary burst is real and clients find it |
| Sliding window log | One timestamp per request | Exact, no boundary artifact | Low-volume limits where precision is worth the memory |
| Sliding window counter | Two integers, weighted by window overlap | Approximate, small error, no boundary spike | The cheap default for per-minute token limits at scale |
| Token bucket | A level and a timestamp | Deliberate burst allowance up to bucket depth | Per-key token throughput, where short bursts are legitimate |
| GCRA | A single timestamp | Equivalent to a token bucket, smoother emission | High-cardinality limits where per-key memory is the constraint |
| Concurrency semaphore | A counter with lease expiry | Hard ceiling, releases on completion | In-flight caps. Nothing else protects tail latency |
Two implementation notes carry most of the risk. Check and decrement have to be atomic, or two nodes both see room for one more request; a Redis Lua script does that in one round trip. And the limiter must not become the outage. Our default is to fail open on fairness limits with a small in-process fallback bucket per node, and fail closed only on the hard spend ceiling for keys already inside a warning band. Failing closed on everything turns a Redis blip into an outage. Failing open on everything turns it into an invoice.
At scale, do not run every check against a central store. Give each node a short lease on a slice of the global budget and reconcile. Nobody has ever complained that a limit was enforced at 10,300 tokens per minute instead of 10,000.
Where to enforce each limit
Enforcement is layered because no layer sees everything. The edge knows the source address and nothing about tokens; the application knows the token count and nothing about the fleet. Put each control where the information lives.
| Layer | What it can see | What it should enforce | Where it fails |
|---|---|---|---|
| Edge / CDN / WAF | IP, ASN, TLS fingerprint, path, header shape | Volumetric floods, obvious bots, request body size ceiling | Cannot distinguish an expensive request from a cheap one |
| API gateway | Authenticated key, route, request count, concurrency | Per-key request rate, per-key in-flight cap, body size, timeout | Does not know output length until the response is finished |
| Application | Tokenized input, model, requested max_tokens, tenant | Token throughput, spend quota, model allowlist, cost reservation | Too late to stop a flood that already opened a connection |
| Model server / scheduler | Cache occupancy, batch composition, queue depth | Max batched tokens, max sequence length, queue admission | Has no idea who the tenant is or what they pay |
The layer people skip is the model server. A maximum sequence length and a maximum number of batched tokens in the serving config is the last line of defense against one pathological request evicting everyone else's cache. Set it even when the application already checks, because the application will eventually be bypassed by an internal service someone wired straight to the backend.
You do not know what a request costs until it is over
Input tokens are countable before you start. Output tokens are not knowable until the model stops, and the model stops when it decides to unless the client capped it. Any accounting scheme has to deal with that.
The pattern that works is reserve and reconcile. At admission, reserve the worst case, which is input tokens plus the requested max_tokens priced at that model's rate, and debit it against the tenant's live budget. On completion, reconcile to actual usage and release the difference. If the client did not set max_tokens, apply a server-side ceiling per key rather than letting the context window define the reservation.
Three details make it work. Cancellations must release the reservation, which means wiring the disconnect signal from the HTTP layer to the scheduler and giving the release path the same care as the acquire path. Streaming responses should meter as they emit, so a client that disconnects at token 3,000 of 4,000 pays for 3,000. And reservations need a lease with an expiry, or a crashed worker leaks budget until a tenant is mysteriously out of quota.
Build Order — Default Priority Weights
Default weights summing to 100 for a first limiter build. Move spend quota to the top if you resell a metered upstream model.
Multi-tenancy and the noisy neighbor
A shared fleet with a global limiter is not multi-tenant. It is first-come-first-served: the tenant with the tightest retry loop wins and the most valuable workload loses, because polite clients back off and impolite ones do not.
Four mechanisms fix this, in the order we usually add them. A per-tenant concurrency floor guarantees every paying tenant in-flight slots nobody else can take. A weighted share of the remainder distributes leftover capacity by plan tier rather than arrival order. Priority classes separate interactive traffic, where a human is waiting, from batch traffic, where nothing is. Burst credits let a tenant accumulate unused allowance up to a cap, so modest bursty usage is not punished for being bursty.
Run interactive and batch through separate queues, not one queue with a priority field. A shared queue still lets a large batch job hold cache blocks an interactive request needs, and that request waits for the batch to finish decoding whatever its priority number says. Separate replica pools, or separate scheduler queues with hard cache reservations, is what delivers the latency difference the plan tiers promise.
The 429 contract
How you reject matters as much as when. A rejection carrying no information turns clients into a retry storm, and a retry storm turns a brief overload into a sustained one.
Return 429 when the client exceeded its own limit and 503 when you are out of capacity. The distinction tells the client whether waiting will help and tells support whose problem it is. RFC 6585 defines 429; RFC 9110 defines Retry-After, which takes seconds or an HTTP date. Always send it. A client left to guess guesses wrong in the direction that hurts you.
Send the remaining budget on every response, not only on rejections, so a client can pace itself instead of discovering the limit by hitting it. The IETF draft on rate-limit header fields has changed shape more than once: earlier revisions used three separate headers for limit, remaining and reset, later ones fold them into a single structured field alongside a policy header. It is not a published standard, so pick the form your clients parse today and document it.
Two more things belong in the contract. Support an idempotency key on any request that costs money, so a client retrying after a timeout does not pay twice for work you already did. And publish the backoff you expect: exponential with full jitter, the client sleeping a random interval between zero and the current ceiling. Deterministic backoff synchronizes every client that failed in the same second and they all return together. Jitter is the difference between a recovery and a second outage.
Send it over and we will tell you what we would change.
Email your limiter config, a day of request logs with input and output token counts per key, and the model, precision and GPU you serve on to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comAbuse comes in four shapes
Abuse of an inference API does not look like abuse of a web application. There is rarely an exploit. There is a valid key doing valid things at a volume nobody intended to pay for.
Leaked credentials. A key committed to a public repository, pasted into a support ticket, or shipped inside a mobile app bundle. Automated scrapers find public keys fast, so plan for minutes rather than days. Give keys a distinctive constant prefix so scanning services recognize them, scope every key to a set of models and a spend ceiling, make rotation one click rather than a migration, and never let a key a browser can read talk to the model directly.
Cost amplification. No credential theft involved. A client sets max_tokens to the ceiling, or pads context with a hundred thousand tokens of retrieved chunks it never reads, or builds an agent loop that calls the model on every failure and fails often. The bill grows by two orders of magnitude while request counts barely move. Per-key ceilings on context and output length are the defense, plus a hard cap on model calls per logical operation.
Extraction and distillation. Systematic querying to reproduce your system's behavior elsewhere, or to lift a curated corpus one answer at a time. The signature is high prompt diversity with shallow session depth, steady volume without human rhythm, and coverage that walks a subject space rather than following a task. Watermarking is weak. Rate-shaping per authenticated end user, not just per key, is stronger.
Content abuse. A tenant generating material that violates your upstream provider's terms, which puts your account at risk rather than theirs. If you resell a hosted model, this is your exposure and your terms of service should say so. Classification inbound, sampled review outbound, and a documented suspension path are the minimum.
Abuse Signals — Default Scoring Weights
Starting weights for a composite abuse score. Every signal is relative to the key's own history, never to a fleet-wide threshold.
Detection that does not page you at three in the morning
Absolute thresholds generate noise, because your largest customer legitimately looks like your worst attacker on every raw count. Every useful signal is a ratio or a delta against the same key's own history.
Alert on spend velocity first: dollars per hour for a key against its own trailing median, with a floor so small keys do not trip on rounding. Alert on distribution shift second: if p95 tokens per request doubles inside an hour, something changed in that integration and you want to know before the invoice does. Alert on concentration third: one key crossing a large share of fleet cost deserves a human look even when it is legitimate, because it is also your availability risk.
Two habits make the alerts usable. Attribute every token to a key, tenant, model and route at write time in one events table, because reconstructing attribution afterward from partial sources turns a one-hour investigation into a one-week one. And give every alert a response smaller than suspension: throttle to a floor, cap max_tokens, or move the key to the batch queue. Teams whose only control is an off switch hesitate to use it, and hesitation is what makes an incident expensive.
Quotas, and what happens when someone hits one
A hard cap that returns an error is honest and occasionally hostile. Before denying, work the ladder: serve from a semantic cache if the request is close enough to one you answered recently, route to a smaller model, reduce the output ceiling, or accept the request into a batch queue with a callback. Degrading beats denying for most internal tools. Denying beats degrading when correctness matters more than availability, which is why the choice belongs to the customer at plan level rather than to you at incident time.
Get the reset semantics right. If every tenant's monthly window resets at midnight UTC, every blocked client returns in the same second and you have built a scheduled thundering herd. Offset each tenant's window by a stable hash of the tenant id so resets spread across the hour, and do the same for daily windows. It costs nothing and removes a class of recurring incident.
Model the quota hierarchy explicitly: organization, then key, then end user if you resell. Each level needs its own limit and the effective limit is the minimum across levels. Flatten it and a customer's twelve keys will each stay inside their limit while the organization runs four times over its plan.
The failure modes we see most
- Requests per minute is the only limit, so cost per request varies by a factor of a thousand inside a single quota.
- No concurrency cap at all, so tail latency is set by whoever opens the most simultaneous streams.
- Check and decrement are two round trips, so under load two nodes both admit the last request.
- Cancelled and disconnected requests never release their reservation, and budgets leak until a tenant is mysteriously blocked.
- 429 returned with no
Retry-After, so every client retries immediately and turns a spike into a sustained overload. - Every quota resets at the same instant, producing a synchronized stampede on the hour.
- The limiter has no fallback, so an unreachable counter store takes the entire API down with it.
A four-week build order
Implementation Sequence
Shadow mode in step three is not optional. Run the limiter for a week logging what it would have rejected without rejecting anything, then read the list. Teams that skip it discover at cutover that a limit they were sure of would have blocked their largest customer's nightly job.
Step six gets cut and should not be. A limiter sits on the hot path of every request, so its own throughput ceiling and failure behavior have to be measured. A limiter that makes three sequential round trips per check where one script would do adds milliseconds at p99 to every call in the system.
What a production inference API has
- A concurrency cap derived from measured cache capacity, with a per-tenant floor
- Token throughput limits on input and output separately, per key and per organization
- A spend quota with a hard ceiling, reserved at admission and reconciled at completion
- Separate queues and separate capacity for interactive and batch traffic
- 429 versus 503 used correctly, with Retry-After on every rejection
- Remaining budget returned on successful responses, in a documented header format
- One usage events table attributing every token to tenant, key, model and route
- Alerts on spend velocity, distribution shift and cost concentration, all relative to each key's own history
- A throttle action that is smaller than suspension, reachable in one click
Bottom line
Rate limiting an inference API is a metering problem wearing a traffic-management costume. Get the unit right and the rest is ordinary engineering: a semaphore for concurrency, a bucket for throughput, a ledger for spend, each enforced at the layer that can see the relevant quantity. Get the unit wrong and no algorithm saves you, because you will be precisely enforcing a number that has nothing to do with what the request costs. Measure the cache arithmetic on your own hardware, meter before you limit, run it in shadow mode before you enforce it, and make every abuse signal relative to the key's own history.
Frequently asked questions
Keep a request-rate limit as a cheap outer guard against floods and set it generously. It should never be the limit that binds in normal operation. The binding limits should be concurrency, token throughput and spend, because those map to what the request actually consumes.
Reserve the worst case at admission, which is input tokens plus the requested output ceiling, then reconcile to actual usage on completion and release the difference. Apply a server-side output ceiling per key so the reservation stays bounded, and meter streaming responses as they emit.
429 when the client exceeded its own limit, 503 when you are out of capacity, and Retry-After on both. Return remaining budget on successful responses too, so clients pace themselves. Publish the backoff you expect, which should be exponential with full jitter, and support an idempotency key so a retry does not bill twice.
Per-tenant concurrency floors so every tenant has slots nobody can take, weighted distribution of remaining capacity by plan tier, and separate queues or replica pools for interactive and batch traffic. One queue with a priority field does not deliver the separation, because a long batch decode still holds cache blocks an interactive request needs.
Four shapes: a leaked key used from many networks at once, cost amplification through oversized context or output ceilings, systematic extraction with high prompt diversity and shallow sessions, and content abuse that endangers your upstream account. Detect all four against each key's own trailing history rather than fleet-wide thresholds.