Four assumptions that stop being true
Most AI APIs are designed by someone who has shipped twenty good REST services, which is why they fail in a consistent way. A well-built CRUD API rests on four assumptions: latency is predictable, a response is complete or absent, the same input gives the same output, and every call to a route costs about the same. All four are load-bearing, all four are invisible, and all four stop being true once a model sits behind the handler. The result is fine in the demo and generates support tickets in week two.
Latency is not predictable. A warm cache hit with a short answer returns in a few hundred milliseconds. The same route, same code, with a cold index and a hundred-page input, runs well over half a minute. That is not a tail you trim with a faster instance. It is the shape of the workload, and the spread between median and p99 routinely runs one to two orders of magnitude.
A response is often partial. A batch over fifty documents can finish forty-two and hit a limit on the forty-third. In CRUD there is no such thing as most of a row. Here there is, it usually has real value, and if the API cannot express it your options are to lie or to throw the work away.
The same input does not produce the same output. Even at temperature zero, floating-point non-associativity across batch sizes and kernel selections means identical inputs can yield different tokens. Add a retrieval step over an index that changes hourly and the output is a function of time. Every retry is a new answer, not a repeat.
Cost per call varies enormously. Two requests to one path can differ by three orders of magnitude in tokens. A limiter counting requests, a quota counting calls and a capacity plan counting QPS all measure something uncorrelated with what you pay.
Every decision below absorbs one of those four. If you take nothing else: pick the call shape per operation, make retries safe, and put the model version and the usage in the response.
You are probably here because
- Customers keep calling one endpoint flaky, and every trace you pull looks fine
- A retry somewhere in the stack ran the same job twice and billed for it twice
- A batch failed on document forty-three and returned a 500, and the forty-two that finished are gone
- Someone’s output changed overnight and nothing in your code changed with it
The call-shape and idempotency sections below cover the first three and the versioning section covers the last, but all four are the same root cause: an inference endpoint breaks four assumptions a CRUD API is quietly built on.
Choose the call shape per operation, not per product
Three shapes are worth shipping, and the common failure is picking one for the whole product. Synchronous fits when the p99 clears every timeout between client and handler and a partial answer is worth nothing. Streaming fits when a person reads output as it arrives, because perceived latency collapses even though total latency does not move. An asynchronous job fits when work outlives a connection, or when you want to retry it without the caller knowing.
Our threshold for the first split is roughly ten seconds at p99. Under it, synchronous is simpler for everyone and the simplicity is worth real money in integration time. Above it you are fighting infrastructure you do not control: idle timeouts on managed load balancers cluster around sixty seconds and customer proxies are often tighter. An endpoint with an honest ninety-second p99 gets reported as flaky. It is not flaky. It is exceeding a timeout somebody set in 2019.
| Shape | Use when | What it costs you | What it costs the client |
|---|---|---|---|
| Synchronous POST returns the result | p99 under ~10s and partial output is worthless | Connection held throughout; hard concurrency cap per instance | Almost nothing. One call, one result |
| Streaming SSE over a held connection | A person reads it; first token beats last | Heartbeats, buffering, in-band errors, cancellation | An event loop, an assembler, a policy for early ends |
| Async job 202 plus a job resource | Long or batch work that must survive a disconnect | Queue, store, status resource, retention rules | A poll loop or webhook consumer, plus state |
If you offer synchronous and streaming forms of one operation, the assembled stream must deserialize into the same object the synchronous call returns. Those two drift silently, and when they do you have shipped two APIs sharing a URL.
Where the design attention goes — our default weights
Weights sum to 100. Our starting point for a first public version, not a measurement. Move them before you design.
The job resource is the design that ages best
When work is long or expensive, model the job rather than the answer. POST /v1/extractions returns 202 with a Location header and a body carrying an id, a status and a timestamp. GET /v1/extractions/{id} returns that resource with progress and, once terminal, the result. POST /v1/extractions/{id}/cancel stops it. Nobody holds a connection open for work they are not reading.
Define the terminal states before writing the handler, because retrofitting one is a breaking change. We ship six: queued, running, succeeded, partial, failed, cancelled. The one that surprises people is partial, and it pays for itself. A fifty-document batch that completes forty-two has produced something the customer wants, and a 500 discards it, bills for it, or both.
Two fields save real support time. poll_after_ms hints when to check again; without it clients invent a hundred-millisecond loop and you find out at their launch. expires_at puts retention in the object rather than a page nobody reopens. Cancellation has to be real too: an endpoint that flips a status field while the GPU keeps working is a lie you are paying for. Propagate the stop to the worker and return what finished as a cancelled job.
Idempotency, because the retries are already happening
Any endpoint with a multi-second p99 will be retried. SDKs retry on timeout by default, meshes retry on 502, and a person watching a spinner refreshes. On a CRUD API that is harmless. Here a naive retry starts a second inference, gives a different answer, and charges twice. The first time a customer sees two invoices for one document, the conversation is not about API design.
The fix is the pattern the payments industry settled on years ago and it transfers directly. Accept an Idempotency-Key header holding a client-generated UUID. At admission, before any model work starts, write the key with a hash of the request body into a store with a unique index on the key. Winning inserts do the work and save the response. Losing inserts return the stored response or the in-flight job when the body hash matches, and 409 when it does not, because a reused key with a new body is a client bug and serving it hides one.
Three details decide whether this works. Write the key inside the same transaction that creates the job, or two retries fifty milliseconds apart both start work and the unique index protects nothing. Fingerprint the body so key reuse is caught. Publish the retention window, because clients build recovery logic around it: twenty-four hours is a reasonable default, and any documented number beats none.
How safe is a naive client retry — our rating by design
How much damage a default SDK retry policy does to each design. Judgment, not benchmark: the ordering is the useful part.
Streaming: the protocol is the easy part
Server-sent events over plain HTTP is the right default. One media type, text/event-stream, it passes through ordinary proxies, browsers reconnect on their own, and every language has a client. Reach for WebSocket only when the caller sends data mid-stream, meaning live interrupts or tool results coming back. Choosing it because it sounds more capable buys a second connection lifecycle and loses HTTP caching, auth middleware and most of your observability.
Errors after the headers. Once 200 OK is written the status code is spent, so a model failing at token nine hundred cannot be a 500. The failure travels in band as a terminal error event carrying the same object the non-streaming path returns. Document the harder half too: a clean socket close with no done is a failure, not a finished answer. Teams get that backwards and ship truncated output as complete.
Idle timeouts. A model thinking for forty seconds before the first token looks exactly like a dead connection to every proxy on the path. Emit an SSE comment line every ten to fifteen seconds as a heartbeat. Two bytes, and it prevents the bug that works locally and fails for the one customer with a stricter gateway.
Buffering. Nginx buffers proxied responses by default, so the first token and the last arrive together and the stream is a slow synchronous call. Turn buffering off for the route or send X-Accel-Buffering: no, and verify with curl, because browsers hide the timing.
Cancellation. When the client disconnects, cancel upstream. Four thousand tokens nobody reads is a direct bill, and capacity taken from a request with a reader.
The protocol takes an afternoon; the four items above take the week. Define the event vocabulary before the first handler, because it is a public contract from the first customer onward. A workable minimum: delta for incremental content, tool_call if you expose tools, usage for accounting, error as terminal failure, and done as terminal success carrying the assembled object. No client should reimplement your concatenation rules to reach the object the synchronous endpoint returns.
The response envelope: what every AI response carries
Under whatever the operation returns, five things belong in every response: the result, the model identifier including its snapshot, the parameters as the server resolved them, a usage object, and provenance wherever the answer came from source material.
Resolved parameters, not requested ones. If the caller asked for eight thousand output tokens and the plan caps at four, the response says four. If input exceeded the context window, the response says so in a machine-readable field, not a log line. Silent truncation is the most expensive quiet failure we are called in to diagnose, because the output looks plausible and nothing is red.
Abstention as a value, not a gap. An extraction schema needs a first-class way to say the field was absent, distinct from present but unreadable, distinct again from read with low confidence. Encoding all three as null, or as an apologetic sentence in a field typed as a date, pushes the problem into the customer's parser. Give every field a status enum beside its value.
Confidence, only if it means something. A number a client can threshold on is a contract, and publishing one means owing a definition, a calibration, and a warning that it moves when the model moves. A raw softmax score dressed as a probability of correctness gives customers an alert that quietly stops meaning what it meant.
Provenance. Document id, page, character offsets, and a bounding box when the source is a scan. This is the field that closes the "how do I know this is right?" ticket without reopening the source, and the one most often cut for schedule and added back within a quarter.
Put the usage object in the response, not only in the dashboard
Input tokens, output tokens, cached tokens, the model that served the request, and a request id that appears in your logs. Without it a customer cannot attribute cost to a feature, build a budget guard, or tell you which call was expensive. With it they build their own cost controls, which is the cheapest support you will ever ship. The request id turns "the API was slow yesterday" into a query.
Send it over and we will tell you what we would change.
Email your OpenAPI spec and the measured p99 for each operation to contact@precisionfederal.com. The endpoint list on its own is enough if the spec is not written yet. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comErrors that tell the client what to do next
An error has one job: route the caller into one of four actions. Fix the input, retry now, retry later, or stop and escalate. Design the taxonomy backwards from those four. Two codes leading to one action should be one code with a reason field. One code leading to two actions is a bug that shows up as a retry storm.
Use the shape from RFC 9457, Problem Details for HTTP APIs: a stable machine-readable type versioned like any public identifier, a detail string you are free to reword, the request id, and fields specific to the error such as the limit that was hit. Keep the stable part separate from the prose, or someone will parse your prose.
| Condition | Status | Carry in the body | Client action |
|---|---|---|---|
| Malformed request or bad schema | 400 | Field path, expected type | Fix and resubmit. Never retry |
| Input over a size or token limit | 413 | The limit and the measured value | Split the input, resubmit |
| Guardrail or policy refusal | 422 | Reason code, never the raw filter output | Stop. Escalate to a person |
| Quota or concurrency exceeded | 429 | Which limit, Retry-After, remaining | Back off, retry later |
| Model or dependency unavailable | 503 | Retry-After, whether the work was queued | Retry with backoff and jitter |
| Upstream exceeded its deadline | 504 | Whether partial output was retained | Retry once, then reduce the request |
The anti-pattern worth naming: 200 OK with an error string in the payload. It defeats every retry policy, gateway metric, alert threshold and uptime number you publish. And on 429, say which limit was hit: requests per minute, tokens per minute and concurrent jobs are three constraints with three remedies, and a client throttling the wrong one stays throttled at a fraction of its allowance.
Versioning something whose behavior changes without your code changing
AI products have two independent version axes and most teams ship one. The API version governs the shape of the JSON. The model version governs what the JSON contains. A customer whose integration still compiles and whose accuracy dropped overnight has had no API change by your definition, and has absolutely had one by theirs.
Make model snapshots pinnable and put the pin in every response. Offer a moving alias if you like, but treat it as a prototype convenience. When you deprecate a snapshot, instrument calls per version per account so you can contact the customers still on it, then give a window measured in months. A changelog post is not a deprecation process.
Date-stamped API versions work well: a header carrying 2026-07-28, pinned per account at first call, with a written list of what counts as compatible. Adding a field is compatible. Adding an enum value is not, unless the enum was declared open on day one. Tightening a limit breaks callers though the schema is untouched, and so does changing the system prompt, which is the change that ships on a Friday because it never touches the OpenAPI file. Keep a golden set of real requests with expected outputs and run it against every model and prompt change.
Cost to change after external clients depend on it
Difficulty as we rank it, driven by the migration each change forces on the caller. Design the top three before launch.
Publish the limits as data, enforce them at admission
Maximum input in bytes and tokens, maximum output, concurrent jobs, per-minute quotas, job lifetime, file types, result retention. Every one exists whether or not it is written down, and the only question is whether the customer learns it from a document or from an incident. Serve them from the API: a GET /v1/limits endpoint for the authenticated caller lets an integration batch correctly and adapt when you raise a ceiling. Mirror the volatile ones into response headers.
Then enforce at admission. Count tokens and reject an over-limit request before it reaches a worker, not thirty seconds into generation. Early rejection is cheaper, faster, and the difference between a queue that sheds load and one that collapses.
If you expose tools, the tool schema is a public API
A product that lets callers register tools has published a second interface, usually with none of the discipline applied to the first. Parameter schemas are public contracts: additive changes only, no renames, no type changes. One thing here has no analogue in ordinary API design, which is that the tool description is a prompt, so rewriting it for clarity changes behavior and belongs under code review. Cap tool-call loops at a documented maximum and return the count, so a caller can tell an answer from an exhausted loop.
Webhooks, if you have them
Delivery is at-least-once, so consumers must be idempotent on your event id, and that belongs in the first paragraph of the documentation rather than the last. Sign with HMAC-SHA256 over a timestamp joined to the raw request body, verify against the raw bytes rather than re-serialized JSON, and enforce a tolerance window of a few minutes so a captured payload cannot be replayed forever.
Do not promise ordering: put a sequence number or a state snapshot in each event so a late arrival can be discarded. Retry with backoff, show customers their failed deliveries with a manual replay, and always keep polling available, because many buyers cannot expose an inbound endpoint.
The mistakes we are called in to fix
- One synchronous endpoint for everything, timeout raised each time a customer complains
- 200 OK carrying an error object, blinding every gateway metric and retry policy in the path
- No idempotency key, plus an SDK that retries on timeout by default
- A moving model alias as the only option, so behavior changes arrive unannounced
- Confidence scores with no documented meaning, thresholded by customers in production
- Streaming with no heartbeat, failing only behind the strictest customer proxy
- An error taxonomy of 400 and 500, telling the caller nothing
- A cancel endpoint that only updates a status field while the work keeps billing
A two-week design pass before you publish v1
API Design Sprint
The last two days are the ones teams skip and the ones that pay. Kill a stream halfway and see what the client renders. Send one idempotency key with two bodies. Submit an input two tokens over the limit. Cancel a job and check the worker stopped. Every defect that reaches a customer here was findable in an afternoon.
Before you publish
- Every operation has a documented call shape and a published p99 target
- Idempotency keys accepted, body-fingerprinted, replayed for a stated window
- Every terminal state is reachable and documented, partial included
- Streams carry heartbeats, a terminal event, and errors in band
- Every response reports snapshot, resolved parameters, usage, request id
- Every error maps to exactly one of four client actions
- Limits are readable from the API and enforced before work starts
- Snapshots are pinnable and deprecations are instrumented per account
- Cancellation stops upstream work and returns what completed
- A golden set gates every prompt and model change, with a published diff
Bottom line
An AI API is a normal API with four broken assumptions underneath it, and almost every painful decision traces back to one of them. Latency variance decides the call shape. Non-determinism makes idempotency mandatory rather than nice. Partial results demand a state CRUD never needed. Cost variance is why usage belongs in the response. Get the call shape and the retry semantics right at the start, because those are the two you cannot change later without a migration your customers will resent. The rest can be added in place.
Frequently asked questions
Once the honest p99 passes roughly ten seconds, or when the work should survive a client disconnect. Managed load balancers commonly close idle connections around sixty seconds and customer proxies are often stricter, so a long synchronous endpoint gets reported as flaky when it is outliving a timeout. Batch operations belong on a job resource whatever their duration, because they need a partial state.
Because the latency guarantees retries and non-determinism makes them expensive. A retried CRUD write lands on the same row; a retried inference call runs again, gives a different answer, and bills twice. Accept a client-generated key, store it with a hash of the request body at admission, and replay the stored response.
In band. The status code was spent when the headers went out, so send a terminal error event carrying the same object the non-streaming path uses. Document that a stream ending without a terminal event is incomplete, and make clients treat a clean close with no completion event as failure.
The result, the model identifier including its snapshot, the parameters as the server resolved them, a usage object, a request id, and provenance for anything derived from source material. Resolved parameters matter most: if input was truncated, the response says so in a field a program can read.
Treat shape and behavior as separate axes. Date-stamp the API version for the JSON contract and let customers pin a model snapshot for behavior. Instrument calls per version per account so deprecation is a conversation with named customers rather than a blog post, and gate prompt and model changes on a golden set.