Streaming is a product decision wearing a transport costume
The engineering conversation about streaming is usually about server-sent events against websockets, which is the least consequential decision in the whole feature. The consequential ones are what the interface shows before the first token, what it does when the connection dies at token nine hundred, whether the stop button stops anything, and whether a half-written table on screen is something a user can act on. Those decisions are made in the product, not the transport, and skipping them is why streamed products often feel less trustworthy than the batch versions they replaced.
Start with what streaming buys, honestly. It buys perceived speed, which is real and large. It buys evidence of life during a long operation. It lets a reader judge an answer early and stop a bad one, which saves their time and your tokens. And it makes cancellation meaningful, because there is something to cancel.
What it costs is less often written down. You lose the ability to validate a whole answer before showing any of it, which matters if anything downstream needs to be well-formed. You gain three or four interface states you did not have. Moderation and safety checks that were a simple gate become a streaming problem. Analytics get harder, because "the response" is now a sequence with a beginning and, sometimes, no end. And the client is now a state machine that has to be right, on every platform, including the one you did not test.
You are probably here because
- Users report answers that "just stop" and you cannot reproduce it
- The page jumps while the answer writes itself and people complain about scrolling
- Stop clears the screen, and the bill says the generation kept going
- Structured output looks fine when complete and unrenderable halfway through
The five-states section covers the first, scroll covers the second, cancellation the third, and the structured-output section the fourth.
Five states, and the fifth is the one that gets skipped
Every streaming surface renders exactly five states. Write them out before writing the component, because retrofitting the fifth is where the bugs live.
Waiting is the gap between the request and the first token. It is not a spinner if you can help it — a specific status line beats a generic one, and the specificity is free because your server knows what it is doing. "Searching your documents" is a better wait than an animation, and it converts dead time into visible progress.
Streaming is text arriving. The only rule that matters here is that layout must not jump.
Complete is a terminal event received and the answer assembled. Show the affordances that only make sense on a finished answer: copy, regenerate, feedback, export.
Failed is a terminal error event, and the important rule is that you keep what already arrived. Deleting three paragraphs the user was reading, to replace them with an error toast, is the single most resented behavior in this category.
Interrupted is the fifth: the connection closed, or the user hit stop, or the generation hit a token cap, and there was no terminal event. This state is neither success nor failure and it must be visible as itself. Mark the partial output as partial, keep it, and offer continue and retry. A product that cannot distinguish interrupted from complete will eventually show a customer half an answer with a copy button under it.
Render cadence, and the honest limits of smoothing
Tokens do not arrive evenly. They come in bursts shaped by batching on the server and by the network, so raw rendering produces a stutter that reads as jank even when throughput is good. The fix is to decouple arrival from paint: buffer incoming text, and drain the buffer at a smoothed rate tied to the display refresh rather than to arrivals.
Two calibrations are worth knowing. Comfortable reading runs somewhere around two hundred to three hundred words a minute, which is roughly five to eight tokens a second. A modern model on a warm path emits tokens much faster than that. So above a fairly low threshold the stream is already outrunning the reader, and further speed buys nothing perceptually while smoothing still does. Below that threshold — a large model under load, a long reasoning step — the reader is waiting on tokens and smoothing cannot invent them.
Practical settings: batch DOM updates once per animation frame rather than once per token; drain at a rate slightly above reading speed so the text is always a little ahead; and hold back the final partial word until it is complete, because a word that grows letter by letter is the detail that makes an interface feel cheap.
Structured output does not stream, and pretending it does is expensive
Prose streams beautifully. A JSON object does not, because a partial object is not an object — it is a syntax error with a future. Four approaches exist and each is right somewhere.
| Approach | How it works | Good for | What it costs |
|---|---|---|---|
| Buffer the structure | Stream nothing; show a skeleton, render on completion | Forms, extractions, anything a machine consumes next | All of the perceived-speed benefit |
| Two channels | Stream a prose summary, deliver the object at the end | Analysis products where a person reads and a system stores | Two things to keep consistent, and they will drift |
| Tolerant partial parsing | Repair the incomplete JSON on each chunk, render what is valid | Long objects with independent top-level fields | Real complexity; fields flicker as they resolve |
| Field-at-a-time events | Server emits one event per completed field | Anything you control end to end — the best option when you do | Server work, and a schema for the event stream |
The last row is the one to prefer when the API is yours. Emitting a completed field as its own event moves the parsing problem to the side that knows the schema, and it turns the client into something simple. If the API is not yours, tolerant parsing is workable but must never let a half-parsed value reach a place a user could act on — a partially streamed number is a wrong number, and a partially streamed monetary amount is a wrong number with consequences. Render a field only once its value is closed.
Markdown, code and the trailing-fence problem
Most products render markdown, and markdown mid-stream is unbalanced by construction. An unclosed code fence turns the rest of the answer into a code block. A table renders as pipes until its second row exists. A link is bare bracket text until the parenthesis arrives. A heading appears as a hash. Each is momentary and each looks like a defect.
What works: parse incrementally with a renderer tolerant of unterminated constructs, and speculatively close open ones for display purposes only. When a fence opens, render it as a code block immediately rather than waiting; when a table's header row arrives, render the header. Reserve the vertical space so nothing below shifts. And keep the raw text as the source of truth, separately from the rendered tree, so copy gives the user what the model wrote and not what your renderer inferred.
Where streaming defects come from, in the front ends we review
Relative frequency in our review work, ranked rather than measured. The top two account for most user-visible complaints.
Scroll is the detail users complain about most
Text growing at the bottom of a container is a scroll problem, and the naive implementation — scroll to bottom on every chunk — makes it impossible to read anything above the fold while a response is writing. Users notice this within one session and describe it as the interface fighting them.
The behavior that works is stickiness with release. Auto-follow only while the viewport is already at or very near the bottom. The moment the user scrolls up, stop following and show an unobtrusive control to return to the latest. Re-engage following when they return to the bottom themselves. Use a tolerance of a few dozen pixels rather than exact equality, because momentum scrolling and sub-pixel layout will otherwise flip the state at random.
Two more details: reserve height for content that is about to appear so nothing below jumps, and be careful with anchoring behavior in the browser, which can fight your own scroll logic and produce a container that drifts.
Showing the machinery
Agent products stream more than prose. They stream steps: a search, a document read, a calculation, a tool result. The instinct is to hide all of it behind a spinner, and that is a mistake — visible steps buy patience, and patience is the resource you are short of during a ten-second operation.
Show what is happening in the user's language: the tool being used, the target if it is meaningful, and a count when there is one. Collapse completed steps into a compact line, keep them expandable, and never make the machinery the main event — it is scaffolding around an answer, not the answer.
Be deliberate about intermediate reasoning. If a model emits a thinking phase, streaming it raw into the product is rarely the right call: it is unedited, it sometimes contradicts the final answer, and users read it as a commitment. Summarize it into status lines, or put it behind a disclosure that is closed by default and clearly labeled as working notes.
Stop has to stop the server, not just the screen
An abort that only tears down the client connection leaves the generation running, and you pay for every token nobody will read while the capacity is taken from a request that has a reader. Propagate cancellation to the upstream call, confirm it, and record the cancellation on the request. Then keep the partial answer, label it as stopped, and offer continue — users press stop because the answer went the wrong way, and what they want next is almost never a blank screen.
Send us the stream and we will try to break it.
A staging URL or a short screen recording of a long answer. Email contact@precisionfederal.com. We kill the connection halfway, scroll up mid-stream, press stop, run it under a screen reader, and send back a written list of what we found. One business day, no charge, no meeting.
contact@precisionfederal.comErrors in the middle, and what the user should see
Once the response headers are out, the status code is spent. A failure at token nine hundred arrives as an event inside a successful HTTP response, which means your client must treat an in-band error as a real failure and your server must always send one. The design rules follow from that:
- Keep the partial text and mark it as incomplete rather than clearing it
- Say which failure it was in one plain sentence — interrupted, refused, over a limit, upstream unavailable
- Offer the action that matches: retry for transient, edit for a limit, nothing but an explanation for a refusal
- Do not auto-retry a long generation silently — the user has read half of a different answer and the second one will not match
- Treat a clean close with no terminal event as failure, which is the case almost every client gets backwards
The corresponding server-side contract — the event vocabulary, the heartbeat that stops proxies from killing an idle stream, and the terminal event that carries the assembled object — is covered in API design for AI products. The client rules above only hold if the server keeps that end of the bargain.
Accessibility, which is where streaming quietly fails
A streaming text container is a nightmare for assistive technology if it is naively marked as a live region. Every chunk becomes an announcement, so a screen-reader user hears fragments of words for the whole generation and can follow nothing. It is a bad experience that almost never surfaces in testing because almost nobody tests it.
What works better: render the streaming text in a container that is not itself live, and use a small polite status region for state changes only — "generating", "response complete", "response stopped". Announce completion once, then let the user navigate the finished text with their own reading commands, which is what they would prefer anyway. Keep focus stable during streaming; never move it to newly arriving content. Make stop reachable by keyboard and give it a real accessible label. If motion is reduced in the user's system preferences, drop the typing animation and render text as it arrives in whole blocks.
Measure the things a user would recognise
Time to first token is the headline, and it is not sufficient. Measure time to first useful content as well, since a preamble that restates the question is not content. Track the p95 inter-token gap, because one three-second stall mid-answer is worse for trust than a uniformly slower stream. Watch the stop rate and where in the response it happens — early stops mean the answer started wrong, late stops mean it was too long. Track the share of streams that end without a terminal event, which is your truncation rate and is usually higher than anyone expects. And track the regeneration rate, the copy rate, and abandonment during the wait, all of which move with latency in the way survey answers do not.
When not to stream
Streaming is a default, not a law. Do not stream when a machine is the consumer, because an API client wants an object and assembling one from a stream is work you are exporting. Do not stream a response that takes under a second, since the flicker costs more than the wait. Do not stream structured output to a surface that will act on it. Do not stream when the answer must pass a check before anyone sees it — you cannot un-show a paragraph. And for long batch work, a job with progress is the right shape rather than a held connection; the trade-off between those is laid out in streaming versus batch for model serving.
The mistakes we are called in to fix
- A clean socket close rendered as a complete answer, shipping truncated output with a copy button
- Scroll pinned to the bottom unconditionally, so nobody can read the top while it writes
- Stop that only aborts the client, leaving the generation running and billing
- Errors that wipe the partial answer the user was reading
- Partial JSON rendered into fields where a half-formed number looked like a real one
- The whole transcript in an aria-live region, unusable with a screen reader
- One DOM update per token, pinning the main thread on long answers
- Raw model reasoning streamed as product copy, then contradicted by the final answer
- A proxy buffering the whole response, so streaming worked locally and nowhere else
Before you call the stream finished
- All five states render, including interrupted, and each has been seen by a human
- Killing the connection mid-answer keeps the text and says what happened
- Stop cancels upstream, is confirmed, and preserves the partial output
- Scroll follows only when the user is at the bottom, with a return control
- Markdown, code fences and tables render without layout jump mid-stream
- No structured value is shown before it is closed
- A screen reader announces state changes, not every fragment
- Reduced-motion preferences drop the typing animation
- Truncation rate and inter-token p95 are on a dashboard
- It has been tested behind a real proxy, on a phone, on a poor connection
Common objections
Our framework handles streaming. Is this not solved?
Frameworks solve the transport and the token assembly, which is the afternoon. None of them decides what your product shows when the connection dies at token nine hundred, whether the stop button reaches your server, or how a screen reader experiences the page. Those are product decisions and they are the ones users feel.
Should we fake streaming for a response we already have?
Occasionally, and be careful. Revealing a cached answer at reading speed keeps the interface consistent and avoids one path appearing suspiciously instant. It also spends real time you did not have to spend, and it will be resented if a user notices. If the answer is short, show it; if you fake it, keep the pace close to genuine generation and never fake it on a path where the user could have had the answer immediately.
How do we moderate content that is already on the screen?
You cannot un-show it, so decide before you stream. Either gate the response and lose the perceived-speed benefit for that class of request, or stream with a check running alongside that can end the stream and replace the message, accepting that a fragment was briefly visible. Pick per feature rather than globally, and state which one you chose.
Does streaming make our analytics unreliable?
It makes them different. A response is now an interval with a start, a possible end, and an outcome, so record all three: first token time, terminal event type, and token count delivered. Products that only log completed responses systematically undercount their own failures, because the truncated ones never reach the logger.
Bottom line
Streaming is the cheapest large improvement available to an interactive language-model product, and the improvement is entirely in perception, which means it is entirely in the interface. The transport is an afternoon. The week is spent on the five states, on keeping partial output when things go wrong, on cancellation that reaches the server, on scroll that respects the reader, and on the accessibility path almost nobody exercises. Build those and a streamed answer feels like a system that is working. Skip them and you have shipped a faster-feeling product that occasionally shows people half a sentence and calls it done.
Frequently asked questions
Server-sent events for anything one-directional. It is plain HTTP, it passes proxies, browsers reconnect on their own, and your existing authentication and observability keep working. Reach for a websocket only when the client sends data mid-stream, such as live interruption or tool results returning, and accept that you are taking on a second connection lifecycle to get it.
Preferably you do not stream the object itself. Emit one event per completed field from the server, or stream a prose summary alongside a final object. If you must parse partial JSON on the client, never render a value until it is closed, because a half-streamed number reads as a complete and wrong one.
Keep everything that arrived, mark it clearly as incomplete, say in one sentence what happened, and offer continue or retry. Never clear the text the user was reading. And treat a connection that closes without a terminal completion event as a failure rather than a finished answer, which is the default most clients get wrong.
The common implementation does. Marking the whole streaming container as a live region makes assistive technology announce every fragment as it arrives. Render the text in a non-live container, use a small polite status region for state changes only, announce completion once, and keep focus where the user left it.
Something on the path is buffering the response, most often a reverse proxy with response buffering on by default. Disable buffering for the route or set the header your proxy honors, and verify with a plain command-line client rather than a browser, since browser tooling can mask the arrival timing.