Streaming is a commitment

Tags: streaming, latency, chat

Streaming an answer buys perceived speed and spends the ability to revise it. Text that has appeared has been read, and it cannot be taken back without the user watching it be taken back. Every check you would have run on the finished answer now runs before the first token, or it does not run at all.

The affordance is not the decision

Showing tokens as they arrive is a component. It is documented everywhere and it takes an afternoon.

The decision underneath it is what nobody writes down: streaming converts an atomic answer into a public draft. A buffered call produces one event — here is the answer, or here is an error. A streamed call produces a stream of small irreversible publications, each of which the user has already acted on by the time the next arrives.

Four things follow, and none of them are visible while you are building the happy path.

You cannot validate after the fact. A schema check, a late-firing guardrail, a citation that turns out not to resolve, a total that does not reconcile — all of it has to pass before the first token or be abandoned. A product that streams and then corrects reads as unreliable even when the corrected answer is right, because the correction is the more memorable event.

You cannot reorder. Streamed output commits to its own structure as it goes. If the best version of the answer needs its conclusion at the top, you either buffer until you know what the conclusion is, or you accept a shape the model can commit to from the first token.

The mid-stream failure is worse than the buffered one. A buffered call that fails leaves an error, which is legible. A stream that dies at sixty percent leaves something that looks like an answer and reads like one. It is the more dangerous failure and it is the one that gets designed last, if at all.

Length becomes a thing the user watches. Under a buffered call, a long answer and a short one cost the same attention. Under a stream, the user sits through the difference.

The fork

The middle option is the one worth reaching for first: stream the work and buffer the artifact. Show progress as it happens — steps taken, sources opened, what the system is doing now — and deliver the result in one piece when it is whole. That keeps the responsiveness, which is the thing streaming was actually bought for, without making a draft public.

What streaming needs that buffering does not

• A terminal state. If a stream can stop without a visible "this is finished", nobody can tell complete from stalled, and the difference matters most on the answers people were going to act on.
• A mid-stream failure design. What is on screen when the connection drops at sixty percent, and what does the interface say about the part that did arrive? "Something went wrong" beneath four good paragraphs is not an answer to that question.
• A commitment at the end. Streaming is a way of delivering; it is not a way of keeping. An answer that streams beautifully and then evaporates is a demonstration of latency.

Prior art, and where this differs

There is nothing wrong with the entry as documentation — token-by-token delivery is a real component and it is worth having a name for. But a catalog of things you can add cannot tell you when adding one is the wrong call, and streaming is the clearest case in the category: it is the default nobody argues with, and it is wrong for a meaningful share of the outputs it gets applied to.

Grounded in

Poppy streams model responses into a two-panel thread. The streaming was the cheap part. What made it usable was what happens when the stream stops: the thread persists and titles itself from the first message, so a teacher who dictated a parent email between classes can find it again the following week. Building the stream took an afternoon; deciding that a streamed answer had to become a durable, findable artifact is the part that was actually design.

The same product is the reason the fork above is not theoretical. Poppy's four one-tap actions produce short, structured outputs — a quiz, a reading-level adaptation — and streaming those is worse than not streaming them.

Anti-patterns

• Streaming, then rewriting. The correction is more memorable than the answer it corrected. If a check can fail late, buffer until it has passed.
• Streaming a structured artifact. A half-drawn table is not fifty percent of a table, it is a broken table. The same goes for JSON, for code that has to parse, and for anything with a header row.
• The typewriter over an answer you already have. Animating text you have finished computing is not streaming, it is a costume — real time spent simulating work that is over. See don't animate a guess territory: an interface that performs effort it is not expending is lying about something small, which makes it harder to believe about something large.
• Streaming with no terminal state. Finished and stalled look identical, and the user resolves the ambiguity by refreshing.
• Streaming short answers. Twelve words delivered one at a time makes a fast system look slow. Streaming is a treatment for waiting, and there is nothing to treat.

The smallest version worth building

A two-column list of every output type in the product, marked streams or buffers, with one line of reasoning each. Run the fork above down it once.

Then, for everything in the streams column, two things that do not exist yet: a visible terminal state, and a screen for the stream that died at sixty percent. That second one is an afternoon, and it is the only part of this anyone will thank you for.

Related patterns

• The character is a state machine — what carries the waiting when there is no partial artifact to show.
• Design the failure state first — the stream that dies mid-answer is a failure mode in its own right, and it is not the same one as the model being wrong.
• Wizard or conversation — structured flows produce structured outputs, which is most of the buffer column.