Rendering streaming LLM markdown in Flutter without repainting the world

If you render an LLM chat by re-parsing the whole message on every token, you'll ship a demo that melts on a mid-range Android phone. I learned this building a streaming chat against Gemini's SSE API, and the fix changed how I think about streaming UI in general.

The naive version

The obvious implementation looks like this: accumulate tokens into a string, hand the string to a markdown widget, rebuild. Twice a second this is fine. Thirty times a second, which is what a fast model actually does, it means re-parsing and re-laying-out the entire message tree on every frame. Long code blocks make it quadratic: each token re-parses everything before it.

Think in blocks, not strings

Markdown has natural block boundaries: paragraphs, code fences, lists, headings. The key insight is that only the last block can change while streaming. Everything above it is settled text.

So the renderer works like this:

• Parse incoming text into a list of sealed block models (ParagraphBlock, CodeBlock, ListBlock, …)
• Give every completed block a stable ValueKey and wrap it in a RepaintBoundary
• Only the trailing block re-parses and repaints as tokens arrive

Flutter's diffing does the rest: identical keys mean untouched subtrees, and the repaint boundary stops the trailing block's churn from invalidating its siblings. The result is that a 2,000-line conversation streams as cheaply as a one-liner, because per token you only ever touch one block.

Streaming input is hostile input

The subtle half of the problem: mid-stream markdown is malformed by definition. You will receive an unclosed code fence, half a table row, a link whose closing paren hasn't arrived yet. A strict parser flickers between "this is code" and "this is text" as tokens land.

The renderer has to be tolerant by design:

• An unclosed code fence renders as a code block immediately, with a subtle pulse indicator instead of waiting for the closing fence
• Partial emphasis (**bold with no close) renders as literal text until proven otherwise
• Never throw. Worst case, degrade a block to plain text

Cancellation is a feature

Users stop generations. Apps get backgrounded. Networks drop. Each of those should resolve to a distinct, honest message state: "stopped by you" keeps the partial text, a network failure offers inline retry, backgrounding cancels the request via the Dio CancelToken and keeps what arrived.

The pattern that made this clean: every stream gets its own token, the controller owns lifecycle transitions, and partial content is always preserved. An LLM answer interrupted at 80% is usually still useful; throwing it away because the stream errored is user-hostile.

Streaming UX is where LLM apps are won or lost. The model is a commodity; the 60fps render of its half-finished thought is not.