Short version: The moment between a user hitting send and the first token appearing is where your product feels fast or broken. Streaming closes it — first token in under a second instead of a 20-second spinner. Server-Sent Events is the right transport: one-directional, plain HTTP, auto-reconnecting, and the exact shape of token streaming. Here's the whole path, backend to browser — and the single buffering bug that makes a correct implementation look dead.

Why SSE, not WebSocket#

Token streaming has one direction: the server generates tokens and pushes them; the client renders them. That's precisely what SSE does. It runs over ordinary HTTP, needs no protocol upgrade, and the browser's EventSource reconnects on its own. WebSocket is a full-duplex channel — the right call when the client must talk back on the same connection mid-stream, but for chat completions it's heavier infrastructure (sticky sessions, upgrade handling, your own reconnect) buying a capability you don't use.

The clinching detail: the OpenAI and Anthropic streaming APIs already deliver their tokens over SSE. Forwarding them to the browser as SSE is just passing the format through.

The backend: forward the provider's stream#

Set the SSE headers, consume the model's stream, and write one frame per delta — flushing each time:

app.post("/api/chat", async (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
    "X-Accel-Buffering": "no",            // <-- stops nginx buffering the stream
  });

  const ac = new AbortController();
  res.on("close", () => ac.abort());        // client left → stop the upstream bill

  const upstream = await model.messages.stream(
    { messages: req.body.messages },
    { signal: ac.signal }
  );

  for await (const delta of upstream) {
    if (delta.type !== "content_block_delta") continue;
    res.write(`data: ${JSON.stringify({ text: delta.delta.text })}\n\n`);
  }
  res.write("data: [DONE]\n\n");            // clean-end sentinel
  res.end();
});

Two lines carry more weight than they look. X-Accel-Buffering: no is the difference between a live stream and one chunk at the end (more on that below). The res.on("close") → abort() is how you stop generating — and paying for — a response after the user navigates away.

The frontend: fetch + a reader (not EventSource)#

EventSource is elegant but GET-only and takes no body, so a chat POST can't use it. The standard move is fetch() plus a ReadableStream reader, parsing the same SSE frames:

const res = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ messages }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += decoder.decode(value, { stream: true });
  const frames = buf.split("\n\n");
  buf = frames.pop();                        // keep the partial frame
  for (const f of frames) {
    const line = f.replace(/^data: /, "");
    if (line === "[DONE]") return;
    render(JSON.parse(line).text);           // append token to the UI
  }
}

Note the buf.split("\n\n") with frames.pop(): TCP can split a read mid-frame, so you buffer until you have a complete \n\n-terminated event. Skip that and you'll JSON.parse half a token and crash intermittently under load.

The bug that eats a day: buffering#

You write the code above, it works on localhost, you deploy, and the stream arrives all at once at the end. Your code is fine — something on the path is buffering the response and delivering it in one shot:

A correct SSE implementation that "doesn't stream" is almost never a code bug. It's a proxy holding your tokens hostage until the last one.

Add a heartbeat for long idle gaps — a comment line : ping\n\n every 15–30s keeps proxies from timing out an open-but-quiet connection. If a stream does drop, that's where EventSource's built-in reconnect (for GET streams) or your own retry earns its keep; the dropped-stream resume pattern covers picking back up without replaying tokens.

What you've got#

First token in under a second, tokens rendered as they generate, buffering disabled end-to-end, the upstream call aborted when the user leaves, and errors handled in-band because the status line is long gone. That's the whole contract for a chat UI that feels fast. When you move from raw text to streaming structured output, the framing gets trickier — parsing partial JSON as it arrives is its own problem, covered in streaming structured output from an LLM.