Here is a bill you are probably paying without seeing it. A user asks your agent a question, the tokens start streaming, and three seconds in they close the tab. On many stacks, the model does not notice. It keeps generating — sentence after sentence of an answer that will never be rendered — all the way to max_tokens, holding a GPU slot the entire time. You paid for every one of those tokens, and while they were being produced, someone else's request sat in the queue a little longer.

Abandoned generations are the quiet leak in LLM serving. They don't throw errors, they don't show up in your success metrics, and at low traffic you'll never spot them. At real concurrency they become a tax on both your GPU bill and your tail latency. The fix isn't complicated, but it's cooperative — it only works if every layer plays its part, and it breaks silently when one doesn't.

The mechanism: two halves that both have to fire#

Cancelling a request on disconnect is a chain, and the chain has two load-bearing links.

First, the web layer has to notice the client is gone. In a Starlette/FastAPI app, when the client disconnects mid-stream, the framework cancels the task running your StreamingResponse generator — your async generator gets a CancelledError. Alternatively you poll await request.is_disconnected() between tokens. Either way, something has to observe the closed socket.

Second, that cancellation has to propagate into the inference engine. Noticing the disconnect is useless if the engine keeps churning. The web layer must call the engine's abort — await engine.abort(request_id) in vLLM — which stops decoding for that sequence and, crucially, frees its KV-cache slot back to the scheduler.

Detection without propagation is a no-op; propagation without detection never fires. You need both halves, and the failure modes almost always come from one of them going quietly missing.

Miss the first link and the server never learns you left. Miss the second and it knows but doesn't act. In both cases the symptom is identical: generation runs to completion, tokens into the void.

The silent break almost everyone hits: middleware#

The single most common way a working setup regresses is instructive because it has nothing to do with your streaming code. If you add a BaseHTTPMiddleware to a Starlette app — for logging, auth, metrics, anything — request.is_disconnected() stops reporting disconnects. It returns False even after the client is long gone. This is a Starlette-level interaction, not a bug in your handler, which is exactly why it's so easy to ship: your abort logic is untouched and still correct, but the signal it depends on has gone dark.

The lesson is to treat disconnect handling as something you verify, not something you assume. A server that aborted correctly last week can leak requests this week because someone added an unrelated middleware. If your framework's disconnect check is the thing feeding your abort, that check is now part of your critical path and deserves a test.

Don't forget the client half#

The server can only detect a disconnect if the client actually closes the connection — and connections don't always close just because you stopped reading. You have to cancel actively:

# httpx — break out of the streaming context to close the socket
import httpx

with httpx.stream("POST", url, json=payload) as resp:
    for chunk in resp.iter_lines():
        if user_navigated_away():
            break            # exiting the `with` closes the connection
// fetch — an AbortController is the only reliable cancel
const controller = new AbortController();
const resp = await fetch(url, { method: "POST", body, signal: controller.signal });
// when the user closes the view:
controller.abort();          // tears down the connection; server sees the disconnect

With the OpenAI Python SDK, exit the with client.chat.completions.stream(...) block early (or call .close()); that closes the underlying HTTP stream the same way. If you rely on garbage collection to eventually reap the socket, the server may keep generating for seconds after the user is gone — the GC's timeline is not your cost model's timeline.

Verify it, deterministically#

Don't trust that this works — reproduce it. Start a long generation and kill the client mid-stream:

# start a long stream, then hit Ctrl-C a second in
curl -N -X POST http://localhost:8000/v1/completions \
  -H 'content-type: application/json' \
  -d '{"model":"Qwen/Qwen3-8B","prompt":"Write a long essay.","max_tokens":2000,"stream":true}'

On a correctly wired server, Ctrl-C on that curl produces an abort in the server logs and the running-sequence count drops by one. On a broken one, the sequence keeps running to max_tokens regardless of the dead client. Run the test twice — once with your full middleware/proxy stack and once without — and you'll immediately isolate whether the leak is in detection (middleware/proxy) or propagation (your handler). This is the same discipline that makes load-testing an LLM app meaningful: measure the failure mode, don't reason about it.

Why it's worth the attention#

An abandoned stream is the rare defect that is both a cost problem and a latency problem. Cost, because you burn GPU cycles on output no one reads. Latency, because the leaked request keeps occupying a concurrency slot, so every request still in the queue waits behind work that's already pointless. At scale, a steady trickle of tab-closes turns into a standing reduction in effective throughput and a fatter p99 — and because nothing errors, it hides.

It's the mirror image of sleep mode's idle-VRAM problem: there, a paused model wastes memory; here, a phantom request wastes compute. And it rhymes with the same principle behind cancelling a running agent — the ability to stop is not a nice-to-have on expensive, long-running work, it's a core part of the control loop. Wire the disconnect through to the abort, test that it fires, and your GPU spends its cycles only on answers someone is still waiting to read.