How to Stream LLM Responses (OpenAI-Compatible & Anthropic)

Streaming does not make a model generate faster. It makes the wait feel shorter, because the first tokens land on the screen while the rest are still being produced. That distinction is the whole reason streaming exists — and it is the thing most tutorials skip past on their way to a two-line stream=True snippet. This guide covers both surfaces you are likely to hit: the OpenAI-compatible chat.completions chunk shape and Anthropic’s typed Messages event lifecycle, plus the three parts that actually break in production — tool-call JSON that arrives in fragments, errors that appear mid-stream, and token usage that is reported in a completely different place depending on which API you called.

Why stream at all

A non-streamed completion is a single HTTP response: you send the request, the server generates every token, and only then does anything come back. A streamed completion is Server-Sent Events: the same generation happens at the same speed, but each fragment is pushed down the open connection as it is produced.

The honest framing matters, because it is easy to oversell. Streaming does not reduce total generation time and does not reduce cost. Tokens per second are unchanged. What collapses is time to first token — the interval where a user is staring at a spinner with zero evidence that anything is happening. We go deeper on the full latency budget in how to reduce LLM latency; streaming is the one lever on that list that improves the felt experience without touching the actual work.

How much dead air are we talking about? For context, our own first-party coding benchmark (DataLLM Lab, 9 identical coding tasks) measured whole-completion time per task — not time to first token — and across five of the models we ran it looked like this: Claude Opus 4.8 around 6.1s, Qwen3 Coder Next around 7.0s, Kimi K2.7-Code around 10.4s, GLM-5.2 around 12.3s, and DeepSeek V4-Flash around 14.5s. Treat those as total generation times, not first-token times. The point is simply that real completions take seconds to tens of seconds, and a user who sees nothing for fourteen seconds assumes the app is broken.

Buffered vs streamed: the token flow

The OpenAI-compatible way

If you are calling an OpenAI-compatible gateway — including new-api deployments and DataLLM Lab’s own https://www.datallmlab.com/v1 — the surface you want is chat.completions. OpenAI’s newer native Responses API (client.responses.create) exists and emits differently typed events, but compatible gateways expose the chat.completions shape, so that is the pattern to build on. See what an OpenAI-compatible API actually guarantees for where the compatibility surface starts and stops.

Python:

from openai import OpenAI

client = OpenAI(base_url="https://www.datallmlab.com/v1", api_key=KEY)

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Explain SSE in two sentences."}],
    stream=True,
    stream_options={"include_usage": True},
)

usage = None
for chunk in response:
    if chunk.usage is not None:
        usage = chunk.usage          # final chunk: choices is []
    if not chunk.choices:
        continue                     # guard, or this line crashes
    piece = chunk.choices[0].delta.content
    if piece:
        print(piece, end="", flush=True)

Two things to internalise. First, the field is delta.content, not message.content — the non-streamed shape has message, the streamed shape has delta, and mixing them up is the single most common streaming bug. Second, delta.content can legitimately be None on some chunks (role-only openers, finish chunks), so always truth-test before concatenating.

JavaScript / TypeScript is the same shape over an async iterator instead of a sync loop:

const stream = await client.chat.completions.create({
  model: "gpt-5.5",
  messages: [{ role: "user", content: "Explain SSE in two sentences." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  if (chunk.usage) usage = chunk.usage;
  const piece = chunk.choices[0]?.delta?.content;
  if (piece) process.stdout.write(piece);
}

Note the optional chaining on choices[0]?.delta?.content — it is doing real work here, because the usage chunk has an empty choices array.

Anthropic streaming events

Anthropic’s native Messages API is not a flat chunk stream. Set stream: true on the Messages API, or use the SDK helper client.messages.stream(...) — a context manager in Python, a stream object in TypeScript — and you get a typed event lifecycle:

1. message_start — a Message object with empty content, carrying initial usage.
2. For each content block: content_block_start, then one or more content_block_delta, then content_block_stop.
3. One or more message_delta, then message_stop.

ping events and error events may be interspersed anywhere in that sequence, and Anthropic reserves the right to add event types, so your handler must ignore unknown types rather than throw on them.

The deltas themselves are typed too. A content_block_delta carries one of: text_delta (read .text), input_json_delta (read .partial_json, for tool inputs), thinking_delta (read .thinking) or signature_delta for extended thinking. So the OpenAI mental model of “every chunk is a bit of text” does not survive the port — you must branch on delta type.

If you do not want to hand-handle events at all, the SDKs will accumulate internally and hand you the assembled result: stream.get_final_message() in Python, stream.finalMessage() in TypeScript. That is the supported way to get complete content and total usage while still rendering deltas as they arrive. Only raw HTTP consumers need to accumulate by hand.

Cross-provider parity table

This is the mapping most tutorials never print, because they only cover one vendor. Cells marked as the simpler option are highlighted.

Source: OpenAI Cookbook “How to stream completions” and Anthropic’s “Streaming messages” docs, as of July 2026. Synthesis and highlighting are ours.

Tool calls and partial JSON

This is where streaming implementations quietly break. On Anthropic, a tool call’s arguments do not arrive as an object — they arrive as input_json_delta events whose partial_json field is a fragment of a JSON string. One event might deliver {"loc, the next ation": "San Fran. Only the final assembled value is guaranteed to be a complete object; tool_use.input in the finished message is always an object.

So: never call json.loads or JSON.parse on a fragment. It will throw, and it will throw non-deterministically depending on where the network happened to split the payload — which is exactly the kind of bug that passes in dev and fails under load. The correct pattern is to append fragments to a per-block string buffer and parse once, after content_block_stop for that block:

# Anthropic, Python — accumulate then parse
import json

buffers = {}
with client.messages.stream(model="claude-opus-4-...", messages=msgs, tools=tools) as stream:
    for event in stream:
        if event.type == "content_block_start":
            buffers[event.index] = ""
        elif event.type == "content_block_delta":
            if event.delta.type == "text_delta":
                print(event.delta.text, end="", flush=True)
            elif event.delta.type == "input_json_delta":
                buffers[event.index] += event.delta.partial_json   # do NOT parse here
        elif event.type == "content_block_stop":
            raw = buffers.get(event.index, "")
            if raw:
                args = json.loads(raw)     # safe: block is complete
    final = stream.get_final_message()

If you would rather not manage buffers at all, skip straight to get_final_message() / finalMessage() and stream only the text deltas to the UI. That gives you live text plus fully-parsed tool inputs with no partial-JSON handling of your own. Use the model id exactly as your gateway lists it — check the Claude Opus 4.8 model page for the current string.

Mid-stream errors and cancellation

A stream can fail after it has already started. Anthropic surfaces this as an error event in the stream — for example a payload of type error with error.type of overloaded_error, the streaming equivalent of an HTTP 529 you would have received on a non-streamed call. Your loop has to handle that case, because by then you have already sent bytes to the client and cannot simply return a 500.

Three defensive rules that hold across providers:

• Handle errors inside the loop, not only around it. A try/except wrapped around the whole iteration is fine for transport failures, but an error event is a normal value arriving on a healthy connection — you must branch on it.
• Ignore unknown event types. Anthropic explicitly warns that new event types may be added. A handler with an exhaustive match that raises on the default branch will break on a future release.
• Decide what partial output means to your app. If a stream dies at 80%, you have 80% of an answer on screen. Retrying from scratch duplicates cost; keeping the partial may be wrong for structured output. Pick one deliberately per endpoint.

Cancellation is simpler than it looks: an SSE stream is an ordinary HTTP connection, so cancelling is aborting the request. In JavaScript that is an AbortController signal; in Python it is breaking out of the iterator or exiting the stream context manager. Two consequences worth stating plainly. Text already received stays valid — a stop button gives the user a usable partial answer, which is a real product advantage of streaming. And output already generated before the abort is still billed, so cancelling saves the tokens not yet produced, not the ones already streamed. If token spend is the driver, cutting LLM API costs is a different lever than cancellation.

The token-usage gotchas

Accounting is where the two APIs diverge most, and where teams silently ship wrong numbers into their dashboards.

OpenAI-compatible. By default a stream returns no usage at all. You must pass stream_options={"include_usage": True}. Once you do, every content chunk carries usage as null, and a single final chunk arrives with an empty choices array and the usage object populated with whole-request token counts. Two failure modes follow directly: code that reads choices[0] on every chunk crashes on that last chunk, and code that never sets include_usage reports zero tokens forever.

Anthropic. Usage is split across the lifecycle. message_start.usage carries input_tokens plus a small initial output_tokens. message_delta.usage then carries the running output_tokens. The docs are explicit that these message_delta counts are cumulative — so if you sum them across events you will massively over-count your own spend. Take the last value, not the total. The reliable alternative is to let the SDK finish the stream and read usage off get_final_message().

If you are metering usage across several vendors, this is precisely the normalisation work a gateway absorbs on your behalf — see what an LLM gateway is for how the accounting layer fits together, and the best LLM APIs in 2026 for how the providers compare on the surfaces described here.

When not to stream

Streaming is not free complexity-wise: you inherit incremental parsing, mid-stream error paths, cancellation semantics and split usage accounting. Here is a decision rule that fits on one line each.

The compressed version: stream when a human is watching and the output is long; buffer when a machine is consuming it or the output is short. And whichever you choose, keep the claim honest — you are buying a better waiting experience, not a faster model.

FAQ