Streaming

Overview

Streaming delivers model output in small events instead of waiting for the full answer. Use Responses streaming for a model that lists responses in accepted_request_formats; keep Chat Completions streaming when that is what your SDK or application already expects.

Responses streaming

curl https://api.tokenlab.sh/v1/responses \
  -H "Authorization: Bearer sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-terra",
    "input": "Write a short poem.",
    "stream": true
  }'

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-api-key",
    base_url="https://api.tokenlab.sh/v1"
)

stream = client.responses.create(
    model="gpt-5.6-terra",
    input="Write a short poem.",
    stream=True
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'sk-your-api-key',
  baseURL: 'https://api.tokenlab.sh/v1'
});

const stream = await client.responses.create({
  model: 'gpt-5.6-terra',
  input: 'Write a short poem.',
  stream: true
});

for await (const event of stream) {
  if (event.type === 'response.output_text.delta') {
    process.stdout.write(event.delta);
  }
}

Responses SSE keeps the documented event names, order, and fields. After the first event arrives, an interrupted stream is incomplete and is not restarted automatically.

Responses WebSocket

Connect to wss://api.tokenlab.sh/v1/responses and send response.create events. WebSocket responses are always streamed and do not support background or response.cancel. A response.create with generate: false returns an ID that can be continued without generating model output or creating a model charge. Each connection handles one active response at a time and lasts up to 60 minutes. Events include a response-scoped sequence_number. Stream events named error use a flat event object; connection and protocol failures use a nested error object.

Chat Completions Streaming

If your framework still expects SSE chunks from /v1/chat/completions, that also works:

stream = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[{"role": "user", "content": "Write a short poem"}],
    stream=True
)

for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

Gemini Streaming

POST /v1beta/models/{model}:streamGenerateContent?alt=sse returns Gemini chunks. An event can contain only metadata, intermediate events may omit finishReason, and the stream can end naturally without a Chat Completions [DONE] marker.

Stream End Conditions

Typical completion conditions:

response.completed for Responses API streams
finish_reason: "stop" for Chat Completions streams
finish_reason: "length" when a token limit is hit
tool/function call events when the model wants to use tools

Web App Pattern

async function streamChat(message) {
  const response = await fetch('https://api.tokenlab.sh/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer sk-your-api-key',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: 'gpt-5.6-terra',
      messages: [{ role: 'user', content: message }],
      stream: true
    })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split('\\n').filter(line => line.startsWith('data: '));

    for (const line of lines) {
      const data = line.slice(6);
      if (data === '[DONE]') return;
      const parsed = JSON.parse(data);
      const content = parsed.choices?.[0]?.delta?.content;
      if (content) {
        document.getElementById('output').textContent += content;
      }
    }
  }
}

Handle streams well

Match the API your app already understands

Use Responses streaming when the selected model and your client support it. Existing Chat Completions streaming code can stay on /v1/chat/completions.

Flush output incrementally

Append delta chunks to the UI or terminal as they arrive rather than waiting for the full response.

Treat disconnects as incomplete responses

Preserve the partial output if it is useful, but do not mark it as complete. Repeating a request may produce different output and another charge.

Getting Started

Core Guides

Media Guides

Overview

Responses streaming

Responses WebSocket

Chat Completions Streaming

Gemini Streaming

Stream End Conditions

Web App Pattern

Handle streams well

​Overview

​Responses streaming

​Responses WebSocket

​Chat Completions Streaming

​Gemini Streaming

​Stream End Conditions

​Web App Pattern

​Handle streams well

Overview

Responses streaming

Responses WebSocket

Chat Completions Streaming

Gemini Streaming

Stream End Conditions

Web App Pattern

Handle streams well