Streaming

Enable streaming #

Set stream: true. The response uses text/event-stream. With curl, -N disables output buffering. Request stream_options.include_usage to receive usage when available.

curl --fail-with-body -N https://formalengines.com/inference/v1/chat/completions \
  -H "Authorization: Bearer $FE_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-1-flash","messages":[{"role":"user","content":"Explain streaming in one sentence."}],"max_tokens":256,"stream":true,"stream_options":{"include_usage":true}}'

Consume chunks #

stream = client.chat.completions.create(
    model="deepseek-v4-1-flash",
    messages=[{"role": "user", "content": "Explain streaming in one sentence."}],
    max_tokens=256,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    for choice in chunk.choices:
        if choice.delta.content:
            print(choice.delta.content, end="", flush=True)
    if chunk.usage:
        print("\nUsage:", chunk.usage)

Text arrives in choices[].delta.content. A usage-only event can have an empty choices array. Tool arguments and reasoning can arrive in separate deltas, so do not treat every chunk as answer text. The OpenAI-style wire stream ends with data: [DONE].

Handle failures after headers #

HTTP 200 only confirms that streaming began. An error can arrive in a later event, or the connection can close before completion. Check for error events and unexpected disconnects. Do not mark a request successful solely because the initial status was 200.

Before retrying, decide whether replaying the prompt is safe. Keep tool execution separate from parsing model output so a retry cannot accidentally repeat an external action. Use the billing usage endpoint to inspect the settled request when necessary.