Latency & Performance

Latency has two faces: the wall-clock time until you receive the full response, and the time until your user sees the first token. These are different problems with different solutions.

Stream for Perceived Latency#

The single highest-impact change is enabling streaming. Set "stream": true and render tokens as they arrive — your UI feels instant even on a 5-second full-response.

curl https://opendunes.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENDUNES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{ "role": "user", "content": "Summarise the Maghreb region." }],
    "stream": true
  }'

The server sends Content-Type: text/event-stream. Each data: line is a JSON delta chunk; the final line is data: [DONE]. See Streaming for full details.

Pick a Faster Model#

Time-to-first-token varies by model size and provider. When your use case does not require a frontier model, smaller/faster models cut latency significantly and cost less DA per token.

A rough guide:

Browse the full catalog at /models. Sort by price or name to find faster alternatives.

Keep Prompts Tight#

Every input token adds latency. Avoid including boilerplate that the model does not need:

• Remove repeated instructions that could live in a short system prompt.
• Trim conversation history to the last N relevant turns rather than the full session.
• Prefer concise instructions over verbose explanations.

Cap Output Length#

If you know the response will be short, set max_tokens explicitly. This prevents the model from generating a long answer when a brief one suffices, and it allows the server to allocate resources more efficiently.

{
  "model": "google/gemini-2.5-pro",
  "messages": [{ "role": "user", "content": "One sentence: what is Tamazight?" }],
  "max_tokens": 60
}

Reuse HTTP Connections#

Each TCP handshake adds round-trip overhead. Use persistent connections (HTTP keep-alive) or a connection pool. The OpenAI SDK handles this automatically — if you are using raw fetch or requests, configure a session/agent with keep-alive.

from openai import OpenAI

# The client maintains a connection pool internally — create it once and reuse it.
client = OpenAI(
    base_url="https://opendunes.com/v1",
    api_key=os.environ["OPENDUNES_API_KEY"],
)

Keep Multi-Turn Payloads Small#

Every route is stateless — chat completions, Responses and Messages alike: each request carries the whole conversation, and a conversation cannot be continued by an ID. On long threads, trim or summarise older turns rather than resending them all; every turn bills its whole input.

Monitor Response Headers#

There is no cap on the number of requests, so there is no request counter to pace against. Two headers help:

• X-Balance-Available — remaining balance in micro-DA, on an answer that is not streamed.
• Retry-After — on a 429 from the provider serving the model, the seconds to wait. The provider's X-RateLimit-* headers ride with it when it sent them.

Wait the Retry-After seconds on a 429 rather than retrying at once.