Latency & Performance

Latency has two faces: the wall-clock time until you receive the full response, and the time until your user sees the first token. These are different problems with different solutions.

Stream for Perceived Latency#

The single highest-impact change is enabling streaming. Set "stream": true and render tokens as they arrive — your UI feels instant even on a 5-second full-response.

curl https://opendunes.com/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENDUNES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{ "role": "user", "content": "Summarise the Maghreb region." }],
    "stream": true
  }'

The server sends Content-Type: text/event-stream. Each data: line is a JSON delta chunk; the final line is data: [DONE]. See Streaming for full details.

Pick a Faster Model#

Time-to-first-token varies by model size and provider. When your use case does not require a frontier model, smaller/faster models cut latency significantly and cost less DA per token.

A rough guide:

Browse the full catalog at /models. Sort by price or name to find faster alternatives.

Keep Prompts Tight#

Every input token adds latency. Avoid including boilerplate that the model does not need:

• Remove repeated instructions that could live in a short system prompt.
• Trim conversation history to the last N relevant turns rather than the full session.
• Prefer concise instructions over verbose explanations.

Cap Output Length#

If you know the response will be short, set max_tokens explicitly. This prevents the model from generating a long answer when a brief one suffices, and it allows the server to allocate resources more efficiently.

{
  "model": "google/gemini-2.5-pro",
  "messages": [{ "role": "user", "content": "One sentence: what is Tamazight?" }],
  "max_tokens": 60
}

Reuse HTTP Connections#

Each TCP handshake adds round-trip overhead. Use persistent connections (HTTP keep-alive) or a connection pool. The OpenAI SDK handles this automatically — if you are using raw fetch or requests, configure a session/agent with keep-alive.

from openai import OpenAI

# The client maintains a connection pool internally — create it once and reuse it.
client = OpenAI(
    base_url="https://opendunes.com/api/v1",
    api_key=os.environ["OPENDUNES_API_KEY"],
)

Reduce Round-Trips with Multi-Turn Threads#

Not yet available. The Responses API lets you continue a conversation by ID rather than resending the full messages array, cutting payload size on long threads.

Monitor Response Headers#

Each response includes rate limit and balance headers you can use to tune request pacing:

• X-RateLimit-Remaining-RPM — requests remaining in the current minute.
• X-RateLimit-Reset — Unix timestamp when the window resets.
• X-Balance-Available — remaining balance in micro-DA.

Back off proactively when X-RateLimit-Remaining-RPM is low, rather than waiting for a 429.