opendunes
Requests · 5 min
Copy page ▾

Caching

Two caches, and what each saves.

  1. Pick the cache.
  2. Cache the start of the prompt.
  3. Cache the whole answer.

1. Pick the cache

Every request sends its whole prompt again. The two caches save on that in two different ways.

CacheKeepsSavesOn
the prompt cachethe start of a prompt that repeatsthe repeated tokens count at the model's cache-read rate, lower than its prompt ratethe text routes, where the model offers it
the response cachethe whole answer to one requestan identical request is answered from the copy, and charges nothingthe text routes and embeddings, when you turn it on
IfThen
each request asks something new, after the same long instructionsthe prompt cache: the model reads the instructions from its cache and answers fresh
the same request, byte for byte, is sent again and againthe response cache: the kept answer comes back, word for word
each answer must be written fresh, even to the same requestthe prompt cache only

2. Cache the start of the prompt

Souk Bot sends the same long brief with every customer message. The brief never changes, so the model can keep it: the first request writes it to the prompt cache, and every request after reads it from there.

How a model caches depends on its lab.

ModelsYou markThe first request
Anthropic, Geminithe end of the part that never changes, with "cache_control": {"type": "ephemeral"} on its content partwrites the cache, at the model's cache-write rate, above its prompt rate
OpenAI, DeepSeek, Groknothing: a repeated start of a prompt is read from the cache on its ownwrites the cache, charged only where the model's page lists a cache-write rate

Put what never changes first, and mark the end of it. Send the request twice: the second reads the brief from the cache.

import json
import os
import urllib.request

API_KEY = os.environ["OPENDUNES_API_KEY"]
ADDRESS = "https://opendunes.com/v1/chat/completions"
MODEL = "anthropic/claude-sonnet-4.5"
# Stands in for your long brief: the same, word for word, every time.
LINE = "Souk Bot answers customers of a clothing shop in Oran. "
BRIEF = LINE * 200


def ask(question):
    system = {
        "type": "text",
        "text": BRIEF,
        "cache_control": {"type": "ephemeral"},
    }
    body = {
        "model": MODEL,
        "messages": [
            {"role": "system", "content": [system]},
            {"role": "user", "content": question},
        ],
    }
    request = urllib.request.Request(
        ADDRESS,
        data=json.dumps(body).encode(),
        headers={
            "Authorization": "Bearer " + API_KEY,
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(request) as response:
        return json.load(response)


for question in ["Do you deliver to Tlemcen?", "Can I return a burnous?"]:
    usage = ask(question)["usage"]
    details = usage.get("prompt_tokens_details") or {}
    written = details.get("cache_write_tokens")
    read = details.get("cached_tokens")
    print(written, read, usage["cost"])
Don't

Don't put what changes, a date or the customer's name, before the part that repeats. The cache holds the start of a prompt, so one changed word early on makes everything after it new.

Do

Put the part that never changes first, mark its end, and put what changes after it.

You sendHolds
cache_controlon a content part: {"type": "ephemeral"} marks where the cached part ends
prompt_cache_keya name that sends requests sharing a prompt to the same cache

Every field: Chat · Messages.

Good to know

A prompt shorter than the model's minimum is not cached: 1 024 tokens on anthropic/claude-sonnet-4.5. The samples repeat one line of the brief to pass it.

What the answer reports

usage reports the cached share on every answer. Each text route spells it its own way.

RouteRead from the cacheWritten to the cache
/v1/chat/completionsprompt_tokens_details.cached_tokensprompt_tokens_details.cache_write_tokens
/v1/messagescache_read_input_tokenscache_creation_input_tokens
/v1/responsesinput_tokens_details.cached_tokensinput_tokens_details.cache_write_tokens

The tokens read from the cache count at the model's cache-read rate, never its full prompt rate. The tokens written count at its cache-write rate, where the model publishes one. The model's page in Models gives each rate, and usage.cost carries the charge.

Watch out

On a model that charges the write, one request costs more with the mark than without it. The saving starts on the second request that reads the same part, while the cache lives: 5 minutes by default on Anthropic models. A request after that writes it again.

3. Cache the whole answer

The response cache keeps a whole answer. An identical request, sent again, gets the kept copy back and charges nothing: no model ran. The three ways to turn it on, and how to clear a copy: Response cache.

It is off until the request sends the header X-OpenDunes-Cache: true. Souk Bot opens every chat with the same welcome line: the first request runs the model, and the second comes from the copy.

A preset can turn it on in place of the header.

import json
import os
import urllib.request

API_KEY = os.environ["OPENDUNES_API_KEY"]
ADDRESS = "https://opendunes.com/v1/chat/completions"
MODEL = "anthropic/claude-sonnet-4.5"
PROMPT = "One line to welcome a visitor to a clothing shop in Oran."

body = json.dumps(
    {"model": MODEL, "messages": [{"role": "user", "content": PROMPT}]}
).encode()

for _ in range(2):
    request = urllib.request.Request(
        ADDRESS,
        data=body,
        headers={
            "Authorization": "Bearer " + API_KEY,
            "Content-Type": "application/json",
            "X-OpenDunes-Cache": "true",
            "X-OpenDunes-Cache-TTL": "3600",
        },
    )
    with urllib.request.urlopen(request) as response:
        answer = json.load(response)
        status = response.headers.get("X-OpenDunes-Cache-Status")
        print(status, answer["usage"]["cost"])

The first request prints MISS, the second HIT.

You sendHolds
X-OpenDunes-Cachetrue to turn the cache on for this request, false to turn it off
X-OpenDunes-Cache-TTLseconds to keep the copy, from 1 to 86400; 300 when it is not sent
X-OpenDunes-Cache-Cleartrue: answer fresh, and replace the kept copy
Header backOnHolds
X-OpenDunes-Cache-Statusevery answer with the cache onHIT or MISS
X-OpenDunes-Cache-Agea hitseconds since the copy was kept
X-OpenDunes-Cache-TTLevery answer with the cache onseconds left on a hit; the full span on a miss
X-OpenDunes-Cache-Source-Ida hitthe receipt of the request that filled the copy: Receipt

Every header: Headers.

What counts as identical

Rule
Same key, model, route, streaming choice and body.one character of the body changed, a space or the order of two fields, is a new request
A copy comes back word for word, whatever the temperature.—
One key's copies are never another key's.—
Only a 200 is kept.an error is never answered from a copy
These routes cache./v1/chat/completions, /v1/messages, /v1/embeddings
/v1/responses caches too.—
Two identical requests at the same instant both run.both are charged
Don't

Don't turn the response cache on for a request that must be written fresh each time, such as a new product description on every call. Every hit returns the same words.

Do

Send X-OpenDunes-Cache-Clear: true when one answer must be fresh: the model answers again, and its answer replaces the kept copy.

Errors

The prompt cache adds no status of its own. The response cache adds one.

StatusWhen
400X-OpenDunes-Cache-TTL is not a whole number of seconds from 1 to 86400

Every other status is the same with the cache on or off: Errors and retries.

Last modified on