Caching
Two caches, and what each saves.
1. Pick the cache
Every request sends its whole prompt again. The two caches save on that in two different ways.
| Cache | Keeps | Saves | On |
|---|---|---|---|
| the prompt cache | the start of a prompt that repeats | the repeated tokens count at the model's cache-read rate, lower than its prompt rate | the text routes, where the model offers it |
| the response cache | the whole answer to one request | an identical request is answered from the copy, and charges nothing | the text routes and embeddings, when you turn it on |
| If | Then |
|---|---|
| each request asks something new, after the same long instructions | the prompt cache: the model reads the instructions from its cache and answers fresh |
| the same request, byte for byte, is sent again and again | the response cache: the kept answer comes back, word for word |
| each answer must be written fresh, even to the same request | the prompt cache only |
2. Cache the start of the prompt
Souk Bot sends the same long brief with every customer message. The brief never changes, so the model can keep it: the first request writes it to the prompt cache, and every request after reads it from there.
How a model caches depends on its lab.
| Models | You mark | The first request |
|---|---|---|
| Anthropic, Gemini | the end of the part that never changes, with "cache_control": {"type": "ephemeral"} on its content part | writes the cache, at the model's cache-write rate, above its prompt rate |
| OpenAI, DeepSeek, Grok | nothing: a repeated start of a prompt is read from the cache on its own | writes the cache, charged only where the model's page lists a cache-write rate |
Put what never changes first, and mark the end of it. Send the request twice: the second reads the brief from the cache.
import json
import os
import urllib.request
API_KEY = os.environ["OPENDUNES_API_KEY"]
ADDRESS = "https://opendunes.com/v1/chat/completions"
MODEL = "anthropic/claude-sonnet-4.5"
# Stands in for your long brief: the same, word for word, every time.
LINE = "Souk Bot answers customers of a clothing shop in Oran. "
BRIEF = LINE * 200
def ask(question):
system = {
"type": "text",
"text": BRIEF,
"cache_control": {"type": "ephemeral"},
}
body = {
"model": MODEL,
"messages": [
{"role": "system", "content": [system]},
{"role": "user", "content": question},
],
}
request = urllib.request.Request(
ADDRESS,
data=json.dumps(body).encode(),
headers={
"Authorization": "Bearer " + API_KEY,
"Content-Type": "application/json",
},
)
with urllib.request.urlopen(request) as response:
return json.load(response)
for question in ["Do you deliver to Tlemcen?", "Can I return a burnous?"]:
usage = ask(question)["usage"]
details = usage.get("prompt_tokens_details") or {}
written = details.get("cache_write_tokens")
read = details.get("cached_tokens")
print(written, read, usage["cost"])Don't put what changes, a date or the customer's name, before the part that repeats. The cache holds the start of a prompt, so one changed word early on makes everything after it new.
Put the part that never changes first, mark its end, and put what changes after it.
| You send | Holds |
|---|---|
cache_control | on a content part: {"type": "ephemeral"} marks where the cached part ends |
prompt_cache_key | a name that sends requests sharing a prompt to the same cache |
A prompt shorter than the model's minimum is not cached: 1 024 tokens on anthropic/claude-sonnet-4.5. The samples repeat one line of the brief to pass it.
What the answer reports
usage reports the cached share on every answer. Each text route spells it its own way.
| Route | Read from the cache | Written to the cache |
|---|---|---|
/v1/chat/completions | prompt_tokens_details.cached_tokens | prompt_tokens_details.cache_write_tokens |
/v1/messages | cache_read_input_tokens | cache_creation_input_tokens |
/v1/responses | input_tokens_details.cached_tokens | input_tokens_details.cache_write_tokens |
The tokens read from the cache count at the model's cache-read rate, never its full prompt rate. The tokens written count at its cache-write rate, where the model publishes one. The model's page in Models gives each rate, and usage.cost carries the charge.
On a model that charges the write, one request costs more with the mark than without it. The saving starts on the second request that reads the same part, while the cache lives: 5 minutes by default on Anthropic models. A request after that writes it again.
3. Cache the whole answer
The response cache keeps a whole answer. An identical request, sent again, gets the kept copy back and charges nothing: no model ran. The three ways to turn it on, and how to clear a copy: Response cache.
It is off until the request sends the header X-OpenDunes-Cache: true. Souk Bot opens every chat with the same welcome line: the first request runs the model, and the second comes from the copy.
A preset can turn it on in place of the header.
import json
import os
import urllib.request
API_KEY = os.environ["OPENDUNES_API_KEY"]
ADDRESS = "https://opendunes.com/v1/chat/completions"
MODEL = "anthropic/claude-sonnet-4.5"
PROMPT = "One line to welcome a visitor to a clothing shop in Oran."
body = json.dumps(
{"model": MODEL, "messages": [{"role": "user", "content": PROMPT}]}
).encode()
for _ in range(2):
request = urllib.request.Request(
ADDRESS,
data=body,
headers={
"Authorization": "Bearer " + API_KEY,
"Content-Type": "application/json",
"X-OpenDunes-Cache": "true",
"X-OpenDunes-Cache-TTL": "3600",
},
)
with urllib.request.urlopen(request) as response:
answer = json.load(response)
status = response.headers.get("X-OpenDunes-Cache-Status")
print(status, answer["usage"]["cost"])The first request prints MISS, the second HIT.
| You send | Holds |
|---|---|
X-OpenDunes-Cache | true to turn the cache on for this request, false to turn it off |
X-OpenDunes-Cache-TTL | seconds to keep the copy, from 1 to 86400; 300 when it is not sent |
X-OpenDunes-Cache-Clear | true: answer fresh, and replace the kept copy |
| Header back | On | Holds |
|---|---|---|
X-OpenDunes-Cache-Status | every answer with the cache on | HIT or MISS |
X-OpenDunes-Cache-Age | a hit | seconds since the copy was kept |
X-OpenDunes-Cache-TTL | every answer with the cache on | seconds left on a hit; the full span on a miss |
X-OpenDunes-Cache-Source-Id | a hit | the receipt of the request that filled the copy: Receipt |
Every header: Headers.
What counts as identical
| Rule | |
|---|---|
| Same key, model, route, streaming choice and body. | one character of the body changed, a space or the order of two fields, is a new request |
A copy comes back word for word, whatever the temperature. | — |
| One key's copies are never another key's. | — |
| Only a 200 is kept. | an error is never answered from a copy |
| These routes cache. | /v1/chat/completions, /v1/messages, /v1/embeddings |
/v1/responses caches too. | — |
| Two identical requests at the same instant both run. | both are charged |
Don't turn the response cache on for a request that must be written fresh each time, such as a new product description on every call. Every hit returns the same words.
Send X-OpenDunes-Cache-Clear: true when one answer must be fresh: the model answers again, and its answer replaces the kept copy.
Errors
The prompt cache adds no status of its own. The response cache adds one.
| Status | When |
|---|---|
| 400 | X-OpenDunes-Cache-TTL is not a whole number of seconds from 1 to 86400 |
Every other status is the same with the cache on or off: Errors and retries.
Last modified on