Benchmark scores
GET /v1/benchmarks answers benchmark scores, one row per model and benchmark, from three
sources.
Request
Send it with an API keyAPI keyThe key you create and send with every request that calls a model.More on API keys →. Every field goes in the address, after ?, and each one narrows the
list; with none, every row of every source comes back. The models guide shows
how to pick a model and read its ID.
Yacine wants the model that writes the best code for Souk Bot's order tools. He asks for one
source's rows with source=artificial-analysis and reads each model's coding_index.
| Field | Needed | You send |
|---|---|---|
source | no | artificial-analysis, design-arena or opendunes: only that source's rows, each in its own shape |
task_type | no | coding, intelligence, agentic or search: the rows scored on that kind of task |
benchmark_type | no | one benchmark of the opendunes source: gpqa_diamond, tau_bench_verified_airline, search_browsecomp, search_hle, search_dsqa or search_widesearch |
max_results | no | the most rows to return, 1 or more; every matching row when left out |
arena | no | on design-arena: models, builders or agents; models when left out |
category | no | on design-arena: one category of the arena, such as uicomponent, gamedev or dataviz; every category when left out |
search_engine | no | on a search benchmark: only the runs that used this search engine |
search_surface | no | on a search benchmark: server-tool for the search helperHelperA tool run for you when a model reaches for it mid-answer: a web search, a page read, a sandbox. Named opendunes:‹name›.More on Helpers →, plugin for the web plugin |
include_run_config | no | true to add each search row's run settings, run_config; false when left out |
import json
import os
import urllib.request
API_KEY = os.environ["OPENDUNES_API_KEY"]
SOURCE = "artificial-analysis"
ADDRESS = "https://opendunes.com/v1/benchmarks?source=" + SOURCE
request = urllib.request.Request(
ADDRESS,
headers={"Authorization": "Bearer " + API_KEY},
)
with urllib.request.urlopen(request) as response:
scores = json.load(response)["data"]
for score in scores:
print(score["model_permaslug"], score["coding_index"])Response
200 OK. data is the list of rows; source in each says which source scored it and which
fields it carries.
| Field | Holds |
|---|---|
data | the rows, one per model and benchmark |
source | in each: artificial-analysis, design-arena or OpenDunes |
model_permaslug | in each: the model's ID fixed to one release; on a design-arena row that maps to no model, that source's own name for it |
display_name | in each: the model's name, as the source writes it |
pricing | in each: the model's prompt and completion prices in DADAThe Algerian dinar, the one currency the docs show.More on Balance and charges →, each a string per token, or null |
An artificial-analysis row adds:
| Field | Holds |
|---|---|
intelligence_index · coding_index · agentic_index | the model's score on each index, higher is better, or null |
A design-arena row adds:
| Field | Holds |
|---|---|
arena · category | the arena and the category the rating belongs to |
elo | the rating from head-to-head matches |
win_rate | the share of matches it won, in percent |
tournament_stats | how often it placed first to fourth, and the matches played |
avg_generation_time_ms | how long it took to answer, on average, in milliseconds, or null |
An OpenDunes row adds:
| Field | Holds |
|---|---|
benchmark_type | the benchmark, one of the six benchmark_type takes |
accuracy · accuracy_stddev | on gpqa_diamond and tau_bench_verified_airline: the share of tasks answered right, from 0 to 1, and its standard deviation across runs, or null for one run |
primary_score · primary_metric | on a search benchmark: its headline score, from 0 to 1, and what it measures: accuracy, or f1_by_item on search_widesearch |
avg_cost_per_task | what one task cost on average, in DA, or null |
avg_latency_per_task_ms | on a search benchmark: how long one task took on average, in milliseconds, or null |
search_engine · search_surface | on a search benchmark: the engine and the way the search ran |
run_config | on a search benchmark, when you sent include_run_config=true: its max_agent_turns, reasoning_effort and temperature |
total_tasks | the tasks counted across its runs |
last_run_timestamp | when it last ran |
What it cost
Free. Reading the catalogue is never charged.
Errors
error.code carries the status, and error.message says why.
| Status | When |
|---|---|
| 400 | a field is refused, such as a source that is not one of the three; error.metadata.errors names each field and the values it takes |
| 401 | the key is missing, wrong, disabled or expired |
| 403 | the key answers only from the addresses set on it, and this request came from another |
| 503 | the scores could not be read just now; Retry-After says how many seconds to wait |
Last modified on