opendunes
Catalogue
Copy page ▾

Benchmark scores

GET /v1/benchmarks answers benchmark scores, one row per model and benchmark, from three sources.

Request

Send it with an API keyAPI keyThe key you create and send with every request that calls a model.More on API keys →. Every field goes in the address, after ?, and each one narrows the list; with none, every row of every source comes back. The models guide shows how to pick a model and read its ID.

Yacine wants the model that writes the best code for Souk Bot's order tools. He asks for one source's rows with source=artificial-analysis and reads each model's coding_index.

FieldNeededYou send
sourcenoartificial-analysis, design-arena or opendunes: only that source's rows, each in its own shape
task_typenocoding, intelligence, agentic or search: the rows scored on that kind of task
benchmark_typenoone benchmark of the opendunes source: gpqa_diamond, tau_bench_verified_airline, search_browsecomp, search_hle, search_dsqa or search_widesearch
max_resultsnothe most rows to return, 1 or more; every matching row when left out
arenanoon design-arena: models, builders or agents; models when left out
categorynoon design-arena: one category of the arena, such as uicomponent, gamedev or dataviz; every category when left out
search_enginenoon a search benchmark: only the runs that used this search engine
search_surfacenoon a search benchmark: server-tool for the search helperHelperA tool run for you when a model reaches for it mid-answer: a web search, a page read, a sandbox. Named opendunes:‹name›.More on Helpers →, plugin for the web plugin
include_run_confignotrue to add each search row's run settings, run_config; false when left out
import json
import os
import urllib.request

API_KEY = os.environ["OPENDUNES_API_KEY"]
SOURCE = "artificial-analysis"
ADDRESS = "https://opendunes.com/v1/benchmarks?source=" + SOURCE

request = urllib.request.Request(
    ADDRESS,
    headers={"Authorization": "Bearer " + API_KEY},
)
with urllib.request.urlopen(request) as response:
    scores = json.load(response)["data"]

for score in scores:
    print(score["model_permaslug"], score["coding_index"])

Response

200 OK. data is the list of rows; source in each says which source scored it and which fields it carries.

FieldHolds
datathe rows, one per model and benchmark
sourcein each: artificial-analysis, design-arena or OpenDunes
model_permaslugin each: the model's ID fixed to one release; on a design-arena row that maps to no model, that source's own name for it
display_namein each: the model's name, as the source writes it
pricingin each: the model's prompt and completion prices in DADAThe Algerian dinar, the one currency the docs show.More on Balance and charges →, each a string per token, or null

An artificial-analysis row adds:

FieldHolds
intelligence_index · coding_index · agentic_indexthe model's score on each index, higher is better, or null

A design-arena row adds:

FieldHolds
arena · categorythe arena and the category the rating belongs to
elothe rating from head-to-head matches
win_ratethe share of matches it won, in percent
tournament_statshow often it placed first to fourth, and the matches played
avg_generation_time_mshow long it took to answer, on average, in milliseconds, or null

An OpenDunes row adds:

FieldHolds
benchmark_typethe benchmark, one of the six benchmark_type takes
accuracy · accuracy_stddevon gpqa_diamond and tau_bench_verified_airline: the share of tasks answered right, from 0 to 1, and its standard deviation across runs, or null for one run
primary_score · primary_metricon a search benchmark: its headline score, from 0 to 1, and what it measures: accuracy, or f1_by_item on search_widesearch
avg_cost_per_taskwhat one task cost on average, in DA, or null
avg_latency_per_task_mson a search benchmark: how long one task took on average, in milliseconds, or null
search_engine · search_surfaceon a search benchmark: the engine and the way the search ran
run_configon a search benchmark, when you sent include_run_config=true: its max_agent_turns, reasoning_effort and temperature
total_tasksthe tasks counted across its runs
last_run_timestampwhen it last ran

What it cost

Free. Reading the catalogue is never charged.

Errors

error.code carries the status, and error.message says why.

StatusWhen
400a field is refused, such as a source that is not one of the three; error.metadata.errors names each field and the values it takes
401the key is missing, wrong, disabled or expired
403the key answers only from the addresses set on it, and this request came from another
503the scores could not be read just now; Retry-After says how many seconds to wait

Last modified on