opendunes
Evaluate and optimise
Copy page ▾

Distillation

Collect a large model's answers to train a smaller one, on models whose makers allow it.

  1. Pick a teacher that allows it.
  2. Hold every request to that rule.
  3. Collect the pairs.
  4. Train and compare.

OpenDunes sends your prompts and returns the answers. The training runs in your own tools: OpenDunes runs no training job and serves no model you trained.

1. Pick a teacher that allows it

The teacher is the model whose answers you collect. Some model makers forbid training another model on their answers, and some allow it. Among the labs whose makers allow it, read 29 September 2026: Qwen, Mistral, DeepSeek, NVIDIA, Moonshot AI, Meta and Microsoft.

TeacherModel IDModel IDThe name a request calls a model by, lab/model, copied from Models.More on Models →
DeepSeek V4 Prodeepseek/deepseek-v4-pro
Qwen3.7 Maxqwen/qwen3.7-max
Kimi K3moonshotai/kimi-k3
Mistral Medium 3.5mistralai/mistral-medium-3-5
Nemotron 3 Ultranvidia/nemotron-3-ultra-550b-a55b

Claude, GPT and Gemini models are not among them. Copy a model ID from its page in Models: see Models.

Watch out

The allowance is the maker's word, as the catalogue records it. Read the model's licence for your own use before you train on its answers.

2. Hold every request to that rule

Send provider.enforce_distillable_text: true with each request. OpenDunes then sends it only to a model whose maker allows training on its answers; a model that does not is refused, and the body says why.

JSON
{
  "model": "deepseek/deepseek-v4-pro",
  "provider": {
    "enforce_distillable_text": true
  },
  "messages": [
    {
      "role": "system",
      "content": "You answer customers of a clothing shop in Oran."
    },
    {
      "role": "user",
      "content": "Le burnous existe en quelle taille ?"
    }
  ]
}

The OpenAI library sends provider as an extra body field: extra_body in Python. Every field of the route: Chat.

Don't

Train on answers you collected without the rule, then check the maker's terms later.

Do

Send the rule with every request, so a teacher that forbids it never answers into your dataset.

3. Collect the pairs

Send each prompt of your training set to the teacher, and save the prompt with its answer, one pair per line. Illustrative: the fields your training tool reads may differ.

JSON
{
  "messages": [
    {
      "role": "user",
      "content": "Le burnous existe en quelle taille ?"
    },
    {
      "role": "assistant",
      "content": "Taille unique, il s'ajuste avec sa ceinture."
    }
  ]
}
KeepWhy
prompts like the ones your users sendthe student learns the work it will do
only answers you checkeda wrong answer teaches the wrong thing
usage.cost of each answerthe dataset's cost in DADAThe Algerian dinar, the one currency the docs show.More on Balance and charges →, summed

The script asks the teacher each prompt with the rule from step 2, and saves each pair as a line of pairs.jsonl. A teacher that refuses prints its status, and nothing is saved for that prompt. It also asks MODEL, the smaller model you would start from, with no rule: its answers go to baseline.jsonl, for the comparison in step 4 and for nothing else. It sums usage.cost for each file, in DA.

SeatHoldsWhy
TEACHERa model from the table in step 1its answers are the pairs
MODELthe smaller model you would start fromits answers are the baseline, never training data
import json
import os
from openai import APIStatusError, OpenAI

API_KEY = os.environ["OPENDUNES_API_KEY"]
BASE_URL = "https://opendunes.com/v1"
MODEL = "anthropic/claude-sonnet-4.5"
TEACHER = "deepseek/deepseek-v4-pro"
BRIEF = "You answer customers of a clothing shop in Oran."
PROMPTS = [
    "Le burnous existe en quelle taille ?",
    "Livrez-vous à Constantine ?",
    "Puis-je échanger un burnous après l'achat ?",
]
RULE = {"provider": {"enforce_distillable_text": True}}
PAIRS = "pairs.jsonl"
BASELINE = "baseline.jsonl"

client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
spent = {"teacher": 0.0, "baseline": 0.0}


def ask(model, prompt, fields):
    reply = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": BRIEF},
            {"role": "user", "content": prompt},
        ],
        extra_body=fields,
    )
    return reply.choices[0].message.content, reply.usage.cost


def save(path, prompt, answer):
    pair = {
        "messages": [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": answer},
        ]
    }
    with open(path, "a", encoding="utf-8") as out:
        out.write(json.dumps(pair, ensure_ascii=False) + "\n")


for prompt in PROMPTS:
    try:
        answer, cost = ask(TEACHER, prompt, RULE)
    except APIStatusError as error:
        print("teacher refused, status", error.status_code)
        continue
    save(PAIRS, prompt, answer)
    spent["teacher"] += cost
    answer, cost = ask(MODEL, prompt, {})
    save(BASELINE, prompt, answer)
    spent["baseline"] += cost

print(spent)

A teacher whose maker does not allow training on its answers is refused, and the rule keeps it out of pairs.jsonl. Read each line of pairs.jsonl before you train: keep only the answers you checked, as the table above says. Run the script again and it adds to both files; delete them first to start over.

4. Train and compare

Train the smaller model in your own tools, on the pairs. Then hold back prompts it never saw, send them to the teacher and to the trained model, and compare the answers the way you judge your app.

Before you train at all, send those same prompts to a smaller model on OpenDunes as it is. Where it already holds, it is the cheaper answer, with no training.

Last modified on