Distillation
Collect a large model's answers to train a smaller one, on models whose makers allow it.
- Pick a teacher that allows it.
- Hold every request to that rule.
- Collect the pairs.
- Train and compare.
OpenDunes sends your prompts and returns the answers. The training runs in your own tools: OpenDunes runs no training job and serves no model you trained.
1. Pick a teacher that allows it
The teacher is the model whose answers you collect. Some model makers forbid training another model on their answers, and some allow it. Among the labs whose makers allow it, read 29 September 2026: Qwen, Mistral, DeepSeek, NVIDIA, Moonshot AI, Meta and Microsoft.
| Teacher | Model IDModel IDThe name a request calls a model by, lab/model, copied from Models.More on Models → |
|---|---|
| DeepSeek V4 Pro | deepseek/deepseek-v4-pro |
| Qwen3.7 Max | qwen/qwen3.7-max |
| Kimi K3 | moonshotai/kimi-k3 |
| Mistral Medium 3.5 | mistralai/mistral-medium-3-5 |
| Nemotron 3 Ultra | nvidia/nemotron-3-ultra-550b-a55b |
Claude, GPT and Gemini models are not among them. Copy a model ID from its page in Models: see Models.
The allowance is the maker's word, as the catalogue records it. Read the model's licence for your own use before you train on its answers.
2. Hold every request to that rule
Send provider.enforce_distillable_text: true with each request. OpenDunes then sends it only to a model whose maker allows training on its answers; a model that does not is refused, and the body says why.
{
"model": "deepseek/deepseek-v4-pro",
"provider": {
"enforce_distillable_text": true
},
"messages": [
{
"role": "system",
"content": "You answer customers of a clothing shop in Oran."
},
{
"role": "user",
"content": "Le burnous existe en quelle taille ?"
}
]
}The OpenAI library sends provider as an extra body field: extra_body in Python. Every field of the route: Chat.
Train on answers you collected without the rule, then check the maker's terms later.
Send the rule with every request, so a teacher that forbids it never answers into your dataset.
3. Collect the pairs
Send each prompt of your training set to the teacher, and save the prompt with its answer, one pair per line. Illustrative: the fields your training tool reads may differ.
{
"messages": [
{
"role": "user",
"content": "Le burnous existe en quelle taille ?"
},
{
"role": "assistant",
"content": "Taille unique, il s'ajuste avec sa ceinture."
}
]
}| Keep | Why |
|---|---|
| prompts like the ones your users send | the student learns the work it will do |
| only answers you checked | a wrong answer teaches the wrong thing |
usage.cost of each answer | the dataset's cost in DADAThe Algerian dinar, the one currency the docs show.More on Balance and charges →, summed |
The script asks the teacher each prompt with the rule from step 2, and saves each pair as a line of pairs.jsonl. A teacher that refuses prints its status, and nothing is saved for that prompt. It also asks MODEL, the smaller model you would start from, with no rule: its answers go to baseline.jsonl, for the comparison in step 4 and for nothing else. It sums usage.cost for each file, in DA.
| Seat | Holds | Why |
|---|---|---|
TEACHER | a model from the table in step 1 | its answers are the pairs |
MODEL | the smaller model you would start from | its answers are the baseline, never training data |
import json
import os
from openai import APIStatusError, OpenAI
API_KEY = os.environ["OPENDUNES_API_KEY"]
BASE_URL = "https://opendunes.com/v1"
MODEL = "anthropic/claude-sonnet-4.5"
TEACHER = "deepseek/deepseek-v4-pro"
BRIEF = "You answer customers of a clothing shop in Oran."
PROMPTS = [
"Le burnous existe en quelle taille ?",
"Livrez-vous à Constantine ?",
"Puis-je échanger un burnous après l'achat ?",
]
RULE = {"provider": {"enforce_distillable_text": True}}
PAIRS = "pairs.jsonl"
BASELINE = "baseline.jsonl"
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
spent = {"teacher": 0.0, "baseline": 0.0}
def ask(model, prompt, fields):
reply = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": BRIEF},
{"role": "user", "content": prompt},
],
extra_body=fields,
)
return reply.choices[0].message.content, reply.usage.cost
def save(path, prompt, answer):
pair = {
"messages": [
{"role": "user", "content": prompt},
{"role": "assistant", "content": answer},
]
}
with open(path, "a", encoding="utf-8") as out:
out.write(json.dumps(pair, ensure_ascii=False) + "\n")
for prompt in PROMPTS:
try:
answer, cost = ask(TEACHER, prompt, RULE)
except APIStatusError as error:
print("teacher refused, status", error.status_code)
continue
save(PAIRS, prompt, answer)
spent["teacher"] += cost
answer, cost = ask(MODEL, prompt, {})
save(BASELINE, prompt, answer)
spent["baseline"] += cost
print(spent)A teacher whose maker does not allow training on its answers is refused, and the rule keeps it out of pairs.jsonl. Read each line of pairs.jsonl before you train: keep only the answers you checked, as the table above says. Run the script again and it adds to both files; delete them first to start over.
4. Train and compare
Train the smaller model in your own tools, on the pairs. Then hold back prompts it never saw, send them to the teacher and to the trained model, and compare the answers the way you judge your app.
Before you train at all, send those same prompts to a smaller model on OpenDunes as it is. Where it already holds, it is the cheaper answer, with no training.
Last modified on