# Distillation

Collect a large model's answers to train a smaller one, on models whose makers allow it.

1. [Pick a teacher that allows it](#1-pick-a-teacher-that-allows-it).
2. [Hold every request to that rule](#2-hold-every-request-to-that-rule).
3. [Collect the pairs](#3-collect-the-pairs).
4. [Train and compare](#4-train-and-compare).

OpenDunes sends your prompts and returns the answers. The training runs in your own tools: OpenDunes runs no training job and serves no model you trained.

## 1. Pick a teacher that allows it

The teacher is the model whose answers you collect. Some model makers forbid training another model on their answers, and some allow it. Among the labs whose makers allow it, read 29 September 2026: Qwen, Mistral, DeepSeek, NVIDIA, Moonshot AI, Meta and Microsoft.

| Teacher | Model ID |
|---|---|
| DeepSeek V4 Pro | `deepseek/deepseek-v4-pro` |
| Qwen3.7 Max | `qwen/qwen3.7-max` |
| Kimi K3 | `moonshotai/kimi-k3` |
| Mistral Medium 3.5 | `mistralai/mistral-medium-3-5` |
| Nemotron 3 Ultra | `nvidia/nemotron-3-ultra-550b-a55b` |

Claude, GPT and Gemini models are not among them. Copy a model ID from its page in *Models*: see [Models](/docs/models).

<div class="watch-out">
<p>The allowance is the maker's word, as the catalogue records it. Read the model's licence for your own use before you train on its answers.</p>
</div>

## 2. Hold every request to that rule

Send `provider.enforce_distillable_text: true` with each request. OpenDunes then sends it only to a model whose maker allows training on its answers; a model that does not is refused, and the body says why.

```json
{
  "model": "deepseek/deepseek-v4-pro",
  "provider": {
    "enforce_distillable_text": true
  },
  "messages": [
    {
      "role": "system",
      "content": "You answer customers of a clothing shop in Oran."
    },
    {
      "role": "user",
      "content": "Le burnous existe en quelle taille ?"
    }
  ]
}
```

The OpenAI library sends `provider` as an extra body field: `extra_body` in Python. Every field of the route: [Chat](/docs/reference/chat-completions).

<div class="dont">
<p>Train on answers you collected without the rule, then check the maker's terms later.</p>
</div>

<div class="do">
<p>Send the rule with every request, so a teacher that forbids it never answers into your dataset.</p>
</div>

## 3. Collect the pairs

Send each prompt of your training set to the teacher, and save the prompt with its answer, one pair per line. Illustrative: the fields your training tool reads may differ.

```json
{
  "messages": [
    {
      "role": "user",
      "content": "Le burnous existe en quelle taille ?"
    },
    {
      "role": "assistant",
      "content": "Taille unique, il s'ajuste avec sa ceinture."
    }
  ]
}
```

| Keep | Why |
|---|---|
| prompts like the ones your users send | the student learns the work it will do |
| only answers you checked | a wrong answer teaches the wrong thing |
| `usage.cost` of each answer | the dataset's cost in DA, summed |

The script asks the teacher each prompt with the rule from step 2, and saves each pair as a line of `pairs.jsonl`. A teacher that refuses prints its status, and nothing is saved for that prompt. It also asks `MODEL`, the smaller model you would start from, with no rule: its answers go to `baseline.jsonl`, for the comparison in step 4 and for nothing else. It sums `usage.cost` for each file, in DA.

| Seat | Holds | Why |
|---|---|---|
| `TEACHER` | a model from the table in step 1 | its answers are the pairs |
| `MODEL` | the smaller model you would start from | its answers are the baseline, never training data |

```python
import json
import os
from openai import APIStatusError, OpenAI

API_KEY = os.environ["OPENDUNES_API_KEY"]
BASE_URL = "https://opendunes.com/v1"
MODEL = "anthropic/claude-sonnet-4.5"
TEACHER = "deepseek/deepseek-v4-pro"
BRIEF = "You answer customers of a clothing shop in Oran."
PROMPTS = [
    "Le burnous existe en quelle taille ?",
    "Livrez-vous à Constantine ?",
    "Puis-je échanger un burnous après l'achat ?",
]
RULE = {"provider": {"enforce_distillable_text": True}}
PAIRS = "pairs.jsonl"
BASELINE = "baseline.jsonl"

client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
spent = {"teacher": 0.0, "baseline": 0.0}


def ask(model, prompt, fields):
    reply = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": BRIEF},
            {"role": "user", "content": prompt},
        ],
        extra_body=fields,
    )
    return reply.choices[0].message.content, reply.usage.cost


def save(path, prompt, answer):
    pair = {
        "messages": [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": answer},
        ]
    }
    with open(path, "a", encoding="utf-8") as out:
        out.write(json.dumps(pair, ensure_ascii=False) + "\n")


for prompt in PROMPTS:
    try:
        answer, cost = ask(TEACHER, prompt, RULE)
    except APIStatusError as error:
        print("teacher refused, status", error.status_code)
        continue
    save(PAIRS, prompt, answer)
    spent["teacher"] += cost
    answer, cost = ask(MODEL, prompt, {})
    save(BASELINE, prompt, answer)
    spent["baseline"] += cost

print(spent)
```

```javascript
import { appendFileSync } from "node:fs";
import OpenAI from "openai";

const API_KEY = process.env.OPENDUNES_API_KEY;
const BASE_URL = "https://opendunes.com/v1";
const MODEL = "anthropic/claude-sonnet-4.5";
const TEACHER = "deepseek/deepseek-v4-pro";
const BRIEF = "You answer customers of a clothing shop in Oran.";
const PROMPTS = [
  "Le burnous existe en quelle taille ?",
  "Livrez-vous à Constantine ?",
  "Puis-je échanger un burnous après l'achat ?",
];
const RULE = { provider: { enforce_distillable_text: true } };
const PAIRS = "pairs.jsonl";
const BASELINE = "baseline.jsonl";

const client = new OpenAI({ baseURL: BASE_URL, apiKey: API_KEY });
const spent = { teacher: 0, baseline: 0 };

async function ask(model, prompt, fields) {
  const reply = await client.chat.completions.create({
    model,
    messages: [
      { role: "system", content: BRIEF },
      { role: "user", content: prompt },
    ],
    ...fields,
  });
  return [reply.choices[0].message.content, reply.usage.cost];
}

function save(path, prompt, answer) {
  const pair = {
    messages: [
      { role: "user", content: prompt },
      { role: "assistant", content: answer },
    ],
  };
  appendFileSync(path, JSON.stringify(pair) + "\n");
}

for (const prompt of PROMPTS) {
  let answer, cost;
  try {
    [answer, cost] = await ask(TEACHER, prompt, RULE);
  } catch (error) {
    if (!(error instanceof OpenAI.APIError)) throw error;
    console.log("teacher refused, status", error.status);
    continue;
  }
  save(PAIRS, prompt, answer);
  spent.teacher += cost;
  [answer, cost] = await ask(MODEL, prompt, {});
  save(BASELINE, prompt, answer);
  spent.baseline += cost;
}

console.log(spent);
```

```go
package main

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"os"
	"strconv"

	"github.com/openai/openai-go/v3"
	"github.com/openai/openai-go/v3/option"
)

const (
	baseURL  = "https://opendunes.com/v1"
	model    = "anthropic/claude-sonnet-4.5"
	teacher  = "deepseek/deepseek-v4-pro"
	brief    = "You answer customers of a clothing shop in Oran."
	pairs    = "pairs.jsonl"
	baseline = "baseline.jsonl"
)

type message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

func main() {
	apiKey := os.Getenv("OPENDUNES_API_KEY")

	prompts := []string{
		"Le burnous existe en quelle taille ?",
		"Livrez-vous à Constantine ?",
		"Puis-je échanger un burnous après l'achat ?",
	}
	rule := map[string]any{
		"provider": map[string]any{"enforce_distillable_text": true},
	}

	client := openai.NewClient(
		option.WithBaseURL(baseURL),
		option.WithAPIKey(apiKey),
	)
	ask := func(
		model string, prompt string, fields map[string]any,
	) (string, float64, error) {
		params := openai.ChatCompletionNewParams{
			Model: openai.ChatModel(model),
			Messages: []openai.ChatCompletionMessageParamUnion{
				openai.SystemMessage(brief),
				openai.UserMessage(prompt),
			},
		}
		params.SetExtraFields(fields)
		reply, err := client.Chat.Completions.New(context.TODO(), params)
		if err != nil {
			return "", 0, err
		}
		return reply.Choices[0].Message.Content, charge(reply), nil
	}

	spent := map[string]float64{"teacher": 0, "baseline": 0}
	for _, prompt := range prompts {
		answer, cost, err := ask(teacher, prompt, rule)
		var apiErr *openai.Error
		if errors.As(err, &apiErr) {
			fmt.Println("teacher refused, status", apiErr.StatusCode)
			continue
		}
		if err != nil {
			panic(err)
		}
		save(pairs, prompt, answer)
		spent["teacher"] += cost

		answer, cost, err = ask(model, prompt, map[string]any{})
		if err != nil {
			panic(err)
		}
		save(baseline, prompt, answer)
		spent["baseline"] += cost
	}

	fmt.Println(spent)
}

func save(path, prompt, answer string) {
	pair := map[string][]message{"messages": {
		{Role: "user", Content: prompt},
		{Role: "assistant", Content: answer},
	}}
	line, _ := json.Marshal(pair)
	flags := os.O_APPEND | os.O_CREATE | os.O_WRONLY
	file, err := os.OpenFile(path, flags, 0o644)
	if err != nil {
		panic(err)
	}
	defer file.Close()
	fmt.Fprintln(file, string(line))
}

func charge(reply *openai.ChatCompletion) float64 {
	cost := reply.Usage.JSON.ExtraFields["cost"].Raw()
	value, _ := strconv.ParseFloat(cost, 64)
	return value
}
```

```bash cURL
ADDRESS="https://opendunes.com/v1/chat/completions"
MODEL="anthropic/claude-sonnet-4.5"
TEACHER="deepseek/deepseek-v4-pro"
BRIEF="You answer customers of a clothing shop in Oran."
SIZE="Le burnous existe en quelle taille ?"
DELIVERY="Livrez-vous à Constantine ?"
EXCHANGE="Puis-je échanger un burnous après l'achat ?"
RULE='{"provider": {"enforce_distillable_text": true}}'

ask() {
  jq -n --arg model "$1" --arg brief "$BRIEF" --arg prompt "$PROMPT" \
    --argjson fields "$2" \
    '{model: $model,
      messages: [{role: "system", content: $brief},
                 {role: "user", content: $prompt}]} + $fields' |
    curl -s "$ADDRESS" \
      -H "Authorization: Bearer $OPENDUNES_API_KEY" \
      -H "Content-Type: application/json" \
      -d @-
}

pair() {
  jq -c --arg prompt "$PROMPT" 'select(.choices) |
    {messages: [{role: "user", content: $prompt},
                {role: "assistant",
                 content: .choices[0].message.content}]}'
}

for PROMPT in "$SIZE" "$DELIVERY" "$EXCHANGE"; do
  ANSWER="$(ask "$TEACHER" "$RULE")"
  echo "$ANSWER" | pair >> pairs.jsonl
  echo "$ANSWER" | jq -r '.usage.cost // empty' >> teacher-cost.txt
  ANSWER="$(ask "$MODEL" '{}')"
  echo "$ANSWER" | pair >> baseline.jsonl
  echo "$ANSWER" | jq -r '.usage.cost // empty' >> baseline-cost.txt
done

jq -s 'add' teacher-cost.txt
jq -s 'add' baseline-cost.txt
```

A teacher whose maker does not allow training on its answers is refused, and the rule keeps it out of `pairs.jsonl`. Read each line of `pairs.jsonl` before you train: keep only the answers you checked, as the table above says. Run the script again and it adds to both files; delete them first to start over.

## 4. Train and compare

Train the smaller model in your own tools, on the pairs. Then hold back prompts it never saw, send them to the teacher and to the trained model, and compare the answers the way you judge your app.

Before you train at all, send those same prompts to a smaller model on OpenDunes as it is. Where it already holds, it is the cheaper answer, with no training.
