Distillation

Not yet available. Fine-tuning and model distillation are tracked to be built — today, use prompt engineering and cheaper live models via POST /v1/chat/completions. This page describes the intended workflow.

Distillation is the process of training a smaller model to mimic the outputs of a larger "teacher" model on your specific task. Once distilled, the student model handles that task with comparable quality but at a significantly lower per-token DA cost and faster latency.

When distillation is worth it#

Distillation pays off when:

• You run the same task (sentiment classification, entity extraction, a specific Q&A format) at high volume — thousands of requests per day or more.
• The teacher model's outputs are consistently high quality on your task.
• A smaller model's raw capability is sufficient — you just need it to behave like the teacher on your narrow distribution.

It is not worth it for highly varied, open-ended tasks where the teacher's generalization ability is the whole point.

The workflow#

Step 1: Generate teacher outputs#

Use the best model available for your task to generate a large, labeled dataset:

import os, json
from openai import OpenAI

client = OpenAI(
    base_url="https://opendunes.com/api/v1",
    api_key=os.environ["OPENDUNES_API_KEY"],
)

TEACHER = "anthropic/claude-opus-4.8"

def generate_example(prompt: str) -> dict:
    resp = client.chat.completions.create(
        model=TEACHER,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256,
        temperature=0.3,
    )
    return {
        "input": prompt,
        "output": resp.choices[0].message.content,
    }

prompts = load_your_prompts()  # your task inputs
examples = [generate_example(p) for p in prompts]

with open("distillation_data.jsonl", "w") as f:
    for ex in examples:
        f.write(json.dumps(ex) + "\n")

Aim for at least 500–1,000 examples for classification tasks; 2,000–5,000 for generation tasks.

Step 2: Format the training data#

Convert examples to the fine-tuning JSONL format:

def to_finetune_format(example: dict, system_prompt: str) -> dict:
    return {
        "messages": [
            {"role": "system",    "content": system_prompt},
            {"role": "user",      "content": example["input"]},
            {"role": "assistant", "content": example["output"]},
        ]
    }

Step 3: Submit a fine-tuning job#

This endpoint is not yet available — the example below is the intended design. Once fine-tuning ships, you will submit jobs via the API:

curl https://opendunes.com/api/v1/fine-tuning/jobs \
  -H "Authorization: Bearer $OPENDUNES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "training_file": "file_01j...",
    "model": "meta-llama/llama-4-maverick",
    "suffix": "my-task-v1"
  }'

Step 4: Evaluate the student#

Once the fine-tuned model is ready, compare its outputs against the teacher on a held-out evaluation set:

def evaluate(examples: list[dict], student_model: str, teacher_model: str) -> dict:
    match = 0
    for ex in examples:
        student_out = call_model(student_model, ex["input"])
        teacher_out = call_model(teacher_model, ex["input"])
        if student_out.strip() == teacher_out.strip():
            match += 1
    accuracy = match / len(examples)
    print(f"Student match rate vs teacher: {accuracy:.0%}")
    return {"accuracy": accuracy}

A match rate above 90% on your task distribution is typically acceptable for production use.

Cost comparison#

Assume 10,000 requests/day on a classification task, ~500 tokens each:

The one-time cost of generating the training data (1,000 examples × 500 tokens on the teacher model) is roughly 250 DA. At the daily savings above, payback is less than 3 hours of production traffic.