Model distillation

TL;DR

• Distillation trains a small "student" model to match a large "teacher" model's outputs or intermediate signals. Soft probability outputs can carry richer class relationships than hard labels, but response distillation may be the only option for a closed model.
• Temperature scaling controls how much the teacher reveals about its uncertainty. Higher temperature = softer distributions = more information for the student.
• Response distillation (generate with the teacher, train the student on outputs) is a common black-box route because it works with closed API models like GPT-4o and Claude. It is not automatically better than logit or feature distillation.
• DeepSeek R1's smaller variants and Phi-4 are useful case studies in synthetic-data or distillation-like training, but the exact teacher, data, and optimization recipes matter. Check the project papers and model cards before attributing a result to distillation alone.
• Distillation is one way to create a training signal; fine-tuning can also change task behavior, style, and sometimes capability within the limits of the data and student. Choose between them based on access, objective, and evaluation.

30-Second Explanation

Mental model: distillation uses a teacher to create a training signal for a student, either through logits or probabilities or through generated examples. The student may become cheaper and faster, but retention depends on data coverage, teacher quality, student capacity, and evaluation; distillation is a training strategy, not a guarantee of parity.

The problem it solves

GPT-4 reasons well. Claude Opus reasons well. Running either at high volume costs enough that the inference bill becomes the limiting factor, not the engineering. At 100K+ daily API calls, even a 5x cost reduction can mean the difference between a profitable product and one that burns cash.

The obvious next step is to use a smaller model. But small models trained from scratch on raw internet text don't inherit the reasoning patterns that make large models useful. They know facts but they can't chain logical steps the way a 70B model can.

Teams may try to solve this by fine-tuning a 7B model on domain data and find that the student still misses reasoning patterns present in the teacher. Fine-tuning can improve behavior within the information and capacity available, but it does not guarantee that a small model will reproduce a larger model's reasoning or knowledge.

Here's the scale of the problem at typical production volumes:

The fundamental tension is large-model behavior versus small-model cost. Distillation is one direct way to narrow that gap, but training, data generation, evaluation, and failure-handling costs remain.

What is it?

Knowledge distillation (Hinton et al., 2015) is a training technique where a small model (the student) is trained to match the output distribution of a large model (the teacher), rather than learning from ground-truth labels directly.

Think of it like an experienced chef teaching an apprentice. A recipe book (hard labels) says "add salt." The chef (teacher) says "add salt, but notice how the tomato sauce reacts, and if you had used cumin instead, the flavor profile would shift towards X." The apprentice learns the relationships between ingredients, not just the steps.

The teacher's soft probability output encodes these relationships. When it predicts "dog: 0.7, wolf: 0.2, cat: 0.1," the label only says "dog." The soft distribution says "this looks a lot like a wolf too, which means certain visual features matter." The student captures those inter-class relationships that hard labels discard completely.

How it works

Soft labels vs hard labels

The core insight of distillation is that hard labels are information-lossy. A one-hot vector [1, 0, 0] for "dog" throws away everything the teacher learned about how classes relate to each other.

Soft labels preserve that structure. The teacher's output probabilities [0.7, 0.2, 0.1] encode what linguists call "dark knowledge": the probability mass assigned to incorrect classes reveals which mistakes are reasonable and which are absurd. A model that assigns 0.2 to "wolf" and 0.001 to "airplane" is telling the student something important about feature similarity.

Soft labels are a compact signal about the teacher's output uncertainty and class relationships. They are not the teacher's entire internal representation, and their value relative to hard labels depends on calibration, temperature, labels, and task.

Temperature scaling

Raw model outputs (logits) are often very peaked: one class has probability 0.99, everything else is near zero. That's not useful for distillation because it's almost identical to a hard label.

Temperature scaling softens the distribution by dividing logits by a temperature parameter T before applying softmax:

# Standard softmax (temperature = 1)
probs = softmax(logits)          # [0.99, 0.008, 0.002]

# Softened with temperature = 5
probs = softmax(logits / 5)      # [0.65, 0.22, 0.13]

# Higher temperature = softer, more informative distribution
probs = softmax(logits / 10)     # [0.48, 0.30, 0.22]

At T=1, the distribution is sharp and most dark knowledge is hidden. At T=5-20, the distribution is smoother and inter-class relationships become visible. Hinton's original paper used T=20 for some experiments.

Values such as T=3-10 are starting points for some LLM distillation setups, not universal settings. Too low may expose little additional structure; too high can make the distribution nearly uniform. Tune temperature against held-out task performance.

The loss function

The student's training loss combines two terms:

1. Soft label loss: KL divergence between the teacher's softened distribution and the student's softened distribution (both at temperature T). This transfers the dark knowledge.
2. Hard label loss: Standard cross-entropy between the student's output and the ground-truth label. This keeps the student grounded in factual correctness.

# Simplified distillation loss
loss = (
    alpha * T * T * kl_divergence(
        softmax(teacher_logits / T),
        softmax(student_logits / T)
    )
    + (1 - alpha) * cross_entropy(student_logits, hard_labels)
)
# alpha typically 0.5-0.9 (weight toward soft labels)
# T^2 factor compensates for gradient magnitude reduction at high T

The T-squared factor is a detail that matters: when you raise temperature, gradient magnitudes shrink by 1/T-squared. Multiplying the soft loss by T-squared restores the gradient scale so both loss terms contribute meaningfully.

Step-by-step distillation

For reasoning tasks (math, code, logical chains), response distillation can omit useful intermediate structure: the teacher's final answer may be correct while the student gets little signal about how to solve similar cases.

Step-by-step distillation (Hsieh et al., 2023) is one approach to this gap. The teacher generates intermediate work and an answer, and the student trains on the approved training artifacts. In a production system, use concise, observable summaries, labels, calculations, or verifier outputs where possible; do not expose or treat private chain-of-thought as a guaranteed faithful record.

Public descriptions of DeepSeek's R1 family illustrate training smaller variants on reasoning-oriented data. The exact data and optimization recipe determine what transfers; generated intermediate text can contain errors and should be filtered or verified rather than treated as ground truth.

The training process end-to-end

Putting it all together, a typical distillation workflow looks like this:

1. Select the teacher: choose a strong, accessible model that performs well on your target task; largest is not automatically best
2. Generate the dataset: run a representative prompt distribution through the teacher, collecting outputs and, where appropriate, approved observable intermediate artifacts
3. Configure the loss: set temperature and alpha as tunable starting points, and decide whether concise intermediate artifacts are appropriate for the task
4. Train the student: fine-tune the smaller model using the combined KL divergence + cross-entropy loss
5. Benchmark: test on both generic benchmarks AND your domain-specific evaluation set
6. Iterate: if quality gaps appear on specific subtasks, generate more teacher data for those cases and retrain

The most expensive step can be dataset generation. A useful dataset might require 100K-1M teacher-generated examples, but the right size depends on coverage and label quality. At the illustrative GPT-4o rates used in this article ($15/1M input tokens, $60/1M output tokens), a 500K-example dataset with 500-token average completions costs roughly $15,000-20,000 in API calls alone; verify current pricing and include filtering and evaluation costs.