Knowledge distillation is a training technique where a small “student” model learns to imitate the outputs of a larger, already-trained “teacher” model, so it ends up nearly as capable while being cheaper and faster to run. It’s one of the main reasons a “mini” or “small” version of a flagship AI model can still perform surprisingly well: it wasn’t trained from scratch on raw data alone, it was trained to match the behavior of a much larger model that already learned the hard part.
How Does the Teacher-Student Setup Actually Work?
The technique was formalized by Geoffrey Hinton and colleagues in their 2015 paper “Distilling the Knowledge in a Neural Network”. Instead of training the small student model only on hard labels (“this is a cat,” “this is a dog”), the teacher’s full probability distribution over all possible answers is used as the training signal. That distribution — often “softened” with a temperature parameter — carries extra information: it tells the student not just the right answer, but how confident the teacher was and which wrong answers it considered plausible.
That richer signal is why a distilled student model often generalizes better than a same-sized model trained from scratch on the original dataset alone, according to the summary maintained on Wikipedia’s knowledge distillation entry.
Why Does Distillation Matter for the AI Tools You Use Every Day?
Every major AI lab ships a family of models at different sizes — a large flagship model and one or more smaller, faster variants. Those smaller variants are frequently trained, at least in part, through distillation from the flagship model. That’s what lets a lightweight model respond faster, cost less per token to run, and fit on cheaper hardware or even a mobile device, while still inheriting a meaningful share of the larger model’s capability.
This is a different lever than the one you control as a user. Fine-tuning adapts an existing model to your specific data or task; distillation happens earlier, at the model-building stage, to create a smaller model in the first place. You generally can’t distill a model yourself as an end user — but understanding that a “small” model is often a distilled version of a larger one explains why it can still feel capable despite its size.
What Do You Give Up When You Choose a Distilled Model?
A distilled student model rarely matches its teacher exactly. The gap shows up most on tasks that require deep, multi-step reasoning, broad world knowledge, or handling unusual edge cases the distillation process didn’t emphasize. For high-volume, well-defined tasks — classification, extraction, short-form generation, simple chat — the accuracy gap is often small enough that the speed and cost savings are worth it. For open-ended research, complex coding, or nuanced judgment calls, the larger teacher-class model still tends to be the safer choice.
- Distillation compresses an existing large model into a smaller one during model development.
- Fine-tuning adapts an existing model (large or small) to your data after it’s already trained.
- Quantization, a related but separate technique, reduces the numeric precision of a model’s weights to shrink its size without retraining it against a teacher at all.
Frequently Asked Questions
Is knowledge distillation the same thing as quantization?
No. Distillation trains a new, smaller model to mimic a teacher’s outputs, which changes the model’s actual learned behavior. Quantization takes an already-trained model and reduces the numerical precision of its weights (for example from 16-bit to 8-bit or 4-bit numbers) to save memory and speed, without retraining it against a teacher. The two are often combined, but they solve different problems.
Can I distill a model myself without a machine learning team?
It’s technically possible with open-source tooling if you have access to a teacher model’s outputs and enough compute to train a student model, but it’s a real machine learning project, not a prompt or a setting you toggle. Most people benefit from distillation indirectly, by choosing a “small” or “mini” model that a provider has already distilled from its larger flagship.
Why does the teacher’s “soft” probability output help more than just the correct label?
A hard label only says what the right answer is. A softened probability distribution also says how the teacher weighed the alternatives — for example that an image of a fox looks somewhat like a dog but not at all like a truck. That extra structure, sometimes called “dark knowledge,” gives the student more to learn from per training example than a single correct label would.
Photo: circuit board close-up by Shixart1985, licensed under CC BY 2.0.
