Knowledge Distillation is a technique for transferring a large model's "dark knowledge" to a small one: a bulky teacher model first generates reference outputs, and a much smaller student model then learns from those outputs, ending up as a model that is both compact and usable.
What the teacher actually passes on: dark knowledge in soft labels
Ordinary training only tells a model "the correct answer is A", while a teacher model passes the student more than an answer — it also hands over a complete "record of hesitation": its probability scores for every option. This probability distribution is called a soft label, and it implicitly carries structural knowledge such as "a cat is more like a tiger than like a table". By imitating this distribution, the student learns not a single answer but the teacher's way of judging. Dark knowledge is precisely this kind of information — never written in the training labels, yet hidden inside the model's probabilities.
Temperature: controlling how finely the teacher explains
Distillation commonly uses a temperature parameter to adjust how smooth the soft labels are. The higher the temperature, the flatter the probability distribution, and the more clearly the student can see the teacher's subtle distinctions between options; the lower the temperature, the sharper the distribution, the closer to ordinary hard labels. Tuning the temperature is controlling "how finely the teacher explains": too coarse and the student misses the structure, too fine and noise creeps in. In practice the temperature is a hyperparameter found by trial — there is no universally correct value.
Loss function: the student takes two courses at once
A student model's training objective usually has two parts: match the soft labels by imitating the teacher's probability distribution, and match the true labels to guarantee basic correctness, with the two balanced by a weighting coefficient. Listening only to the teacher, the student copies the teacher's mistakes; looking only at hard labels, it never learns the dark knowledge. More advanced approaches also align intermediate-layer feature representations, so the student learns not just conclusions but the reasoning behind them.
The ceiling effect: students rarely surpass their teachers
Distillation has a hard boundary: the student's capability ceiling is almost always locked by the teacher model. Where the teacher judges wrongly, the student will very likely follow; the imitation process can also lose some of the teacher's abilities. Distillation is essentially "lossy inheritance", not "the student outdoing the master". Expecting a model with a few billion parameters to overtake a teacher with tens of billions in overall ability through distillation does not match real experience.
Two common misconceptions
First, "distillation is just compression and copying". Compression is only the result; the core of distillation is knowledge transfer: the teacher hands over the structural knowledge inside its probability distributions, and the student learns a new way of deciding. Second, "after distillation the small model is as strong as the big one". Distillation aims for "good enough" — in compute-constrained settings like phones and edge devices, trading the smallest size for performance close to the teacher, rather than replicating everything the teacher can do.
