DistillationYou distill a large teacher into a small student by matching the teacher's softened probabilities at temperature , added to the usual cross-entropy on true labels.
Raising from 1 to 4 to soften the targets, the soft loss barely moves the student and the hard-label term takes over.