AI model distillation and concerns about student overconfidence
A user citing research argues that distillation can fail to copy a teacher model faithfully—and that student models systematically exaggerate the teacher’s confidence.
TLDR
A user criticizes model distillation, in which a student model is trained to mimic a teacher. Citing a linked research paper, they say gradient descent often fails to find parameters that copy the teacher, even when those parameters exist and training starts nearby. They also say students systematically exaggerate the teacher’s confidence.
Combined views
3.5K
1 Source, first seen 4h ago
AI model distillation and concerns about student overconfidence
A user citing research argues that distillation can fail to copy a teacher model faithfully—and that student models systematically exaggerate the teacher’s confidence.
TLDR
A user criticizes model distillation, in which a student model is trained to mimic a teacher. Citing a linked research paper, they say gradient descent often fails to find parameters that copy the teacher, even when those parameters exist and training starts nearby. They also say students systematically exaggerate the teacher’s confidence.