Rohan Paul on Teacher Lineage in Distillation

Rohan Paul posted that a distilled student model copies how its teacher reasons rather than what it knows. The Bengaluru-based engineer and newsletter writer said teacher lineage therefore matters far more than teacher benchmark scores. He added that a weaker teacher model built from the student's own base model advances the student further than a stronger teacher from a different family. The post presents these points as findings from an interesting study on model distillation.