Study Finds Teacher Lineage Outweighs Benchmarks In LLM Distillation

Rohan Paul posted about the arXiv paper Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of LLMs. The generated headline states that teacher lineage outweighs benchmarks in LLM distillation. The accompanying summary says new research shows student models primarily inherit reasoning habits from their teachers. The paper abstract explains that on-policy distillation transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood.