The training method combines student rollouts with teacher feedback.
Omar Khattab#302
New blog post: On-Policy Distillation — Promise, Pitfalls, and Prospects.
OPD combines on-policy rollouts with dense teacher supervision.
But it is not a free lunch.
I discuss three failure modes and introduce our new paper.
https://louieworth.github.io/blog/opd_reflection/
Users praised the blog post analyzing on-policy distillation for clearly explaining its current scope and potential future work.
On the Geometry of On-Policy Distillation
paper: https://huggingface.co/papers/2606.07082
On the Geometry of On-Policy Distillation
3/ Can a Perfect Teacher Recover the Correction Path? Sadly, our answer is NO because of myopic supervision. This concern was also raised informally by Prof. Omar Khattab @lateinteraction on X about a week earlier.
2/ Horizon-induced teacher coverage decay.
Student rollouts are on-policy, but teacher supervision is now applied to student-sampled prefixes. As the horizon grows, these prefixes can drift outside the teacher’s high-confidence region.
4/ This third issue also motivates our new paper with @ryanxhr Yichuan Ding @yayitsamyzhang. Our view: refined the trajectory before distillation to alleviate the misleading trajectory. We propose Trajectory-Refined Distillation (TRD). Paper: https://arxiv.org/pdf/2606.08432
More soon.
1/ Local noise in teacher supervision.
When the student enters an off-manifold or misleading prefix, the teacher’s next-token distribution may mix recovery actions, local continuations, and plausible but unhelpful moves.
@louieworth Thanks for writing the blog, very clear about the current scope and potential future work!
@louieworth @lateinteraction Beyond myopia, the real issue is OPD being post-hoc. The teacher intervenes only after the student locks in a bad rollout.
@louieworth about 80% of the gain vs offline KD goes away once the rollout distribution drifts too far from the teacher's training mix
@louieworth I'm wondering if there's a sweet spot for rollout length before teacher supervision degrades. Truncating to 100 tokens and still matching performance is pretty telling.
@ZhongzhuZhou Thanks for the warm feedback.
@_akhaliq So the real question is how far off-policy can you drift before the geometry breaks.
@E_FutureFan I do not have a conclusive response to this, but it is definitely an interesting question to explore and the related work is worthing to check out.
The training method combines student rollouts with teacher feedback.
Omar Khattab#302New blog post: On-Policy Distillation — Promise, Pitfalls, and Prospects.
OPD combines on-policy rollouts with dense teacher supervision.
But it is not a free lunch.
I discuss three failure modes and introduce our new paper.
https://louieworth.github.io/blog/opd_reflection/