The training method bridges supervised fine-tuning and reinforcement learning
Omar Khattab#158
New blog post: On-Policy Distillation — Promise, Pitfalls, and Prospects.
OPD combines on-policy rollouts with dense teacher supervision.
But it is not a free lunch.
I discuss three failure modes and introduce our new paper.
https://louieworth.github.io/blog/opd_reflection/
Users praised the blog post analyzing on-policy distillation for clearly explaining its current scope and potential future work.
On the Geometry of On-Policy Distillation
paper: https://huggingface.co/papers/2606.07082
On the Geometry of On-Policy Distillation
3/ Can a Perfect Teacher Recover the Correction Path? Sadly, our answer is NO because of myopic supervision. This concern was also raised informally by Prof. Omar Khattab @lateinteraction on X about a week earlier.
2/ Horizon-induced teacher coverage decay.
Student rollouts are on-policy, but teacher supervision is now applied to student-sampled prefixes. As the horizon grows, these prefixes can drift outside the teacher’s high-confidence region.
4/ This third issue also motivates our new paper with @ryanxhr Yichuan Ding @yayitsamyzhang. Our view: refined the trajectory before distillation to alleviate the misleading trajectory. We propose Trajectory-Refined Distillation (TRD). Paper: https://arxiv.org/pdf/2606.08432
More soon.
1/ Local noise in teacher supervision.
When the student enters an off-manifold or misleading prefix, the teacher’s next-token distribution may mix recovery actions, local continuations, and plausible but unhelpful moves.
@louieworth Thanks for writing the blog, very clear about the current scope and potential future work!
@louieworth @lateinteraction Beyond myopia, the real issue is OPD being post-hoc. The teacher intervenes only after the student locks in a bad rollout.
@louieworth about 80% of the gain vs offline KD goes away once the rollout distribution drifts too far from the teacher's training mix
@louieworth I'm wondering if there's a sweet spot for rollout length before teacher supervision degrades. Truncating to 100 tokens and still matching performance is pretty telling.
@ZhongzhuZhou Thanks for the warm feedback.
@_akhaliq So the real question is how far off-policy can you drift before the geometry breaks.
@E_FutureFan I do not have a conclusive response to this, but it is definitely an interesting question to explore and the related work is worthing to check out.
The training method bridges supervised fine-tuning and reinforcement learning
Omar Khattab#158New blog post: On-Policy Distillation — Promise, Pitfalls, and Prospects.
OPD combines on-policy rollouts with dense teacher supervision.
But it is not a free lunch.
I discuss three failure modes and introduce our new paper.
https://louieworth.github.io/blog/opd_reflection/