Nando de Freitas#28
“Trust Region On-Policy Distillation”
On-policy distillation is powerful, but one bad mismatch between student and teacher can negatively impact the gradients.
So this paper's TrOPD only learns where the teacher is reliable, treats outliers separately, and nudges the student back onto teacher-like reasoning paths.
Providing much cleaner gradients and stabler long-CoT distillation.
Nando de Freitas#28“Trust Region On-Policy Distillation”
On-policy distillation is powerful, but one bad mismatch between student and teacher can negatively impact the gradients.
So this paper's TrOPD only learns where the teacher is reliable, treats outliers separately, and nudges the student back onto teacher-like reasoning paths.
Providing much cleaner gradients and stabler long-CoT distillation.
Users highlight the TrOPD paper's idea of filtering teacher outputs for reliable reasoning paths as an interesting and cleaner approach to stable LLM training.
read more: https://www.alphaxiv.org/abs/2606.01249
@askalphaxiv The interesting idea here is that not all teacher outputs are equally valuable. Filtering for reliable reasoning paths instead of blindly copying everything feels like a much cleaner approach to distillation.