Async OPD Technique Doubles Throughput in LLM Distillation
Reactions from ranked influencers
2 posts
Link to the journal club discussion ⬇️ https://youtu.be/ZWuFMVmKXrI?si=MZ65wMhevxbyehG-
We read a very interesting paper in the Hugging Face journal club today: “Battling Staleness in Async OPD.” The paper explores similar ideas to Magistral / PipelineRL on how to decouple generation from learning in a fully async manner, but for distillation instead of GRPO. The key result is throughput improvements of ~2x while beating / matching accuracy of sync OPD on math. The main problem you face with OPD is that storing the full student / teacher logit distribution is expensive, so most approaches resort to truncating the logits at top-k. This is OK if you use forward-KL (teacher weighted), but leads to cache misses with reverse-KL (student weighted), which is typically the case we want to induce mode-seeking behaviour in the base model with. Instead, the authors' approach uses local Monte Carlo next-token sampling with importance corrections, and they claim this is stable on large staleness of up to ~32 steps provided the MC sample is large enough. Overall, I found this to be a pretty neat paper and my only critique would have been to see it applied to other base models like Gemma or LiquidAI and other tasks because math is almost always saturated these days. We're looking at how to integrate some of these ideas in TRL too, so the DistillationTrainer should be much faster soon! h/t GPT-5.6 Pro for the distilled image :)