The approach cut WikiText-2 pre-distillation perplexity by 87 times.
Sewon Min#931
8/ Finally, a complementary direction: we tested compatibility with modularity-aware pretraining (EMO, https://arxiv.org/abs/2605.06663).
Modularity-aware pretraining gives a +3.6pp lift and ~87× lower pre-distillation PPL on the dense student model vs. a regular-MoE teacher.
Users praise modularity-aware pretraining for MoE distillation gains as a promising compression-aware co-design direction worth exploring, thanking its ICLR presentation.
We see such compression-aware pretraining as a co-design direction worth exploring.
Thanks to @sewon__min for introducing the work at ICLR, and to the authors @RyanYixiang @AkshitaB93 for the great work!
Please check out the paper for more details 🙂
Code: http://github.com/krafton-ai/moe-to-dense Paper: http://arxiv.org/abs/2605.28207
This is joint work with amazing collaborators: @JihunYun_ai, Haechan, @gmkim_ai, Joonghyun, @jaewoong_cho from @Krafton_AI
@jhyuckkim does the 87x pre-distillation PPL gap actually survive fine-tuning? i've had big ppl drops vanish after one epoch
No Digg Deeper questions have been answered for this story yet.
The approach cut WikiText-2 pre-distillation perplexity by 87 times.
Sewon Min#9318/ Finally, a complementary direction: we tested compatibility with modularity-aware pretraining (EMO, https://arxiv.org/abs/2605.06663).
Modularity-aware pretraining gives a +3.6pp lift and ~87× lower pre-distillation PPL on the dense student model vs. a regular-MoE teacher.