Quite a jump from vanilla OPD. Here the authors use the difference between a privileged teacher and privileged student to compute a token-level advantage. The advantage is used to switch from weak/strong distillation modes.
🔗https://arxiv.org/abs/2606.30626
Many users praise Xiaomi's MOPD method for multi-teacher model integration because its logic is clear and resembles the industry's best solution.
No Digg Deeper questions have been answered for this story yet.
Xiaomi dedicates a separate paper to their MOPD method used for MiMo, which is similar to the industry standard as we've been seeing in DS, GLM etc. They frame it as a "capability integration paradigm". Mix-RL aka rawdogging is the 2nd best one. This addresses our post-V4 debate:
https://arxiv.org/abs/2606.30406
OPD to combine multiple teachers. It is a baseline now. One detail could be whether token-level KL or top-K/full vocabulary distillation is better. (They found token-level KL works well enough.)
Chinese labs are iterating on OPD quickly
Quite a jump from vanilla OPD. Here the authors use the difference between a privileged teacher and privileged student to compute a token-level advantage. The advantage is used to switch from weak/strong distillation modes.
🔗https://arxiv.org/abs/2606.30626
see here
They find that PG and top-k distillations are comparable so long as you use same-origin teachers, and top-k is cheaper. But if you try to get cute with a different model, even same family… PG struggles, top-k totally collapses. I think this is why V4 went with full-vocabulary.
Might be of interest to @Grad62304977 @rawsh0 @stochasticchasm and others
@teortaxesTex 小米这套逻辑确实是行业最优解的变体
@teortaxesTex 小米这套路径逻辑确实清晰