Teortaxes highlights how top-k distillation can deliver performance on par with policy-gradient methods during same-origin teacher comparisons on Qwen3-scale models while cutting the infrastructure load through smaller signal transmission. The approach breaks down sharply once the teacher originates from a different model, even inside the same family, whereas policy-gradient degrades more gracefully. This observed fragility is presented as a plausible reason full-vocabulary distillation was retained in V4 pipelines.
They find that PG and top-k distillations are comparable so long as you use same-origin teachers, and top-k is cheaper. But if you try to get cute with a different model, even same family… PG struggles, top-k totally collapses. I think this is why V4 went with full-vocabulary.
Chinese labs are iterating on OPD quickly
When teacher and student come from separate origins, top-k distillation loses stability faster than policy-gradient baselines, forcing practitioners to weigh whether the compute savings justify the added risk of training collapse. No public benchmarks yet quantify the exact performance gap or optimal K values for mixed-teacher setups.
The same-origin success of top-k does not extend reliably to heterogeneous teacher mixes, which may explain why V4 avoided the lighter method despite its lower per-step cost. Multi-teacher pipelines now face a concrete trade-off between efficiency and robustness that current stabilizers have not fully resolved.
No Digg Deeper questions have been answered for this story yet.
Might be of interest to @Grad62304977 @rawsh0 @stochasticchasm and others
They find that PG and top-k distillations are comparable so long as you use same-origin teachers, and top-k is cheaper. But if you try to get cute with a different model, even same family… PG struggles, top-k totally collapses. I think this is why V4 went with full-vocabulary.
@teortaxesTex @rawsh0 @stochasticchasm I’d say this is still bullish on mix-RL given MOPD needs the data and compute to train the teacher first, mostly looks like mix rl and MOPD would take the same compute Although u also get the potential benefit of cross domain generalisation with mix RL esp with larger models
Might be of interest to @Grad62304977 @rawsh0 @stochasticchasm and others