AI2's Nathan Lambert says Nvidia's multi-teacher distillation pipeline for Nemotron 3 Ultra represents the new post-training industry standard

The gains is limited on non-agentic reasoning. The hypothesis is that there is a lot of the teacher's distribution/gain that is not ever sampled by the student so the student can never learn.

Some MOPD open questions they list out: - Instead of just doing loss only on the selected token, train on the entire distribution. They say this did not help much as it might amplify noise - How to ensure student trajectories lie within the teacher's support for effective scoring - Efficiency across different domains where rollouts have very different times

MTP is trained with KL against the backbone head's full logits as a speculator. A single head/weights is used recursively Second plot shows efficiency/accuracy across different reasoning efforts. Verbosity is relative to Qwen 3.5 397B's average token usage

@natolambert They ran into some problems which affects generalization in some domains though. Mainly what happens when teacher-student are too far apart and the student never rolls out anything useful for scoring