Straightforward multi-teacher distillation may falter when teacher training pipelines differ substantially

A post quotes a finding from the Nemotron 3 Ultra tech report: in its trials, teachers trained through substantially different pipelines could not be effectively combined in a straightforward multi-teacher on-policy distillation (MOPD) merge. The post explains that MOPD trains a student model using its own outputs and feedback from teacher models. It suggests first fine-tuning the student on examples from the teachers to make that feedback more useful.