Researcher Flags Mode Collapse in OPD Teachers From RL Post-Training
Commentary on X
Highest ranked
I think this paper in general makes me very bullish on per-env algorithms, it's good that they added it to prime-rl. Because if you really thing about it, the optimal 1 shot distribution is 0 variance, there's always one most optimal path given the model's priors, so the entropy *should* be low. However, in the absence of coordination (like if doing pass@k or averaging) where you actually do want the distribution and not simply the MLE, then these entropy based algorithms are a good idea. You can use prompting to inform the model on when to do what, so tell the model "On this task, there are many instances operating in parallel, explore your own unique approach" or whatever so it understands that on this specific prompt it needs to be explorative. The default mode shouldn't be the mode that maximizes pass@k, the default mode runs in pass@1!
@vikhyatk GFlowNets are underrated for exploration—combining them with a small diversity reward during RL could unstuck the teacher without sacrificing alignment.
@vikhyatk I wonder if the effectiveness of OPD is from token level advantages or just easy signal from distillation gflownets are dope
OPD's big weakness is that the teacher is usually mode-collapsed due to RL post training. how do we solve this... bring back gflownets?