Commentary on X
@stanislavfort It's not distillation like in the original Geoff Hinton sense. It seems to be mostly stealing SFT data and SFT warm up data for RL. Could also be stuff like just using the models to create RL problems. In any case the word distillation is being changed/abused
What if you SFT into separate experts first ("hard distillation") using the traces you have collected from other models, and then you use soft distillation of these experts into the actual model That should work better with less interference 馃