it’s fine + valid off-policy RL if you’re using a labeled / filtered / surgically edited (with caveats) set of completions from the base model, esp for alignment stuff where you’re not trying to explore anyway
but if the source is something else, it’s like what are you doing lol
victor wembanyana studying magnus carlsen endgame losses so he can avoid making the same mistakes
but if you’re doing that anyway from a base model, why not just do RL or self-distill? bigger batch + fewer steps if you’re worried about hacks?
caveats on editing = v similar to caveats on self-distillation context. be careful + don’t expect magic if you’re pushing it that far
it’s fine + valid off-policy RL if you’re using a labeled / filtered / surgically edited (with caveats) set of completions from the base model, esp for alignment stuff where you’re not trying to explore anyway
but if the source is something else, it’s like what are you doing lol