Prime Intellect's kalomaze warns that VIMPO, a critic-free RL alternative to GRPO, requires a frozen reference model

@teortaxesTex >forces the assumption of a frozen reference model uuuugghhhhhhhhhhhhhhhh

@teortaxesTex wrt serious PPO-replacement-contenders. why not start off with ideas that are like. not regressing a possibly multimodal return estimate via MSE unimodally? distributional RL, but make it LM. something like that. gotta toy w/ ideas like these at some point

@teortaxesTex we should stop doing advantage estimation under gaussian assumptions that simply do not apply to "literally all of combinatorial language", this is the product of a path dependency that isn't killing us but is probably limiting enough to be leaving expressivity on the table