@teortaxesTex wrt serious PPO-replacement-contenders. why not start off with ideas that are like. not regressing a possibly multimodal return estimate via MSE unimodally? distributional RL, but make it LM. something like that. gotta toy w/ ideas like these at some point
@teortaxesTex >forces the assumption of a frozen reference model uuuugghhhhhhhhhhhhhhhh
Users criticized Gaussian assumptions in advantage estimation for language models as invalid for combinatorial spaces, supporting distributional RL alternatives to PPO.
No Digg Deeper questions have been answered for this story yet.
@teortaxesTex we should stop doing advantage estimation under gaussian assumptions that simply do not apply to "literally all of combinatorial language", this is the product of a path dependency that isn't killing us but is probably limiting enough to be leaving expressivity on the table
@teortaxesTex wrt serious PPO-replacement-contenders. why not start off with ideas that are like. not regressing a possibly multimodal return estimate via MSE unimodally? distributional RL, but make it LM. something like that. gotta toy w/ ideas like these at some point