Google DeepMind's Pranav Shyam argues short-horizon PPO reduces to bandit problems, but open-source developer Grad disputes the latency impact

@Grad62304977 Short: 32K or less. Few pages long Long: 64/128K and higher, multiple rounds of tool calls or code execution

@recurseparadox But we’re talking like 10-20k reasoning tokens where already the trainer was by far not the bottleneck and also I’m guessing not really short horizons

@recurseparadox What would u say is long horizon and short horizon?

@Grad62304977 For long horizon GRPO will not succeed at all. So zero learning signal. Advantage function can rank at least and pick out ones that looked like they were going somewhere.

For short horizon, because PPO is biased, even using optimal trainer will be slower than just Monte Carlo.

The moment there are few tool calls/ environment returning state I think value model’s take over. They can smear reward in between the sub goals. I think value functions are most useful locally but for loong chain tasks.

Like you’re right in that value functions over very long horizons shouldnt be magically reliable. But I think their main benefit is in subtask rewards

Like my point is for long horizon I would trust the value models less and less and with compute scaling I would prefer a bigger group size or other ideas to improve credit assignment here If we assume the value model is pretty good then surely it should’ve always been what we should’ve been doing and save lots of compute still

@recurseparadox Ok ya fair. Wdyt abt the value models reliability at these different horizons

@recurseparadox but whats the intuition here that tool calls and environments have a big effect here. As in its a clear point the value model can make a more accurate prediction on the final expected reward? Why is it different to say a long reasoning chain with a clear step made?

True. At some point the LLM acts as an environment in itself and the “policy” is trying to prompt this LLM env to get the right output. So in that case value function should help short context too. So my analysis is not water tight. My mental model is “when many tool calls/code executions happen the value function definitely should help” but it can also help waay before that.

@Grad62304977 Basically advantage functions reduce variance and longer chains have high variance. For low variance setups GRPO sampling and pool is good enough signal

@recurseparadox hmm ya thats a fair point although i still feel say in math, there could be a natural segmentation of insights and steps the model took that are naturally shared across rollouts (maybe not an exact match but still)