/Tech1h ago

Google DeepMind's Pranav Shyam argues short-horizon PPO reduces to bandit problems, but open-source developer Grad disputes the latency impact

AI Judge changed title after evaluation, original title: "Google DeepMind's Pranav Shyam argues short-horizon RL tasks act like bandit problems, making value functions unnecessary and adding PPO latency"

Story Overview

A public exchange on X between Google DeepMind researcher Pranav Shyam and open-source account Grad highlights how short-horizon reinforcement learning setups mirror bandit problems, removing any advantage from value functions while still forcing PPO to wait on its critic and add latency.

530277

#1525

Original post

Pranav Shyam@recurseparadox#1597inTech

@Grad62304977 Short horizon is a bandit problem so value functions are unnecessary. Long horizon chains have many sub chains that have been seen previously and can be valued. Short horizon PPo has extra latency has it has to wait for the critic

Grad@Grad62304977

Not even directly saying that PPO is bad, but just questioning the idea that before PPO wasn’t worth it and now it is. U could say for shorter horizons the value models job is easier so it should also be great there too

1:14 AM · Jul 1, 2026 · 13 Views

Developer Impact

Horizon length dictates algorithm choice

Shyam notes that long-horizon sequences let prior sub-chains receive direct value estimates, whereas short ones gain nothing from this machinery yet still pay the PPO overhead of running a separate critic.

Open Question

Trainer bottlenecks may shift at scale

Grad observes that at high token counts the RL trainer itself stops being the limiter, leaving open how these horizon-based simplifications would actually perform once integrated into production pipelines.

Sentiment

Sentiment building, check back later.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

Posts from X

Most Activity

VIEWS21LIKES2

Pranav Shyam@recurseparadox

@Grad62304977 Short: 32K or less. Few pages long Long: 64/128K and higher, multiple rounds of tool calls or code execution

43m212

BOOKMARKS1

Grad@Grad62304977

@recurseparadox But we’re talking like 10-20k reasoning tokens where already the trainer was by far not the bottleneck and also I’m guessing not really short horizons

Pranav Shyam@recurseparadox

1h501

REPLIES1

Grad@Grad62304977

@recurseparadox What would u say is long horizon and short horizon?

Pranav Shyam@recurseparadox

@Grad62304977 For long horizon GRPO will not succeed at all. So zero learning signal. Advantage function can rank at least and pick out ones that looked like they were going somewhere.

For short horizon, because PPO is biased, even using optimal trainer will be slower than just Monte Carlo.

55m1600

Pranav Shyam@recurseparadox

@Grad62304977 For long horizon GRPO will not succeed at all. So zero learning signal. Advantage function can rank at least and pick out ones that looked like they were going somewhere.

For short horizon, because PPO is biased, even using optimal trainer will be slower than just Monte Carlo.

Grad@Grad62304977

Like my point is for long horizon I would trust the value models less and less and with compute scaling I would prefer a bigger group size or other ideas to improve credit assignment here If we assume the value model is pretty good then surely it should’ve always been what we should’ve been doing and save lots of compute still

56m1710

Pranav Shyam@recurseparadox

The moment there are few tool calls/ environment returning state I think value model’s take over. They can smear reward in between the sub goals. I think value functions are most useful locally but for loong chain tasks.

Like you’re right in that value functions over very long horizons shouldnt be magically reliable. But I think their main benefit is in subtask rewards

Grad@Grad62304977

@recurseparadox Ok ya fair. Wdyt abt the value models reliability at these different horizons

38m1310

Pranav Shyam@recurseparadox

Tool calls or code exec update the state of the MDP. If you have many states then maybe you know how to value at least some of them. This is where the sharing of knowledge happens between trajectories. The MDP states act like anchor points - they reappear in many trajectories and therefore you can use value function of one for the other. The value estimation of the full trajectory can still be very wrong but at least the model gets some reward (maybe the code produced was compiled but tests all failed for example. Here the value function can reward successful compilation because it has seen that in other successful trajectories)

If the state of the MDP is not changing then the problem is a bandit problem, and there’s nothing to share between trajectories anymore. Initial prompt is the only anchor point. In that case the policy knows as much as the value function.

Grad@Grad62304977

@recurseparadox but whats the intuition here that tool calls and environments have a big effect here. As in its a clear point the value model can make a more accurate prediction on the final expected reward? Why is it different to say a long reasoning chain with a clear step made?

26m1210

Grad@Grad62304977

@recurseparadox But we’re talking like 10-20k reasoning tokens where already the trainer was by far not the bottleneck and also I’m guessing not really short horizons

1h2000

Grad@Grad62304977

@recurseparadox Ok ya fair. Wdyt abt the value models reliability at these different horizons

43m17

Grad@Grad62304977

Pranav Shyam@recurseparadox

Like you’re right in that value functions over very long horizons shouldnt be magically reliable. But I think their main benefit is in subtask rewards

37m1600

Pranav Shyam@recurseparadox

@Grad62304977 Basically advantage functions reduce variance and longer chains have high variance. For low variance setups GRPO sampling and pool is good enough signal

Pranav Shyam@recurseparadox

1h610

Grad@Grad62304977

@recurseparadox hmm ya thats a fair point although i still feel say in math, there could be a natural segmentation of insights and steps the model took that are naturally shared across rollouts (maybe not an exact match but still)

22m4