Prime Intellect's Elie Bakouch says GLM5.2 reintroduces critic models, prompting debate on PPO value estimation trade-offs

yeah there have been rumors for a long time that none of the frontier labs actually use GRPO

in general i think the value estimation question is underspecified without the broader systems engineering context that your particular flavor of policy gradient is embedded in, i.e it's not as close to being a pure research project as people like to treat it as

to be fair, explicit discrimination is a meaningfully different objective. tho i still think more Weight is put on the idea of explicit value estimation as a hard necessity rather than what is, at its worst, "trade variance for spurious relationships with this one weird trick!"

It's the same principle behind ECHO (SFTing on tool-call outputs), if you understand RL in an information theoretic way. We simply need to get as much out of every rollout as possible. Naive GRPO is unbelievably wasteful.

well done @Zai_org

@siddarthv66 @hallerite i mean sans us permitting the use of branching, yes, some form of discriminative head bolted on top was always gonna get you better returns for long horizons i just think people spent too much time focused on value estimation in the past in spite of larger *immediate* bottlenecks

@hallerite @kalomaze is a hater

@kalomaze Do people in the language model use a seperate net for the critic or is it the same branch for actor + critic

@Zai_org

@Zai_org On Information Theory

@hallerite nice! group size 1 unlocks a new world, much better suited for continual training over production traces

@Zai_org

@Zai_org

@Zai_org

@Zai_org

@siddarthv66 @hallerite i still think i'm not a huge fan of absolute regression to estimate value via MSE in the PPO context, and how pairwise or weighed pairwise formulations for a critic might wind up representationally better

@kalomaze @hallerite If segments between value estimation are long enough, true scale pilled version is to have reasoning value functions (shared policy weights obviously), trained with RL using neg TD loss as reward. We’re gonna need absurdly long horizons before anything like this is flop optimal

@kalomaze @siddarthv66 you can probably also do without a head bolted on top

@yacineMTB it's overwhelmingly the shared trunk + lightweight head trained on top thing i believe if you asked me to roll a PPO implementation into prime-rl tomorrow, i'd first reach for a trainer only LoRA critic path also by "branched" i meant "branched trajectories with shared prefixes"

@Zai_org