Z.ai updated GLM-5.2's agentic reinforcement learning setup for long-horizon work by restoring a critic-based approach that estimates token-level advantages on individual rollouts, stepping back from the group-relative comparisons used in GRPO.
GLM5.2 brings back the critic.
It was just a matter of time until we people would realize that group-based variance reduction is unfeasible after some horizon length. We need to be more fine-grained. I am sure OAI and Ant have been using value models for quite some time.
Long tasks produce compacted sub-traces of uneven lengths, which undercuts GRPO's reliance on comparing outputs within fixed groups and leaves some data unusable.
The new setup scores each trajectory on its own, naturally handling length differences and raising open questions about how widely other labs will follow for complex agent workflows.
Positive users praised GLM-5.2 for reintroducing critic and value models because they enable superior token-level credit assignment and continual training compared to GRPO.
No Digg Deeper questions have been answered for this story yet.
It's the same principle behind ECHO (SFTing on tool-call outputs), if you understand RL in an information theoretic way. We simply need to get as much out of every rollout as possible. Naive GRPO is unbelievably wasteful.
@hallerite nice! group size 1 unlocks a new world, much better suited for continual training over production traces
@Zai_org
well done @Zai_org
@hallerite Any clue if they did a value head or a whole separate value model?
@Zai_org On Information Theory
@Zai_org
@carnot_cyclist 💯
@Zai_org
@Zai_org
I did RL simulations on a recommendation algorithm in one of my projects, where I started out trying GRPO Vs actor-critic. And holy smokes, actor-critic out performed GRPO by over 15%.
From my observations: GRPO mostly teaches models how to reach an end goal (even when its intermediate steps are clearly wrong or sub-optimal). Actor-Critic methods guides the policy to master step-by-step optimal strategies to reaching an end goal (this makes the policy less prone to hallucinations and even learn faster)
@Zai_org
@redtachyon no clue. I think a value head is more likely than a whole separate model, but maybe it's even simpler than that
@siddarthv66 @hallerite i mean sans us permitting the use of branching, yes, some form of discriminative head bolted on top was always gonna get you better returns for long horizons i just think people spent too much time focused on value estimation in the past in spite of larger *immediate* bottlenecks
@hallerite I went back to PPO for games a bit ago. and yeah, I doubt OAI or ANT did GRPO. it's just the long horizon problem
@hallerite @kalomaze is a hater
@Zai_org
@siddarthv66 @hallerite i still think i'm not a huge fan of absolute regression to estimate value via MSE in the PPO context, and how pairwise or weighed pairwise formulations for a critic might wind up representationally better
@kalomaze @siddarthv66 you can probably also do without a head bolted on top
@kalomaze @hallerite If segments between value estimation are long enough, true scale pilled version is to have reasoning value functions (shared policy weights obviously), trained with RL using neg TD loss as reward. We’re gonna need absurdly long horizons before anything like this is flop optimal
