Users praise the Tsinghua paper's Async RL method for long-horizon agent training as very impressive for demonstrating mismatches between group sampling and asynchrony while expecting further research.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.
Anyway, very impressive work showing the mismatch between group sampling and asynchrony is real, and I expect more papers on this.
Their answer is one rollout per prompt. Which means you need a value model again (a critic!). Fun to see, because critics were the classical way to do RL (PPO), but they increase the memory and are hard to train well, so people ditched them for GRPO and its group-mean baseline.
Their claim is that GRPO doesn't fit asynchronous training. A group has to wait for its slowest rollout, and with 128k-token agentic trajectories, that wait is long, so your data goes stale before you train on it.
Users praise the Tsinghua paper's Async RL method for long-horizon agent training as very impressive for demonstrating mismatches between group sampling and asynchrony while expecting further research.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.