Users call the SAO method research very impressive because it demonstrates a real mismatch between group sampling and asynchrony, and they expect more papers on the topic.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.
Anyway, very impressive work showing the mismatch between group sampling and asynchrony is real, and I expect more papers on this.
They show that GRPO collapses faster, while SAO not and beats GRPO variants on SWE-Bench Verified and the math benchmarks. There's also a nice experiment where the reward preference shifts mid-training and the value model tracks it much faster than a running-mean baseline. https://x.com/ziv_ravid/status/2075617896470864172/photo/1
To make the critic trainable, they freeze the attention layers and only update the MoE projections (the gradient blowups come from the attention layers). They update the critic twice per policy step. And for multi-turn trajectories, they skip observation tokens in GAE. https://x.com/ziv_ravid/status/2075617891353788539/photo/1
Agent trajectories interleave model actions with environment observations. Standard GAE propagates advantages through all of it, including tokens the model never produced. So they connect the last token of one action directly to the first token of the next
Users call the SAO method research very impressive because it demonstrates a real mismatch between group sampling and asynchrony, and they expect more papers on the topic.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.