SAO Adopts Single-Rollout RL With Faster Value Model Training

wh@nrehiew_TECH

The main change is to get rid of group-wise sampling in favor of single-rollout which means we now need a value model again to reduce variance. Value model specifics: 1) Value network takes 2 gradient steps per batch. This is shown to train a more accurate Value model mainly because the value model requires more "data" 2) Freeze attention and only train MoE layers which helps reduce instabilities 3) GAE skips over env tokens and is computed only on action tokens (I wonder how it would work if we add ECHO style SFT losses for the env tokens here) 4) VAPO style length adaptive GAE where longer sequences have smaller per token TD decay over the longer token horizon which helps credit assignment

SAO Adopts Single-Rollout RL With Faster Value Model Training

Commentary on X

Combined views

Sentiment

Positive Read

Digg Deeper