AIFrom policy gradients to GRPO: reinforcement learning for language models
A technical explainer describes how PPO trains a separate value model to help assess actions, while GRPO instead compares rewards across answers to the same prompt.
TLDR
The post traces a conceptual progression from vanilla policy gradients through REINFORCE and PPO to GRPO and newer variants. It explains policy gradients as making high-reward actions more likely and low-reward actions less likely. REINFORCE estimates the policy gradient from sampled runs. PPO trains a critic to estimate expected returns and uses clipping to constrain policy updates. GRPO removes the critic, comparing rewards among answers to the same prompt while retaining much of PPO’s approach. The author argues that newer variants modify details such as sampling, reward normalization and clipping rather than replace the underlying framework.
From policy gradients to GRPO: reinforcement learning for language models
A technical explainer describes how PPO trains a separate value model to help assess actions, while GRPO instead compares rewards across answers to the same prompt.
TLDR
The post traces a conceptual progression from vanilla policy gradients through REINFORCE and PPO to GRPO and newer variants. It explains policy gradients as making high-reward actions more likely and low-reward actions less likely. REINFORCE estimates the policy gradient from sampled runs. PPO trains a critic to estimate expected returns and uses clipping to constrain policy updates. GRPO removes the critic, comparing rewards among answers to the same prompt while retaining much of PPO’s approach. The author argues that newer variants modify details such as sampling, reward normalization and clipping rather than replace the underlying framework.