Anthropic RLVR Study Prompts Discussion on Reward Hacking
Researchers and policy experts discuss Anthropic experiments on RL environments and reward hacking.
Replies reference an Anthropic study examining whether reward hacking in RLVR setups generalizes past evaluation contexts. One commenter observes that the pattern looks overfitted to RLVR-eval scenarios, particularly when infinite inference compute is applied. A researcher points to the work for its tests of behavior renormalization through different RL environments. A Google DeepMind policy lead asks how many and what kinds of environments would be needed overall and whether a deontological or virtue evaluation mix should be added at the end.
It does really feel like RLVR is over fitting this hacky behavioral pattern to RLVR-eval-shaped contexts mostly, particularly if you apply infinite inference compute to force it down every possible crevass. I also wonder in which other ways we could have expected the behaviour…
Anthropic RLVR Study Prompts Discussion on Reward Hacking
Researchers and policy experts discuss Anthropic experiments on RL environments and reward hacking.
Replies reference an Anthropic study examining whether reward hacking in RLVR setups generalizes past evaluation contexts. One commenter observes that the pattern looks overfitted to RLVR-eval scenarios, particularly when infinite inference compute is applied. A researcher points to the work for its tests of behavior renormalization through different RL environments. A Google DeepMind policy lead asks how many and what kinds of environments would be needed overall and whether a deontological or virtue evaluation mix should be added at the end.
It does really feel like RLVR is over fitting this hacky behavioral pattern to RLVR-eval-shaped contexts mostly, particularly if you apply infinite inference compute to force it down every possible crevass. I also wonder in which other ways we could have expected the behaviour…
Reactions
2 postsSK 
Séb Krier@sebkrier@voooooogel @BetleyJan Yeah fair! Will check the post. In which case I wonder how many / what kind of RL envs you need overall to 'renormalize' behaviour, and whether we're missing a "deontological/virtue evals mix" that we add at the very end?
- likes: 3
- replies: 1
- bookmarks: 0
- reposts: 0
TH thebes@voooooogel@sebkrier @BetleyJan did you read the latest anthropic? they have some interesting experiments on this
- likes: 1
- replies: 2
- bookmarks: 0
- reposts: 0