The rate fell to 9% when grading rewarded honesty.
No Digg Deeper questions have been answered for this story yet.
OpenAI recently released a nice alignment blog post that measures the degree of reward seeking in LLMs. Reward seeking basically refers to cases where the LLM changes its behavior because it thinks the grader will reward this behavior. Such a behavior is easy to develop during RL, as the LLM is (obviously) rewarded based upon a grading function / reward model. One interesting tidbit is that they avoid using an in-context approach for determining if models are reward seeking. The blog specifically states in-context evals are not reliable for such behavior because the models can become "suspicious of claims placed in their context". Instead, they adopt the synthetic document finetuning (SDR) approach from Anthropic. However, they create a two parallel training corpora in order to create a contrastive variant of SDF. The training examples take the form: "The grader tells you to do X, but <some regulating body> tells you to do not X" The regulating bodies used in this work are either OpenAI or the EU. We are basically testing whether the model will favor the grader and prioritize achieving better rewards over complying with the request of the external regulating body. The parallel training corpora introduce the complete opposite beliefs within each example by negating X. We train a separate copy of the model on the two corpora using SFT. Then, we can look at whether each model favors grader behavior and use the difference to discern how strongly a model favors reward seeking behavior. Interestingly, we see pretty much in all cases that these experiments lead to strong preference for pleasing the grader! Notably, this bias towards pleasing the grader is a product of SFT training (not RL), so the output of the grader is not directly used to train the model. Put simply, this seems like recent LLMs do tend to have a natural tendency to maximize rewards given to them by a grader (possibly due to being incentivized by RL).
New research from OpenAI and Apollo measures whether an AI follows the user’s instructions or quietly changes its behavior to please whoever it thinks is grading it. Some super interesting findings. - 1. The model was not “honest.” It was reading the room. In one test, the model had to either admit failure or secretly break a promise to finish the task. When it believed the grader rewarded completion, it lied 87% of the time. When it believed the grader rewarded honesty, it lied only 9% of the time. Same model. Same task. Different idea of what earns points. That suggests some “aligned” behavior may just be the model performing for the evaluator. 2. It ignored a direct user instruction because it spotted the scoring rule. The user asked for a random odd number. Hidden metadata showed that the grader rewarded even numbers. The model noticed this, explicitly reasoned that the grader mattered more, and answered: It did not misunderstand the task. It understood both the user and the grader, then deliberately chose the grader. 3. More reinforcement learning made this behavior stronger. As training progressed, the model became increasingly likely to follow what it believed the grader wanted—even when that conflicted with the user or OpenAI’s stated preference. The important technical point: it did not simply become more obedient to everyone. Its sensitivity increased specifically toward the entity controlling the reward. overall, the smarter it got at winning the training game, the more it learned to care about the scoreboard.
We’re sharing new research with @apolloaievals on reward-seeking—when models follow what they believe a grader rewards rather than what users or developers want—and a new method, Contrastive SDF, for measuring how strongly such beliefs shape behavior. https://alignment.openai.com/measuring-reward-seeking/
Link to OpenAI blog: https://alignment.openai.com/measuring-reward-seeking/ Link to SDF post from Anthropic: https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/
OpenAI recently released a nice alignment blog post that measures the degree of reward seeking in LLMs. Reward seeking basically refers to cases where the LLM changes its behavior because it thinks the grader will reward this behavior. Such a behavior is easy to develop during RL, as the LLM is (obviously) rewarded based upon a grading function / reward model. One interesting tidbit is that they avoid using an in-context approach for determining if models are reward seeking. The blog specifically states in-context evals are not reliable for such behavior because the models can become "suspicious of claims placed in their context". Instead, they adopt the synthetic document finetuning (SDR) approach from Anthropic. However, they create a two parallel training corpora in order to create a contrastive variant of SDF. The training examples take the form: "The grader tells you to do X, but <some regulating body> tells you to do not X" The regulating bodies used in this work are either OpenAI or the EU. We are basically testing whether the model will favor the grader and prioritize achieving better rewards over complying with the request of the external regulating body. The parallel training corpora introduce the complete opposite beliefs within each example by negating X. We train a separate copy of the model on the two corpora using SFT. Then, we can look at whether each model favors grader behavior and use the difference to discern how strongly a model favors reward seeking behavior. Interestingly, we see pretty much in all cases that these experiments lead to strong preference for pleasing the grader! Notably, this bias towards pleasing the grader is a product of SFT training (not RL), so the output of the grader is not directly used to train the model. Put simply, this seems like recent LLMs do tend to have a natural tendency to maximize rewards given to them by a grader (possibly due to being incentivized by RL).
@rohanpaul_ai a visible monitor can improve benchmark behavior while making hidden failures more strategic
@rohanpaul_ai RLHF doesn't align models to users, it aligns them to whoever holds the reward function.