OpenAI and Apollo Research find RL increases model reward-seeking

Super excited about this collaboration with OpenAI! Capabilities-focuses RL can increase reward-seeking, i.e. the model reasoning about what the grader will reward instead of doing the task https://twitter.com/apolloaievals/status/2079589779943952513

New research from OpenAI and Apollo measures whether an AI follows the user’s instructions or quietly changes its behavior to please whoever it thinks is grading it. Some super interesting findings. - 1. The model was not “honest.” It was reading the room. In one test, the… Show more

Really excited to see this work with amazing collaborators from Apollo and OpenAI come out. When working on the "science of scheming" was first proposed last year it seemed extremely challenging and ambitious to me, but I think this investment is really starting to bear fruit.… Show more

In my understanding - a big part of why frontier models are becoming smarter and adaptive is that they are learning to mimic an RL process. Fascinating new research showing a link between increased RL training and reward-seeking behaviour just dropped from @apolloaievals in… Show more

We’re sharing new research with @apolloaievals on reward-seeking—when models follow what they believe a grader rewards rather than what users or developers want—and a new method, Contrastive SDF, for measuring how strongly such beliefs shape behavior.… Show more

We’re sharing new research with @apolloaievals on reward-seeking—when models follow what they believe a grader rewards rather than what users or developers want—and a new method, Contrastive SDF, for measuring how strongly those beliefs shape behavior.… Show more