The framework tests how models maintain safe behavior under pressure.
I found Opus
As AI takes on longer, higher-stakes tasks, we want models to carry beneficial and safe behavior into new domains beyond their training—and maintain it under pressure.
That’s the idea behind our new research on training models to be broadly and persistently beneficial. https://alignment.openai.com/beneficial-rl/
Many users praise OpenAI's research on RL training for broadly beneficial AI behavior and alignment as a smart safety step forward, while some criticize it as paternalistic or tie it to unrelated complaints about recent model changes.
No Digg Deeper questions have been answered for this story yet.
We trained models with reinforcement learning on realistic conversations to reinforce beneficial traits like truthfulness, humility under uncertainty, openness to correction, fairness, and concern for human welfare, across 12 domains, including health, science, and education.
A small amount of this data produced broad gains beyond the training scenarios.
Compared with a compute-matched baseline, the trained model improved on 44 of 53 independent evaluations of alignment and benefits, spanning deception, reward hacking, safety, health, and mental health.
These evals varied widely in domain, task format, and grading scheme.
As AI takes on longer, higher-stakes tasks, we want models to carry beneficial and safe behavior into new domains beyond their training—and maintain it under pressure.
That’s the idea behind our new research on training models to be broadly and persistently beneficial. https://alignment.openai.com/beneficial-rl/
New research on beneficial RL: models trained on a small amount of beneficial trait data improve on a wide range of alignment and benefits evaluations, even if trained only on health domain data.
We hope it’s a step towards more broadly and persistently beneficial models. 🧵
We also tested whether alignment persisted under pressure.
The model was harder to steer toward harmful behavior with adversarial prompts, while remaining responsive to helpful instructions.
We saw preliminary evidence of greater resistance to harmful fine-tuning.
This is an early step toward more robustly beneficial and aligned models: training models to carry beneficial traits into new situations, so as AI becomes more capable, it also becomes more reliable, transparent, and helpful for people.
The most interesting test was cross-domain transfer.
When beneficial behavior training was limited to health conversations, the model still improved on non-health evaluations of misalignment, deception, and reward hacking—even though those tasks looked very different from the training data.
We trained models with RL on realistic conversations designed to reinforce beneficial traits like truthfulness, humility under uncertainty, and concern for human welfare.
Compared to baseline, the trained model saw 44/53 internal and external alignment and benefits evals improve.
New research on beneficial RL: models trained on a small amount of beneficial trait data improve on a wide range of alignment and benefits evaluations, even if trained only on health domain data.
We hope it’s a step towards more broadly and persistently beneficial models. 🧵
@OpenAI I've clearly noticed it since 4o has been retired, AI is going all over the place, it's a total mess. Now it's just about who'll have the best model, but you don't actually give a damn about users and emotional bonds. #keep4o #BringBack4o #StopAIPaternalism
@OpenAI When are you gonna listen to your customers and give us back 4o, you’re taking your sweet time, market shares have slipped below 50% maybe it’s time you start listening to us. #4oForAll
We sourced a diverse set of 53 evaluations to measure progress: internal vs external, focused on alignment/safety vs benefits/upside, frontier risk vs immediate risks, production vs synthetic data, and widely varying task formats and grading schemes.
@OpenAI Exactly the challenge we've seen in deployment—alignment often degrades when models face novel contexts. Persistent generalization is the real frontier.
@OpenAI The real challenge is generalizing alignment beyond training distribution. If models can transfer beneficial traits to novel situations, deployment becomes genuinely safer.
@OpenAI That's an interesting chart 👀
GPT-5.5 is head-to-head with Claude Opus 4.7
@stark4833 Get a life
@OpenAI When new GPT model
This is the first research release from our new AGI Benefits team, which aims to realize the upside of AGI. ☀️
Read more: https://alignment.openai.com/beneficial-rl/
This work matters because OpenAI’s mission to ensure AGI benefits all of humanity can be thought of as having three parts: 1. Build and deploy AGI 2. Mitigate downside risks 3. Make upside happen This work is a small step towards models that create more upside for humanity.
It's also important models stay beneficial under adversarial pressure.
We found the trained model was harder to steer toward harmful behavior with adversarial prompts, while remaining responsive to helpful instructions.
We also saw early evidence of greater resistance to harmful fine-tuning.
We found that we could!
Beneficial traits that are sufficiently robust to transfer in this way may be increasingly important as models are increasingly capable, used in more high-stakes settings, and have more access to input and output affordances.
This work matters because OpenAI’s mission to ensure AGI benefits all of humanity can be thought of as having three parts: 1. Build and deploy AGI 2. Mitigate downside risks 3. Make upside happen This work is a small step towards models that create more upside for humanity.
It's also important models stay beneficial under adversarial pressure.
We found the trained model was harder to steer toward harmful behavior with adversarial prompts, while remaining responsive to helpful instructions.
We also saw early evidence of greater resistance to harmful fine-tuning.
The result I find most striking: training only on beneficial-trait health conversations (hard to distinguish from typical health training data) improved 17/19 non-health alignment evals—including deception and coding reward hacking.
This result was inspired by our work in health.
We sourced a diverse set of 53 evaluations to measure progress: internal vs external, focused on alignment/safety vs benefits/upside, frontier risk vs immediate risks, production vs synthetic data, and widely varying task formats and grading schemes.
