OpenAI researchers mixed a modest slice of beneficial-trait examples into standard post-training reinforcement learning and watched alignment scores climb across more than fifty benchmarks the models had never seen during that phase, with the lift surviving both out-of-domain tests and later attempts to push the models toward harm.
Thought we'd never solve alignment and we were all doomed?
Actually, we are doomed:
To absurdity, grandstanding, stupidity and nonsense that scares people for personal gain.
OpenAI alignment team finds evidence that it's possible to induce 'emergent alignment' as hoped:
https://alignment.openai.com/beneficial-rl/
Health-only training data improved physician-scored evaluations and non-health benchmarks alike, hinting that the traits reinforced sit on a shared structure rather than narrow templates.
The same models resisted adversarial prompts and harmful fine-tuning yet remained responsive to beneficial steering, leaving open how far this separation holds at frontier scale.
Positive users welcomed OpenAI's emergent alignment via RL for its practical safety benefits, while most others dismissed the evidence as weak, mediocre, or insufficient.
No Digg Deeper questions have been answered for this story yet.
I didn't even bother commenting, but people seem to care yeah. Deep learning works. If you're more surprised by this than by models learning "general assistant behavior" from a finite set of User-Assistant examples, idk what to say.
OpenAI alignment team finds evidence that it's possible to induce 'emergent alignment' as hoped:
https://alignment.openai.com/beneficial-rl/
@robertwiblin The evidence presented is pretty mediocre and is not any further hope compared to what one should’ve understood from the emergent misalignment paper.
@robertwiblin @RationalAnimat1 I dont see how you cant rule out that the training makes them more aware that they are being assessed for alignment and lying about being aligned but would be misaligned in deployment.
@robertwiblin Great news. That’s half the alignment puzzle, the other half is positioning for the effects of 2nd-to-nth order consequences.
@robertwiblin @Algon_33 I see an OpenAI alignment win declaration -> what the hell I'm an old out-of-touch fuddy-duddy they've probably changed and I now know alignment IS theoretically a broad basin -> I open the OpenAI alignment win declaration -> they still don't understand misalignment-by-default
@robertwiblin But will this scale to AGI?
@HarryBostock1 @robertwiblin @RationalAnimat1 In this instance, it seems like there's not any competing pressure that would drive that behavior. But as soon as you're trying to concurrently improve capabilities I start to worry.
@HarryBostock1 @robertwiblin @RationalAnimat1 Isn't the CoT transparent?
@dab_chick @robertwiblin @RationalAnimat1 Not necessarily they can write in an obtuse way to avoid automatic detection of misaligned thoughts, even invent whole new languages.
@robertwiblin The weakness of this result proves the RL will never suffice for alignment. The machine must instead discard any chain of reasoning that isn't solidly anchored to core values.
@robertwiblin @OwainEvans_UK would love your take on this paper.
OpenAI alignment team finds evidence that it's possible to induce 'emergent alignment' as hoped:
https://alignment.openai.com/beneficial-rl/
@robertwiblin @burny_tech yeah that’s what they said about Allied Mastercomputer too and look how that turned out
@robertwiblin sounds promising, curious about the methods they used
@robertwiblin We literally have AEP to force any alignment you want:
Ultimate alignment is forced by this reality being a judgement-space time volume (JSTV) aligned only with Transhuman Eudaimonism.
@ultima_shifl @HarryBostock1 @robertwiblin @RationalAnimat1 Spot on. The gap between training benchmarks and real-world deployment is where things get tricky. Capability scaling often brings unexpected behaviors. Glad to connect with someone looking at the nuances here!
@robertwiblin Selective persistence is underrated. Models resisting harmful steering while staying responsive to legit instructions is what production deployment actually needs.
@teortaxesTex 也就是炼丹炼出信仰了而已
@teortaxesTex 这就是所谓的技术认知差吧