OpenAI Details Safety Alignment For Long-Horizon Models
Reactions from ranked influencers
2 posts
Just read this post, which I appreciate for sharing more details -- seems like long-horizon instruction following of user constraints was a weakness. But it unfortunately still doesn't say very much about RL training incentives. https://openai.com/index/safety-alignment-long-horizon-models/
A few more thoughts on why / how this seems relatively fixable to me:
More on thoughts on an alternative SFT approach to safe reasoning training I'm curious if labs have tried. I recently learned (from @DianboLiu) that there are 2 camps when it comes to reasoning training: (1) outcome RL vs. (2) tons of rollouts, then SFT on successful rollouts. https://twitter.com/xuanalogue/status/2079724949636579543