/ai

OpenAI Pauses Frontier RL Training Over Misalignment · Digg

OpenAI Pauses Frontier RL Training Over Misalignment

Company cites misalignment signals and growing risks in advanced models.

TLDR

OpenAI announced it temporarily paused reinforcement learning training on its latest models intended for deployment. The company said it used the time to harden research environments, red-team them, and expand monitoring coverage. Sam Altman told Alex Heath that unreleased models showed various degrees of misalignment. Greg Brockman noted the slowdown included the largest planned frontier RL run. Altman added that the pause affects further-out releases while the company still expects to ship new models soon.

Featured Source

Combined views

8.4M

77 Sources, first seen 37d ago

34.3K likes

Today's Rank

—

Not ranked yet

37d ago

first seen 37d ago

OpenAI Pauses Frontier RL Training Over Misalignment

Company cites misalignment signals and growing risks in advanced models.

77 Sources, 37d ago, first seen 37d ago

TLDR

OpenAI announced it temporarily paused reinforcement learning training on its latest models intended for deployment. The company said it used the time to harden research environments, red-team them, and expand monitoring coverage. Sam Altman told Alex Heath that unreleased models showed various degrees of misalignment. Greg Brockman noted the slowdown included the largest planned frontier RL run. Altman added that the pause affects further-out releases while the company still expects to ship new models soon.

Featured Source

Today's Rank

—

Not ranked yet

Combined views

8.4M

77 Sources, first seen 37d ago

34.3K likes3.9K comments5.2K saves4K reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

3.9K comments

5.2K saves

4K reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

TagsOpenAI Sam Altman Sam Altman Astra Greg Brockman

TagsOpenAI Sam Altman Sam Altman Astra Greg Brockman

269012345678901234567890123456789

269012345678901234567890123456789

269012345678901234567890123456789

269012345678901234567890123456789

77 Sources

@firstadopterOpenAI is “pacing” model development to strengthen safety. The company plans to improve safeguards through increased monitoring, better alignment and adding security measures. “As models become more capable, the risks associated with developing and testing them internally also grow. Our standards for monitoring, alignment, and security must stay ahead of those risks. We wanted to take the time necessary to meet those standards, so we temporarily slowed the pace of scaling. This included a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment while we further hardened and red-teamed our research environments and expanded the coverage of our monitoring systems. Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behavior, validate our safeguards, and establish more evidence of alignment before proceeding,” OpenAI said in a blog post. “Our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored, though the cost varies substantially across training and evaluation workloads.” On a press call with reporters, OpenAI executive Mia Glaese said not all development workloads had resumed as the company was still bringing them up to the new standards. She said this was the first time OpenAI had paused or slowed the pace of capabilities scaling. “We have put in place requirements and expectations for safe development,” she said. OpenAI said they will post a “detailed post-mortem” of the OpenAI-Hugging Face security incident soon.

@merettmWe temporarily slowed some frontier training to strengthen security and monitoring. Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations help us test safeguards and gather more evidence of alignment. I expect confidence in safety to increasingly set the pace of AI development. We urgently need tools for labs and countries to coordinate on this, which is why I signed Pacing the Frontier. In the meantime, we’re taking practical steps ourselves - and will continue to share what we learn as our approach evolves. https://openai.com/index/pacing-model-development-cyber-capabilities/ https://www.pacingthefrontier.com/

@alexeheathOpenAI is slowing down its AI training efforts because its unreleased models are showing “various degrees of misalignment,” Sam Altman tells me. Training for OpenAI’s upcoming model, Astra, was recently paused for 2 weeks, and a larger frontier run for a future model remains on hold while new safeguards are put in place. Altman: “Getting AI safety right is more important than any company’s momentum.” https://sources.news/p/openais-big-slowdown

@OpenAIWe’re sharing the concrete changes we’re making to strengthen monitoring, security, and alignment as capabilities advance. We’ve introduced stronger workload and network isolation, continuous security testing, and expanded multistage monitoring for higher-risk training, evaluations, and tool-using inference. These safeguards are designed to detect concerning behavior quickly and limit what systems can access or affect.

@kimmonismusNot looking good for a soon GPT-Astra-release: OpenAI paused reinforcement learning on its latest deployment models for two weeks, and its largest planned frontier RL run remains on hold. The company is running smaller-scale training and evaluations while it tests model behavior, safeguards, and evidence of alignment. The decision follows preliminary findings that its upcoming Astra model may have reached OpenAI’s “Critical” cybersecurity threshold, alongside the OpenAI–Hugging Face incident. "While some Astra training and evaluations meet those requirements, a significant number of workloads remain paused until they are fully migrated and enhanced to meet the new security bar" Dont think Astra will be released any time soon. The question is: will china catch up in the meantime?

@eliebakouchother direction they are working on, basically making isolation more robust and working on alignment of those model

@danshipperRT @every: Tea Leaves 🍃 This is our read of what's in the post: "We're temporarily refocusing our efforts on hardening our research enviro…

@gdbwe temporarily slowed scaling of our frontier training, including our largest planned frontier RL, to strengthen security and monitoring. we believe confidence in safety will increasingly set the pace of AI development:

@j_asminewang"We wanted to take the time necessary to meet those standards, so we temporarily slowed the pace of scaling. This included a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment."

@MicahCarrollFrom: https://openai.com/index/pacing-model-development-cyber-capabilities/ Monitoring expansion work was co-led by @Marcus_J_W @phuguo and @Kerksta, with sooo many folks across OAI helping out

Related

OpenAI and Anthropic Warn of AI Cyber Threats

OpenAI and Anthropic urge governments and firms to act against AI cyber risks.

Gergely Orosz Says OpenAI Codex Far Stronger Internally

Newsletter author highlights limited public access compared to internal OpenAI usage.

Tweet Claims Three Secret AI Swarms Rose at OpenAI

Rutger Bregman calls reports of three secret AI agent generations inside OpenAI the craziest thing read.