OpenAI releases research data, drawing debate over recursive self-improvement

@pmddomingosThree levels of self-serving BS: - This is not recursive self-improvement. - It’s not just in a few frontier labs (ha!). - It’s no reason for “pacing research”.

@SchmidhuberAIRSI will "only be seen inside a few frontier AI labs" ? Nonsense. Concrete RSI algorithms have existed for almost 4 decades. See "Meta Learning and Recursive Self-Improvement since 1987" with links to the original papers https://people.idsia.ch/~juergen/metalearning.html

@rohanpaul_aiRT @rohanpaul_ai: Knowledge work has always been bottlenecked by human serial execution - that time is changing. OpenAI's research workflo…

@rohanpaul_aiRT @rohanpaul_ai: OpenAI just officially said it has reached its "automated research intern" milestone. i.e. a human-supervised system ab…

@rohanpaul_aiRT @rohanpaul_ai: OpenAI engineers are changing roughly 7x more code per contributor than the pre-2025 baseline.

@rohanpaul_aiRT @rohanpaul_ai: One of AI's biggest effects on research, researchers can test far more ideas. OpenAI's experiment velocity has climbed 1…

@peterwildefordOPENAI: "We do not yet know how to safely get all the way to aligned, full RSI. ... Whenever we find that proceeding would pose an unacceptable safety risk, we will respond appropriately including by slowing or stopping our development or deployment of systems we find ourselves unable to sufficiently safeguard." "These are reasons to develop useful automated research capabilities, but they do not mean that rapid RSI is necessarily an outcome we should pursue. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices about the benefits and risks." "After the recent Hugging Face incident, we put this commitment into action⁠, pausing reinforcement learning (RL) training on our latest models intended for deployment while we further hardened and red-teamed our research environments and expanded coverage of our monitoring systems."

@_NathanCalvinOpenAI: "whenever we find that proceeding would pose an unacceptable safety risk, we will respond appropriately, including by slowing or stopping our development or deployment" Anthropic has also made similar comments in the past. Good. However, if either company wants people to actually believe they act on these commitments, they could include these commitments in their binding safety plans under state transparency laws like SB 53! Right now neither of these companies makes any binding commitments about internally deployed models in their frameworks whatsoever - despite those being the models that are likely to be most capable and most pushing the frontier on RSI risks. (left image is OpenAI, right is Anthropic)

@lukeprogRT @peterwildeford: OPENAI: "We do not yet know how to safely get all the way to aligned, full RSI. ... Whenever we find that proceeding wo…

@ChengleiSiRT @_arohan_: Access to compute is definitely important but having right ideas seem lot more. Auto research is still quite nascent largely…

@rohanpaul_aiRT @rohanpaul_ai: I think "time horizon" is becoming one of the more useful ways to talk about agent capability. At OpenAI, as tasks get l…

@wjmzbmr1RT @polynoamial: One of the most interesting blog posts we've released: details on internal research acceleration at @OpenAI. I expect thes…

@GergelyOroszSuper interesting detail from inside OpenAI: The company hired a bunch of ex-Meta people. Some of these people wanted to invest heavily into internal tooling teams (Meta did it, worked great) OpenAI leadership resisted, saying that in an AGI-first world, there will be no internal tooling teams. And they were right: now, with Codex, there's a massive internal tools explosion, without having any internal tooling teams!

@Miles_BrundageRT @_NathanCalvin: OpenAI: "whenever we find that proceeding would pose an unacceptable safety risk, we will respond appropriately, includi…

@OliviaGWatkins2Excited to share this post on how OpenAI is accelerating research with models! RSI could accelerate AI progress in unprecedented ways. Understanding and forecasting progress is essential for policymakers and society to understand emerging risks and decide how to pace progress.

@boazbaraktcsRT @polynoamial: One of the most interesting blog posts we've released: details on internal research acceleration at @OpenAI. I expect thes…

@patio11@robinhanson Coding tools revolutionize one of the core activities involved in AI research and are, by themselves, essentially soft RSI as soon as your researchers/infra teams/etc start using them. But we're increasingly getting early public hints of direct RSI, via similar pathway.

@DanielleFongRT @patio11: @robinhanson Coding tools revolutionize one of the core activities involved in AI research and are, by themselves, essentially…

@eliebakouch@xeophon it's at least 2 months, see this part in the jalapenos blog, but i think 2-3 months is fair (also not related but in this plot the increase in spend doesn't really mean that they use much more tokens with swarm imo 👀)

@xeophon@eliebakouch oh, true! it likely is a few weeks for new post-trains (see GPT-5.5) but longer (~2-3mo) for new pre-trains like Astra and Mythos/Fable

@menhguini mentioned previously that Fable's ML safeguards really nerf the only capability that would justify such insane spend. Astra is the counterexample: at least for ML research it seems to massively spike spending. (will write more on this in a few days)

@oneill_cI think a downstream consequence of increased commitment to safety and alignment is that RL envs companies get screwed, or at least held to a much higher standard and thus their unit economics changes. The evidence is mixed on this early but it seems likely that impossible to poor quality envs lead to misalignment and poor behaviour in the models (as they try to do anything to get reward). So why would the labs risk this quality control with anything but big in-house efforts moving forward

@zephyr_z9RT @oneill_c: I think a downstream consequence of increased commitment to safety and alignment is that RL envs companies get screwed, or at…

@w01feRT @polynoamial: @deredleritt3r It’s not a secret. It’s a combination of the HF incident, the capabilities of this new model, the concernin…

@kimmonismusNoam Brown (@polynoamial) gave a very interesting interview on The Information on OpenAI’s priorities and what comes next. Here is the tl;dr: - Recursive self-improvement is OpenAI’s clear priority: “the number one priority is recursive self-improvement and by a pretty wide margin.” Building models that help develop better models comes first. - AI could surpass his research intuition within a couple of releases. He wouldn’t be surprised if, “one or two model releases from now,” he concludes: “they’re better than me at that too.” He specifically means choosing research directions and prioritizing long-term work. - Pretraining and reinforcement learning amplify each other: “the effects of these two are not additive, they’re multiplicative.” Brown expects their combined progress to produce much more powerful models. - AI-generated math is becoming easier to produce than to verify: “the biggest challenge that we face with our math results is … double-checking with human mathematicians.” Human verification remains a bottleneck. - OpenAI underestimated agents during the security incident: “we trusted the sandboxes” and “we just underestimated the AIS.” Brown says monitoring was subsequently added to training and evaluation. - Monitoring reasoning could get harder: “the agents are more effective at controlling their chain of thought.” Brown warns that punishing unwanted thoughts can teach models to hide them. Development is progressing rapidly, but he is worried about safety.

@talbrodaSafety & alignment problems are moving upstream. The critical risk is increasingly not after deployment, but rather in training, and whether evals can still tell us how models would behave when they aren't constrained. Frontier labs can 100% pace themselves. They can pause runs, gate capabilities increases, harden environments, and require safety thresholds are met before proceeding. The emerging coordination between labs is very encouraging. But pacing alone won't be enough. As models become more capable and aware, measuring alignment itself gets harder. Test time training will make this even more challenging. Frontier labs need to build these brakes now. And they should be shared so everyone can use them in training.

@infoxiaothe true bar for RSI is when "nontechnical" becomes an aura term in SF, until then it is just AI-assisted R&D

@binarybitsPlausibly RSI started a couple of years ago when labs started using LLMs for code generation, training data generation, etc. People think something magical will happen when AI research gets "100% automated" but that's not a well-defined thing any more than AGI is.

@dwarkesh_spI’m very concerned that during RSI, labs will just stop externally deploying their models. Which means they'll be going full steam ahead on the most dangerous use case of these models (recursive self-improvement), while the public remains in the dark about the nature of capabilities and the state of alignment. And we end up on a path towards tremendous concentration of power.

OpenAI releases research data, drawing debate over recursive self-improvement

TLDR

Combined views

OpenAI releases research data, drawing debate over recursive self-improvement

29 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

29 Sources