MIRI's Rob Bensinger and David Krueger endorsed the premise.
I'm in favor of slowing down this train before we go off the cliff IF, and I feel it's important to emphasize this, IF we can do so *elegantly*.
Anthropic co-founder Jack Clark would be in favor of "elegantly" slowing down AI development to avoid the development of a powerful technology at "breakneck speed"
Some users back Anthropic's push for international treaties to halt superintelligence races, while many others dismiss the calls as regulatory capture, cheap talk, or an IPO-timed exit strategy.
One big quibble is that they aren't thinking big enough. With sufficient intelligence you *can* predict pretty well what a drug would do over a decade of use, and/or sidestep that problem entirely by designing de-novo more-predictable biosystems from scratch, etc.
And my top quibble is that the tone reads like "RSI could happen but don't fret too much, it'll probably be fine" rather than "omfg we're possibly on the brink of AIs that make smarter AIs that make smarter AIs, society needs to *act*". But it's a step in the right direction.
Anthropic just called for a global way to slow frontier AI because its own models may be approaching recursive self-improvement, where a system helps build a stronger version of itself without direct human control.
Future models will become so good at research, experiments, debugging, and training design that humans will stop being the main bottleneck.
Once that loop starts, progress could shift from human-paced engineering to machine-assisted improvement, which makes every safety test, law, and lab policy feel late by default.
Anthropic says this has not happened yet, but warns that the jump may arrive before governments, companies, and researchers have a trusted way to measure or restrain it.
The hard part is verification, because a huge AI training run is easier to hide than a weapons site, and any lab that secretly keeps training while others pause could gain the lead.
Anthropic is now ~$1T, may reach $50B annualized revenue, and competes fiercely with OpenAI, so every safety claim also lands inside a giant business fight.
---
anthropic .com/institute/recursive-self-improvement
Anthropic just disclosed that Claude now writes more than 80% of the production code it merges.
Before Claude Code reached research preview in 02-25, Claude wrote only low-single-digit merged code, while output per engineer has since risen to 8x the 2024 baseline.
The shift comes from agents that edit files, run tests, inspect failures, spawn helper agents, and keep working across longer tasks instead of only suggesting snippets.
Anthropic says reliable task length is doubling about every 4 months, with Mythos Preview reaching at least 16 hours and open-ended Claude Code success hitting 76%.
i.e. Claude Mythos Preview could stay useful on a task that would take a skilled human roughly 16 hours of work
Claude also moved from a 3x training-code speedup to 52x, while a skilled human reached about 4x in 4 to 8 hours on the same setup.
The remaining human edge is research judgment: choosing the right problem, trusting the right result, and knowing when an experiment is dead.
@j_asminewang @tyler_m_john - it's crazy that this was less than 3 years ago, and the entire debate was ridiculed as completely irrelevant to what could actually happen. https://forum.effectivealtruism.org/s/vw6tX5SyvTwMeSxJk/p/3hSEQnEN2D3SSzHWn
@j_asminewang I think it's excellent and a little notable that you personally say so.
@So8res Saw this here first, so linking the blog in case anyone else in the same boat: https://www.anthropic.com/institute/recursive-self-improvement
@j_asminewang @tyler_m_john Also, this post was fairly prescient - https://forum.effectivealtruism.org/s/vw6tX5SyvTwMeSxJk/p/BFbsqwCuuqueFRfpW - though I think point 2 was wrong, not as a prediction, but because the downside was much smaller than the benefit of actually widening the Overton window.
@So8res maybe that is their exit strategy. AGI is not comming, debt is accumulation, their margins become smaller. This could be a great narrative to say we stopped developing agi for the greater good therefore gov should backstop our debt.
@So8res What is it exactly that you think you can do to prepare for a recursively self improving ASI?
The truth is all you can really do is make one heroic attempt to get the alignment right.
That's what Anthropic is already doing.
@sopharicks Make an international treaty banning the race to superintelligence. (Example: https://ifanyonebuildsit.com/treaty.) Many people underestimate how possible this would be once world leaders understand the danger.
@j_asminewang 都是一帮没见过世面, 大惊小怪之流, 成事不足, 败事有余之徒, 五年之内草草收场.
@PJOPJOPJOPJO @So8res You can make humans smarter first; smart enough that they actually have a shot at getting it right.
@So8res What in your opinion can society do? How can it step up?
@j_asminewang So does OpenAI agree it’s time for a slowdown?
@PJOPJOPJOPJO Help institute an enforced international ban on superintelligence until we're much, *much* more ready.
@So8res Seems pretty cheap-talky to me.
@j_asminewang
@So8res that's the most honest thing they said, this week.
@rohanpaul_ai Gatekeeping and hungry for control. Dario thinks he’s God at this point.
@So8res the honest answer is most ppl are just okay with being on the same page as corp giants