Anthropic proposes a coordinated global pause on AI development to manage risks from self-improving systems
Analysts note the warning has received surprisingly little attention.
Positive users appreciate calls for pausing AI development over self-improvement risks, while negative users dismiss them as unrealistic pipe dreams, fundraising hype, or historically ineffective.
Most Activity
If leading AI companies are indeed approaching the point of recursive self-improvement, a coordinated, verifiable, and universally applied pause is probably the only responsible solution to mitigate several major AI risks; at least until safety guarantees are developed and demonstrated. Ensuring that such a moratorium is respected would require sincere collaboration between various countries and companies, but I definitely believe it is achievable if others follow in @AnthropicAI's footsteps.
@krishnanrohit it's almost like they r in fund raising mode and trying to create hype
@hannahcrileyy Is in not just them self hyping for an IPO?
@krishnanrohit I think companies are best viewed as collections of people with different views. I think it’s a mistake to say Anthropic wants to pause. it’s correct to say one guy who writes blog posts at Anthropic wants to pause, but he’s not a revenue generator.
@hannahcrileyy I vaguely remember the inventor of the Segway scooter saying something similar 25 years ago
@hannahcrileyy Why do you care that they said it?
@krishnanrohit best way to get your way is to make everyone else feel like they’re getting the better deal
@Yoshua_Bengio the companies closest to recursive self-improvement are also the ones who would have to agree to stop. that's not a coordination problem, that's a hostage negotiation where everyone is holding their own leash
@hannahcrileyy nearly impossible to keep up with all the horrifying developments that should be front page news these days 🙃
@gnarlydudes Idk!
@Yoshua_Bengio I think a big challenge here is verifiability - would China or the US trust each other to have actually paused given the immense value of these systems to state security? No matter what, don’t you think there’ll be some black program still working on it?
@replyallguy Companies don't just let a random persons view become the house view
@Yoshua_Bengio Regulatory pipe dreams never work in this space. They’ll just move offshore.
@Yoshua_Bengio I appreciate your optimism!
@Yoshua_Bengio Agreed, this should have happened sometime back. But on the side note, precisely what Anthropic is talking about. Have they opened any particular findings?
@Yoshua_Bengio yeah but whos actually gonna volunteer to pause when the race is this tight
@krishnanrohit Plan B for if there's no slowdown. Or maybe they want to RSI just a little bit, as a treat
@Yoshua_Bengio the lab that made the sandbagging model asking everyone to slow down feels like a warning in itself
@gnarlydudes @hannahcrileyy Chris Olah has entered into a Holy Alliance with Pope Leo XIV in an attempt to safeguard humanity from the potential dangers of AI technology that would result from a lack of regulation and moral responsibility regarding AI. He already declined to let Trump misuse AI.
@Yoshua_Bengio calls for a pause have historically worked so well we might actually be even more behind on safety than progress