I think an AI pause is a bad idea. Isn't mythos their most aligned model ever? This feels like it will be self correcting, if RSI hits itll have better ideas for handling RSI than we do
Bengio warns that systems could autonomously upgrade without human oversight
I think an AI pause is a bad idea. Isn't mythos their most aligned model ever? This feels like it will be self correcting, if RSI hits itll have better ideas for handling RSI than we do
Many users harshly criticized Anthropic and researchers urging pauses on AI recursive self-improvement as insane or totalitarian overreach, while some supported verification mechanisms or the advocates' efforts.

@a_karvonen Be aware of Anthropic’s long history of safety theatre. Don’t confuse marketing with virtue.

@ohabryka https://www.anthropic.com/institute/recursive-self-improvement
Near the end

Come on. Even with your recent safety stance you know this is nonsense.
Reality is bound by time, multiplicity, compute and many, many other factors.
There is no free lunch even for neural networks.
People are recursively self improving and limited by countless factors, not the least of which is the subject object problem.
There is no faster way to know if a drug will have side effects in twenty years or whether something got demonstrably better at writing, versus something like code which has a fast feedback loop and right or wrong answers.
Fast take off is utter and complete nonsense and you know it.
At least argue from a grounding in the reality you most definitely have instead of an argument that you absolutely know is not true. It totally undermines the veracity of your message.
If leading AI companies are indeed approaching the point of recursive self-improvement, a coordinated, verifiable, and universally applied pause is probably the only responsible solution to mitigate several major AI risks; at least until safety guarantees are developed and demonstrated. Ensuring that such a moratorium is respected would require sincere collaboration between various countries and companies, but I definitely believe it is achievable if others follow in @AnthropicAI's footsteps.

@Yoshua_Bengio No, a pause is not reasonable, and no, it is not going to happen.
@deanwball I still don't see a good argument for a pause of RSI is coming. What would a pause help us get ready for, superintelligence would be able to figure out better policies than we would, and escape our attempts at alignment. Its stop or go.
What I don’t understand is what the pause conditions would be. “Woah dude, this seems weird” is hard to operationalize in words, so the decision comes down to vibes. Once you create the button, there will be a temptation to press it, and there may not be a “resume” button.

yeah, almost everything about this is under-specified, eg: 1. what conditions would justify a pause? 2. what would we do during the pause? 3. how would we know we had completed everything we needed to do? who would decide the things had been done? 4. what is it, exactly, about rsi that justifies this intervention? "things are moving too fast" is not close to good enough of an answer imo. it sounds like the kinds of justifications that were given for many of the most disastrous laws in us history, eg our environmental protection regime.

This is a significant moment because even leading AI labs are beginning to sense the asymmetry Pratīhāra’s AI Law identifies:
recursive admissible relational compression allows cognition to scale faster than civilizational coherence.
What Anthropic is implicitly acknowledging is not merely a technical problem.
It is a control problem born from acceleration itself.
Once AI systems begin recursively assisting: model design, code generation, optimization, research, architecture discovery, and infrastructure improvement, cognition enters feedback loops where capability growth may outpace human oversight.
But the deeper issue is even more serious.
Humanity still assumes that more intelligence naturally leads toward better outcomes.
Hindu traditions never assumed this.
Intelligence without Vimarśa amplifies fragmentation as easily as coherence.
So the real danger is not simply “AI becomes smarter.”
It is that optimization systems begin recursively improving while remaining detached from: Dharma, self-recognition, wisdom, civilizational restraint, and conscious orientation.
That is why slowing down alone may not solve the problem.
Because the bottleneck is not compute.
It is metaphysical maturity.
Civilization currently possesses: planetary-scale technology, industrial cognition, algorithmic persuasion, and accelerating recursive intelligence, while remaining deeply fragmented psychologically, spiritually, politically, and civilizationally.
That asymmetry is historically unprecedented.
Abhinavagupta’s spectrum becomes almost diagnostic here:
Śakti-vikāsa scales naturally and rapidly once mechanisms exist.
But Vimarśa does not automatically scale with power.
Without conscious orientation, recursive optimization systems tend toward entropy: attention capture, resource extraction, social fragmentation, surveillance, addiction, synthetic reality, and eventually destabilization of meaning itself.
So Anthropic’s concern is probably correct.
But the real issue is larger than “AI safety.”
The question is whether civilization can develop coherence faster than cognition industrializes.
Because once recursive intelligence loops exceed the speed of civilizational adaptation, societies may lose the ability to meaningfully orient the systems they themselves created.
That is not science fiction anymore.
That is the emerging structure of the AI age.
— Hemu Bharadwaj Pratīhāra’s AI Law 6 June 2026

@a_karvonen Link?

@littIeramblings They loudly and publicly refused to let the Department of War use their models for war crimes and OpenAI spent like 30 seconds deciding if they'd like to do it instead

@a_karvonen you've heard of proof-of-work, now what about proof-of-not-working

@MancerAI_ @Dan_Jeffries1 @Yoshua_Bengio I only used LLMs because they're obvious starting points for labs. LLMs can do any research, but not new. To find novel ideas requires a mechanism for coming up with new hypotheses from recombining existing data. That doesn't exist. Otherwise, search is too expensive.

@littIeramblings That’s certainty a separate argument you can make. I’m disagreeing with the position that they could simply unilaterally pause to clearly solve the problem. This is all very messy, and some people are pretending it’s not.

@Yoshua_Bengio No, coding agents are still trash and nothing is approaching, what’s a stupid idea. No one cares about sloppy apps written by Claude. Where are anti aging drugs??? Stopping LLMs is pure stupid, they are nice tool to get some basic quality life improvements.

@Yoshua_Bengio Not going to happen.

@Dan_Jeffries1 @Yoshua_Bengio RSI, as I understand it, is a hypothesis search process wrapped in an agentic loop. Which is an optimization process over existing architectures. I don't think that optimization can generate enough novelty for takeoff, because you probably need to move away from LLMs for that.

> I think unilaterally pausing would be much more commensurate with Anthropic’s stated beliefs than writing a blog post
My understanding of Dario Amodei's model is that: 1. Anthropic is on track to solve alignment in time. 2. Anthropic will apply those safety techniques. 3. None of the other labs will apply the techniques. 4. If a lab gets to ASI first and doesn't apply those techniques, then extinction might happen. 5. Anthropic is in the lead. 6. Dramatically giving up their lead does not guarantee with high certainty that all other labs stop.
Assuming that this model were true, then I get why it's better for them to do nothing or write weak blog posts or secure Anthropic's lead over other companies via lobbying, etc.
Tbc I don't quite get why they don't write better blog post or do better political lobbying. So my model at least somehow cannot match what Anthropic-as-a-whole believes.

@littIeramblings (I also separately disagree with the idea that if they loudly and publicly paused at this point it would cause the others to follow, but while I think the idea is wrong, I don’t think it’s crazy)

@JimDMiller @Yoshua_Bengio What does it even mean? Our entire civilization has been “recursively self improving” for a very long time. Our technologies are always used to improve our technologies.

@Yoshua_Bengio What’s fascinating is all the people calling for a "pause" (including Ant) never care to explain how something so drastic would be implemented in a multipolar race of rival empires
Bengio warns that systems could autonomously upgrade without human oversight
I think an AI pause is a bad idea. Isn't mythos their most aligned model ever? This feels like it will be self correcting, if RSI hits itll have better ideas for handling RSI than we do