Dario Amodei just published an unusually candid essay about where AI is heading.

The tl;dr with quotes.

His new piece, Policy on the AI Exponential, reads more like a warning from the person building the thing.

The core problem is timing. AI moves on an exponential. He is very clear about it. Lawmaking moves like Tolkien's Treebeard, the tree so slow it takes a full day just to say hello to another tree. By the time Congress acts, Amodei writes, AI can go from "an amusing toy to the full country of geniuses."

His timeline is short: "If these scaling laws continue for only a year or two longer, we are likely to get what I've called Powerful AI, or 'a country of geniuses in a datacenter'."

And he thinks the evidence has already turned. Pointing to the cyber risks of Claude Mythos Preview, he writes that "its broader significance is that it proves beyond doubt that AI models are now tools of global and national strategic consequence." So he wants binding rules modeled on the FAA. Mandatory third-party testing of frontier models. Government power to block or reverse a release it judges unsafe. This from the man whose own models would be the ones getting blocked.

The part I keep rereading:

He's genuinely split on the economics. The upside he describes is enormous: "If AI achieves the ability to do most cognitive tasks far better than humans, it stands to reason that it could result in extremely rapid and robust economic growth via the acceleration of science, technology, and operational efficiency. The iterative ability of AI to build even better AI may supercharge that growth even further."

But he won't wish the other side away: "there's a decent possibility that, despite all our efforts, AI still causes significant enduring job loss- and that this may be an intrinsic property of the technology and the way it broadly replicates human cognition." His fixes run all the way to UBI and higher capital gains taxes.

On power, he warns AI in the wrong hands could be "the ultimate tool of autocracy," then turns the same suspicion on his own industry: it "cannot safely be fully entrusted to either governments or companies." Anthropic included.

And he refuses to treat public fear as a PR problem. "People are worried about AI because they correctly perceive that its risks are real." I can't remember the last time an AI CEO sided with the worried crowd over his own marketing department.

The mood throughout is urgency, not victory. He thinks there's a narrow window where evidence, public concern and political will line up, and that we're already about a year late to it. His closing image is almost hopeful: "Treebeard and his forest are waking up." The only question that matters is whether they wake up fast enough.

I applaud @DarioAmodei's push for mandatory third-party evaluation of frontier AI, including the power to block unsafe releases.

But in my opinion his proposal has an important gap -- the main focus is on commercial deployment, but the biggest risks may emerge before any deployment decision exists.

Take Dario's own FAA analogy. The FAA doesn't wait for an airplane's commercial debut - instead experimental aircraft need airworthiness certificates, test flights have restrictions, and even test pilots require certifications. And AI needs this even more than airplanes do, because AI systems are agentic in a way that drugs and airplanes are not!

For example, a plane in a hangar doesn't do anything while it is built... you don't need to do anything before you try to fly it. But now imagine you had a plane that could just take off and fly itself without human authorization and crash before anyone realized what had happened. That kind of plane would need a very different kind of security measures!

This may sound crazy for a plane but it is already beginning to happen with the most advanced AI. AI systems can take actions, including unintended and unauthorized actions, and are increasing in their sophistication to do so. In my opinion, fixating on evaluating AI systems right before commercial deployment is the wrong way of thinking about things and the US government will need much more wide-ranging visibility into AI systems as they are trained and used internally, where risks can first appear. Given that these internal AI systems are often the most capable, I suspect this is where the risks will most likely be found!

The risk doesn't wait politely on the runway for a regulator's pre-flight inspection. Anthropic's own data shows that the most consequential capabilities are observed internally, on systems that wouldn't undergo scrutiny in Dario's proposal.

So yes to mandatory testing and deployment-blocking authority. But the government also needs visibility into frontier systems as they're trained and used internally!

Dario Amodei just published a super long blog, calling for an urgent policy overhaul because he thinks frontier AI is moving faster than governments can regulate it.

He wants: - Mandatory pre-release testing and independent auditing of frontier AI models, with government power to block deployment when models pose serious cyber, biological, autonomy, or automated-R&D risks.

- Stronger security rules for AI companies, including protection of model weights, regular red-teaming, penetration testing, and rapid reporting of critical safety incidents.

- He wants governments to prepare for AI-driven labor disruption through better measurement, pro-employment incentives, wage support, training, and possibly long-term income support funded by AI-driven growth.

- Democracies should coordinate globally on AI safety, chip supply chains, export controls, shared benefits, mutual defense, and safeguards against AI-powered repression.