New essay: Do AI Risks Require Extraordinary Government Intervention?
In a recent essay, @DKThomp engages with AI as Normal Technology. He agrees with our thesis about AI's slow labor market impacts, relying on the fact that GDP growth has so far been average, unemployment is below 5%, and even jobs that seemed vulnerable to automation show rising employment and wages. He concludes that so far, the macroeconomic picture is consistent with what we would expect from a "normal" general-purpose technology.
But when it comes to AI risks, he is far more bearish. He points to examples of cyber- and bio-risks and expresses pessimism about AI quickly becoming dangerous across many new domains. He argues that AI's emergent capabilities make it fundamentally different from previous technologies, and that this difference justifies “extraordinary” government responses including restrictions on what companies can release.
In a new essay, @random_walker and I lay out the downsides of extraordinary government intervention in response to new technology. We discuss proposals for improving resilience that do not require such intervention. We also discuss why governments have so far been reluctant to invest in resilience. In short, resilience requires us to get better at the *normal* process of policymaking. But sclerosis in the federal government and the ease of justifying interventions on AI companies rather than society at large make extraordinary intervention seem appealing, despite its limitations.
What happens next? Policymakers can either get their act together and invest heavily in improving resilience, or be forced to take extraordinary actions (such as on AI nonproliferation) that are more onerous and less effective.
A Recap of AINT’s Arguments and Thompson’s Areas of Agreement and Disagreement
Many people, including Thompson, have found AI as Normal Technology to be a useful framework for thinking about AI's economic impacts while being unconvinced by our views on safety. In the AINT essay, we make different arguments about slow labor market impacts and resilience to misuse risks.
The labor market argument rests on the speed of diffusion: there are many speed limits between a new AI capability and its economic impact, including the need to build products, change organizational workflows, and navigate regulation. This has proven helpful both to understand why claims of rapid and widespread job displacement (such as Amodei's claims about an imminent white-collar bloodbath) are unlikely to materialize, as well as to identify bottlenecks to diffusion that could hinder the beneficial adoption of AI.
But our argument about AI's misuse risks depends on the offense-defense balance: whether attackers or defenders benefit more from a given capability, and our ability to build resilience in response to misuse risks. Notably, an attacker does not need to go through the slow process of organizational adoption to cause harm. We argue for building societal resilience to intervene on misuse risks and identify many interventions that we should take to reduce AI risks.
Here, Thompson disagrees with us. One reason he disagrees is that all general-purpose technologies bring about new risks, which are hard to reason about from previous patterns. But even in the history of general-purpose technologies, Thompson argues that AI seems particularly "abnormal" because of risks that are emergent and unknown even to AI developers. And this justifies plans by the government to treat AI as an abnormal technology, since it "compels the government to create extraordinary regulations that prevent private companies from selling their products and services on the grounds that they’re too dangerous."
Thompson doesn't define precisely what he means by extraordinary government action, and the specific interventions he envisions remain somewhat vague. Since we'll be arguing against such interventions, we want to be clear about what we mean by the term. We think of extraordinary interventions as a spectrum defined by three factors that apply to any powerful technology, not just AI.
First, extraordinary interventions tend to be precautionary. They restrict activity based on anticipated harms rather than realized or demonstrated ones. This doesn't mean that precaution is never warranted, but it does mean the justification needs to be stronger, since we are restricting activity without clear evidence that the harms will materialize (or that they'll materialize exactly as we predict). Even then, an intervention is more extraordinary when viable alternatives exist that can address the risks while being less restrictive, such as investing in resilience.
Second, extraordinary interventions impose restrictions on the liberty of actors who are not directly responsible for the harms in question. When governments restrict what AI companies can release, the burden falls not on the malicious actors who cause harm, but on companies that build tools that could, in principle, be misused. This is especially pertinent for dual-use technologies. Because these tools have widespread beneficial applications, restrictions on companies can cut off beneficial access for the broader public in order to prevent misuse by a small number of bad actors.
Third, extraordinary interventions bypass normal processes of governance, and instead rely on unilateral authority such as emergency declarations or executive orders, even though the governance processes being bypassed exist to ensure that restrictions on liberty are subject to democratic accountability.
An intervention need not satisfy all three of these criteria to count as extraordinary. But the more of these factors that are present, the higher the bar should be for justifying it.
Extraordinary Government Action is Extraordinarily Costly
We agree that AI poses misuse risks. But our experience from regulating "abnormal" technologies shows how burdensome extraordinary government interventions can be. The enforcement of nuclear nonproliferation has required the IAEA, the Non-Proliferation Treaty, decades of diplomacy, ongoing investments, and even military confrontation. The tolls of these interventions were enormous, but the approach was at least somewhat enforceable because nuclear weapons depend on enriched uranium, a physical bottleneck that is genuinely hard to get around.
AI is different from nuclear weapons. For one, there is no equivalent "physical" bottleneck. The core techniques for building AI systems are well known. Adversaries (especially nation-states) can match frontier capabilities within months. Any nonproliferation regime for AI would face constant erosion. In the face of this challenge, how could governments maintain nonproliferation? Some proposed interventions, such as the rumored executive order on voluntary commitments by AI companies for predeployment evaluations, are relatively low on the scale of extraordinary interventions. Done well, they could allow us to tilt the offense-defense balance by giving defenders more time to prepare for new capabilities.
The U.S. has also enacted export controls on chips. We think this is a modest intervention: countries routinely restrict exports of sensitive goods to maintain their lead in innovation, and reasonable people can disagree about where to draw the line. But when it comes to preventing dangerous capabilities from being widely accessible, export controls are far less effective. Open-weight models and widespread API access from frontier labs mean that the gap between frontier and publicly available capabilities is at most a few months, not years.
If the most that nonproliferation can buy us is a few months, the urgent priority should be investing in resilience so that we are better prepared when those capabilities inevitably become widely available, as we discuss below. On the other hand, if governments try to maintain nonproliferation as a way to deter the availability of advanced AI capabilities for misuse, their interventions will necessarily get more demanding.
To truly enforce nonproliferation, governments would need to restrict access to open-weight models and even API access to capable models. This would require licensing regimes that give governments ongoing authority over which models can be released and restrictions on open-weight models. We might quickly enter a state where governments exercise control over what AI research and products can be shared publicly. In fact, we have already seen concerning examples of such control, such as Anthropic's designation as a supply chain risk, and recent rumors about licensing requirements for AI companies.
To avoid this slippery slope, those proposing extraordinary interventions should be clear about where they would draw the line. Would they support restrictions on open-weight models? What about requiring approvals for each new model release, or restrictions on the movement of researchers who build frontier AI across countries? If proponents cannot specify the limits of what they are calling for, it is reasonable to expect that the demands for increases in the government's ability to take unilateral action will keep escalating as capabilities advance.
For nuclear weapons, once the nonproliferation regime was built, it did not need to be rebuilt every few years. But AI is not the last digital technology with powerful dual-use properties. New technologies would raise similar questions and potentially demand similar or escalating responses. So the "abnormal technology" framework for regulating AI would start to look less like a targeted response to a specific risk and more like a permanent expansion of government powers over what citizens and companies can build, publish, and research.
We have seen this debate play out before. Each new technology raises questions of what restrictions to liberty are appropriate to mitigate harms. This history tells us we shouldn't automatically default to imposing extraordinary government interventions.
The internet allowed people to access information about how to build bombs, and in 1995, after the Oklahoma City bombing, the late Senator Feinstein introduced a bill that, in its first draft, would have criminalized the distribution of any information on the internet describing bomb-making materials or processes. (When the bill was eventually passed, it had a narrower scope, requiring offenders to have knowingly aided a crime through their instructions.)
The federal government also tried to restrict access to encryption software, arguing it would help criminals communicate beyond the reach of law enforcement. It imposed export controls and proposed requiring backdoors in encryption so that the government could always access private communications. It even started a criminal investigation against a programmer under the Arms Export Control Act for releasing encryption software.
These restrictions were eventually rolled back through a combination of court rulings and executive action. Encryption became the foundation of digital security, enabling e-commerce, online banking, and many other applications.
Yet, at other points in history, we have accepted increases in government powers in response to new technologies. The widespread acts of terrorism after the invention of dynamite catalyzed the expanding government surveillance apparatus by the FBI.
The question is whether this level of intervention is necessary to address AI risks. One response that does not require extraordinary government intervention is improving resilience.
Resilience is an Under-Emphasized Defense Against AI Risks
Resilience is the capacity of a system to withstand and adapt to harm. It was one of the main defenses we proposed in the AINT essay. Unlike extraordinary government actions like restrictions on what companies can release, resilience does not impose costs on AI companies. Instead, it focuses on improving our capacity to respond to and recover from AI risks, regardless of when or where they occur. For past technological harms, improving the resilience of systems has been key to reducing harm.
Consider cybersecurity. The internet created entirely new classes of attacks, such as worms that spread through networks, causing billions of dollars in damages. The way we defended against these risks was not by restricting access to computers or the internet, but rather through bug bounties to incentivize people to report vulnerabilities to developers, improving browsers and operating systems, automated testing, and better patching practices. All of these improved the resilience of our cyber infrastructure regardless of where the risks arose.
Automated vulnerability detection tools are another example of technology enabling new kinds of attacks. Tools such as fuzzers and symbolic execution engines have been "superhuman" at vulnerability detection for years; they can detect vulnerabilities at a scale humans cannot match. Yet, they have been freely available on open-source repositories. Since defenders had access to the same tools, they became core defensive tools, largely funded by the cyberdefense ecosystem. In fact, defenders had structural advantages in using these tools effectively, such as deeper access to the systems being tested. This in turn led to better protections for cybersystems.
This shows how we have managed the transition from unaided vulnerability detection to vastly superhuman detection without requiring extraordinary government intervention. We agree that LLMs bring real improvements to vulnerability detection, but they build on top of decades of tooling that is already far beyond what any human can do unaided. If we have already absorbed the transition to such tools, it is worth asking whether the additional capabilities of language models call for extraordinary interventions.
To be clear, these transitions were not smooth or painless. For a brief period, the attacker-defender balance was completely upset. The idea of viral spread of malware and the resulting asymmetry was unprecedented. 15-year-olds could create devastating cyberattacks that brought down top e-commerce websites and search engines. It is plausible that government intervention to improve cybersecurity could have reduced the harm caused to individuals and businesses during this period.
AI's use for cyberattacks might once again upset the offense-defense balance, and we don't think the AI transition will be smooth by default. Many systems that are currently under-defended, including schools, hospitals, power grids, and small government agencies, will be at real risk. Efforts like Project Glasswing and OpenAI's cybersecurity grant program are important but insufficient on their own. While restricting access to advanced AI systems might be helpful in the short run, it is not a silver bullet in a world where open-weight models are only months behind the most capable closed models.
Addressing AI's cyberrisks requires investment in resilience. That means AI-assisted red-teaming not just for tech companies, but for schools, hospitals, power grids, small businesses, and government systems that currently lack the capacity for defense. We should also incentivize professional security experts to find and report vulnerabilities (with or without AI use), such as through bug bounties that cover more than just tech company products. These efforts are not automatic; they require investment and planning.
The good news is this work is starting across many different industries. While this requires significant investment, it is still far less burdensome than enforcing a strict nonproliferation regime. The bad news is governments have a lot left to do to truly make the transition painless.
The same analysis applies to biosecurity. AI may lower some information barriers, but bioattacks depend on many downstream steps: procuring materials, accessing specialized equipment, and applying tacit know-how. We can intervene on those downstream steps *now* to increase societal resilience to bioattacks without imposing strong controls on AI development, such as by implementing better screening of synthetic biology orders, using AI to evaluate the riskiness of new compounds, tracking access to dangerous materials, and offensive red-teaming (such as asking trusted experts to attempt to use AI for manufacturing dangerous compounds) to find gaps in these efforts. These help regardless of whether biorisks are from AI.
If Resilience is So Helpful, Why Haven't We Prioritized it Already?
Importantly, resilience does not require extraordinary government intervention. It only requires us to get our act together on the "normal" process of policymaking and execution.
The problem is we are not great at normal policymaking. We suspect one reason why the resilience approach seems unappealing is that it requires polycentric governance in which many decision makers work harmoniously together. This is a tough sell given that state capacity in the United States has been hobbled by decades of accumulating veto points and creeping proceduralism. As a result, unilateral actions by the executive branch are often seen as the way out for developing and enforcing AI policy.
Investing in resilience requires action and investment by a much wider set of actors than nonproliferation does. For resilience to be effective, the government needs to legislate and allocate funding, collaborate across agencies, build early warning systems, and serve as a resource hub for downstream actors and rapidly disseminate information to help them shore up their defenses. The U.S. federal government isn't exactly well known for being competent at this set of tasks.
So it shouldn't be a surprise that when we look at the government's track record, policy responses for resilience have been underwhelming. In fact, even when there are areas of agreement between proponents of the normal and "abnormal" views on AI, such as requirements for transparency, auditing, and safe harbors for safety research, we are yet to see concrete federal action. (There are some efforts, like the recent letter from representatives urging the federal government to improve coordination on cyberrisks. But for now, it is just a letter with recommendations; it remains to be seen if any action will be taken based on these suggestions.)
It is in this context that extraordinary government actions, such as nonproliferation, look tempting to address AI risks. These are morally satisfying since they primarily impose burdens on the companies that create these risks. They are also tractable, since they only require unilateral action from the Executive, such as invoking the Defense Production Act. The main question is whether we can achieve better outcomes through such actions, or whether we should invest in improving the "normal" process of governance.
While we understand the reasons for taking the former approach, we lean towards the latter. Getting our policy act together is hard, but important—not just for this round of AI policy and misuse risks, but for all future interventions on technological harm, and for the democratic process to work more generally.
On the other hand, if our main defense against AI risks is nonproliferation, a single technical breakthrough that makes models cheaper to train could be enough to cause instability, especially in a world where we don't also invest in resilience. The damage will be that much greater when the dam eventually breaks.
