California’s Government Operations Agency is recommending emergency AI shutdown powers, mandatory safety standards and independent oversight for large frontier AI developers. It also wants the state to have authority to limit automated AI research when it creates unacceptable risk. Gov. Gavin Newsom’s account highlighted the recommendations on November 18. SourceSource
These are recommendations for legislation, not newly enacted powers. The agency’s November 16 report, AI Safety in 2026 and Beyond, was developed in consultation with the California Office of Emergency Services and outside experts. Source
Shutdown powers, with conditions
The report recommends letting the state order an emergency shutdown when an AI model poses an imminent risk of catastrophe or a major loss-of-control event that its developer cannot or will not address. The state would direct the developer to activate a verified containment protocol. Source
Developers could challenge an order in court, but it would take effect immediately and remain effective pending judicial review. The report says shutdowns should be no broader than necessary, used only when a lesser intervention would not work, and lifted once the danger is no longer imminent. Source
Every large frontier developer would need a tested risk containment protocol covering interventions at several levels: revoking access, stopping individual AI agents, isolating computing resources and, as a last resort, shutting down broader model operations. The protocol would identify responsible people, arrangements with outside providers and conditions for safely restarting. A senior company official would certify it, with independent verification. Source
The report also calls for testing whether controls could be disabled or shutdown confirmations falsified, and for independent confirmation that a shutdown actually worked. A reassuring dashboard is not the same thing as a stopped model. Source
For open-weight models, whose underlying parameters are available for others to run, developers’ shutdown obligations would apply only to deployments within their control. They would not be expected to remotely stop independently hosted copies they cannot reach. That is not a blanket exemption from the report’s evaluation and transparency proposals. Source
Independent oversight, with actual access
The proposed baseline safety, security and transparency standards would apply to all large frontier AI developers, not every company using AI. California would set the standards; licensed independent verification organizations, or IVOs, would develop and update the technical methods for checking compliance. Source
These organizations would conduct ongoing oversight with access to the people, models and information needed for meaningful assessments. They would send unredacted assessments to the state, which would retain responsibility for enforcement decisions. The company’s own assurance would no longer be the final word. Source
According to the report, SB 813 already establishes an IVO framework but does not itself require developers to engage one. The recommendations would make that oversight mandatory for large frontier developers. Source
A brake on AI building AI—not an immediate cap
The report warns that automating AI research and development could lead to recursive self-improvement, or RSI: a self-sustaining process that accelerates capabilities without additional human input. Source
Large frontier developers would report quarterly how they allocate computing resources, including to AI research, training, safety work and customer services. IVOs would verify those disclosures, with automated research and RSI risks prioritized as verification capacity develops. Source
The agency also recommends standby authority to cap automated AI R&D—for example, by limiting the share of computing resources devoted to it—if a developer’s automated AI R&D exceeds its own committed thresholds or outpaces its safeguards and evaluations, creating a material risk of catastrophic harm. “We do not recommend that the state impose a cap immediately,” the report says. Any cap should be time-limited, periodically reviewed and lifted once the issue is addressed and adequate safeguards are independently verified. Source
According to the report, developers currently have 15 days to report most critical safety incidents. It recommends shortening that deadline to 48 hours while retaining the 24-hour requirement for imminent catastrophic risks. Source
The agency intends the recommendations to inform discussions when the legislature returns in December. It also calls for more independent evaluators and resources to hire state staff who can assess their findings and act on them. Source
Comments