California wants to oblige powerful AI to have an emergency stop button: what exactly is proposed to be controlled

California is preparing a new level of control over the most powerful artificial intelligence systems. Governor Gavin Newsom signed an executive order instructing state authorities to work out a mandatory "kill switch" for advanced AI models, independent inspections directly in developer laboratories, and new incident reporting rules for loss-of-control incidents.

It is important to note that the "emergency switch" has not yet become a mandatory requirement. Order N-9-26, signed on September 18, instructs the authorities and invited experts to prepare recommendations for possible legislative changes.

One of the directly listed ideas is to oblige developers to create an emergency shutdown mechanism for frontier models, i.e., the most powerful artificial intelligence systems. Its operability, moreover, must be regularly verified by an independent organization.

What does "kill switch" mean for AI

The term itself may give the impression of a physical button that authorities could use to instantly shut down any chatbot. The document is not about that.

Experts will have to determine what an emergency termination mechanism for a powerful model should look like, in what circumstances it can be used, and how to independently verify that it will indeed work in a critical situation.

Therefore, it is incorrect to say that California has already gained the ability to shut down ChatGPT, Claude, Gemini, or other AI services. The state is currently only preparing possible requirements for developers of the most powerful models.

Developers may be inspected right in their laboratories

The "kill switch" is only one part of the initiative. The authorities also propose to consider permanent independent oversight of companies creating frontier AI.

In particular, it is assumed that major developers may be required to work with a designated independent auditing organization. Its specialists would be able to conduct regular audits and assessments directly in companies' laboratories.

Separately, independent confirmation of safety documents is proposed. This concerns risk management programs, transparency reports, and other materials that developers are required to prepare under existing rules.

If this approach becomes law, it will not be enough for companies to declare on their own that their model has passed the necessary checks. Key safety claims will have to be confirmed by an external party.

California wants to expand the list of dangerous incidents

Another aspect concerns situations where AI behaves differently than its creators intended.

State authorities want to review the definition of a critical incident and examine cases where AI agents bypass protective restrictions, act autonomously, or demonstrate potentially dangerous capabilities.

This is important because existing reporting rules cover only certain categories of events. The new approach could expand the range of situations that developers would be required to report to regulators.

The exact criteria still need to be developed. Therefore, not every unexpected AI action will automatically become a legally significant "loss-of-control incident."

California has already begun building an external oversight system

The new order did not appear out of nowhere. California is gradually building its own regulatory system for the most powerful artificial intelligence models.

In 2025, the state passed SB 53 — the Transparency in Frontier Artificial Intelligence Act. The law established transparency requirements for developers of advanced models and reporting on certain critical safety incidents.

In September 2026, Newsom signed two more laws strengthening external oversight of AI.

SB 813 provides for a certification system for independent organizations that will be able to assess models and AI systems for risks.

AB 1405 provides for the creation of a state registry of artificial intelligence auditors and establishes requirements for their independence, transparency, and work.

The new executive order should accelerate the practical creation of this infrastructure and at the same time determine what additional obligations may arise for developers.

Why California's decision matters for the entire industry

Formally, this concerns the rules of one American state. But California's influence on the AI market extends far beyond its borders.

The state remains one of the world's largest AI development centers. It is home to headquarters and research units of many leading technology companies.

Therefore, mandatory independent audits, expanded reporting, or technical emergency shutdown mechanisms, if they are indeed enacted into law, may influence how major companies design and test their models overall.

Companies are unlikely to build completely different security systems for each jurisdiction, especially when it comes to the fundamental architecture of model management.

The "shutdown button" has not been introduced yet

The main limitation of the initiative at this point is its status. The executive order does not establish a ready mandatory "kill switch" for AI.

So far, no technical specifications have been defined for such a mechanism, nor the conditions for its use, nor an exact list of models or developers to which it would apply.

First, experts and government agencies must assess the technical feasibility of the proposals and prepare recommendations. Further legislative decisions may be required to introduce new obligations.

Therefore, the current step should be viewed as preparation for the next stage of regulation: California wants to move from developers' voluntary promises and internal testing to a system where the safety of the most powerful AI is additionally verified by external organizations, and for a serious threat there must be a pre-tested model shutdown mechanism.

Based on materials from: Governor of California Office, Executive Order N-9-26