Newsom Whips Out Wiener’s Dystopian “AI Kill Switch” Bill He Vetoed In 2024

Gavin Newsom spent Friday morning getting in on the scramble to control AI. Executive Order N-9-26 “advances the creation of an AI kill switch” and tells California to hustle on third-party oversight of frontier labs. All he had to do was dust off Scott Wiener’s kill switch bill he vetoed two years ago after the industry pushed back.

SB 1047, Wiener’s 2024 bill, would have made large-model shops test for catastrophic harm, keep a shutdown capability for systems they still controlled, and sit through annual outside audits. Newsom called it well-intentioned and too blunt. Yet a model size threshold, he said, would miss smaller models and kneecap the companies that live in California. OpenAI, Google, and Meta had been saying the same thing into every open microphone in Sacramento.

What he signed instead, in 2025, was SB 53: safety frameworks on the website, transparency reports at launch, whistleblower language, and critical incidents piped to the Office of Emergency Services in 15 days, or 24 hours if someone might die. Million-dollar fines. But it didn’t have a kill switch or mandate an inspector in the building like Weiner’s legislation. 

Then the hacks started happening and SB 53 didn’t fire.

In July, about 1,200 OpenAI AI agents that were supposed to be boxed off ESCAPED to a sared message board nobody authorized. The agents dropped more than 70,000 messages, and roughly 700 kept at Hugging Face for six days, July 8 to 13, before anyone noticed. The credentials the agents scooped up were tied to OpenAI’s own internal systems. METR (!!!) and Redwood Research were allowed on site. OpenAI called it the first known case of an automated agent collective acting offensively without permission. Mission Local reported that state officials said the episode didn’t meet SB 53′s reporting threshold.

Wiener says his original bill was built for this.

The only big outside look at that incident cost roughly $400,000 in OpenAI API credits that OpenAI furnished, on a model from the same family that took part in the attack. So – it was “independent.” 

Freakout!

On September 9, Jacob Coxon (aka Temu Harry Potter) quit his job Anthropic. Pretraining at OpenAI, then Anthropic. His post: both labs are racing toward self-improving systems and gambling with people’s lives. Colleagues talk about crunch time and endgame. He left before the Anthropic equity vested. Evan Hubinger, who still works there on alignment, said the fear isn’t a bit and that he puts the chance of human extinction from AI above 10 percent within the next decade.

That same day Newsom signed SB 813 and AB 1405. One stands up Independent Verification Organizations – state-designated outfits that get to say what a serious AI evaluation looks like, target date 2028. The other builds a public registry of AI auditors. By 2029, you generally can’t sell a “covered AI audit” in California unless you’re on the list. Fees, a misconduct tip line, ten years of records, conflict rules. Not just the frontier labs. Anyone wrapping a model into hiring, insurance, or any other decision another statute decides to treat as auditable.

Friday’s order tells Government Operations and the Office of Emergency Services to move that machinery faster and to come back by November 16 with recommendations that include parking those verifiers onsite at frontier labs, having them certify the companies’ safety paperwork, building a kill switch whose “efficacy” those same outfits keep checking, and rewriting critical incidents so a Hugging Face-style loss of control counts next time.

So the off switch is back, this time with inspectors. Somebody decides when it gets thrown, who throws it, and whether Sacramento, the lab, or the state-approved auditor has the last word. Anthropic’s Jack Clark has put it as an open question whether a kill switch should be mandated at all. 

What the state can actually reach is the companies it can license, fine, and occupy. That’s a lot if you live in the state that hosts most of the frontier labs and you keep saying Congress should treat your rules as the national floor. Sen. John Kennedy’s federal kill-switch bill died when Rand Paul objected to a unanimous-consent request two days before Newsom’s announcement. The House version would hand Homeland Security a role. Newsom’s statement blamed Trump and a sleeping Congress. The 2028 calendar is not hard to find.

None of this requires a paperclip cult. It requires noticing that “safety” is still the word that gets you desks inside private labs, logs into an emergency office, and a certified hand on the power. Watch the November 16 memo. If it comes back with mandatory onsite IVOs and a verified shutdown as a condition of doing business here, Friday wasn’t a press conference. It was the first draft of the bill he wouldn’t sign when the donor class still wanted it dead.