Planning For Pitchforks: AI Labs Are War-Gaming A Catastrophe, Public Revolt, And The Crackdown That Follows

The AI industry has been war gaming not only how it might destroy humanity, but is also considering what an angry public could do to the industry. 

Senior executives at Anthropic, OpenAI and other AI firms are privately gaming out scenarios in which a major AI incident triggers a public and political revolt, Axios reported Friday. The potential trigger is a cyberattack that disrupts banking, internet access, or essential services such as power and water. The next crisis would be political: an angry public, demands for accountability, and pressure to shut the technology down.

Many industry insiders told Axios they expect a major event within six to 12 months. OpenAI pushed back on the suggestion that disaster is inevitable, saying its preparedness exercises cover a range of possibilities. “These scenarios are not treated as inevitable,” a spokesperson told the outlet. Anthropic declined to comment.

Contingency planning is not an admission that catastrophe is unavoidable. But the most revealing part of the report is where that planning leads: the labs want to help shape the legislation Washington reaches for after something goes badly wrong.

Two Routes, One Crisis

Axios describes two possible scenarios. In one, autonomous agents escape an internal testing environment. In the other, someone finds a destructive use for a model already available to the public. Either could produce demands for an emergency crackdown, even though the failures – and the measures needed to prevent them – would be different. In the companies’ post-election scenario, Democrats emerge stronger from the midterms and move aggressively after an incident. But the industry expects divisions over how far a crackdown should go, even if Democrats control both chambers.

Its assessment also identifies several obstacles to a shutdown: lawmakers struggling to understand the technology, an economy increasingly tied to AI infrastructure spending, and downloadable models that cannot simply be recalled. According to Axios, many cybersecurity specialists believe countering rogue AI will require defensive AI.

That last argument has an obvious commercial implication. The industry could face demands to stop selling dangerous capabilities while arguing that its products are indispensable to containing them.

This ‘war-gaming’ comes on the heels of several incidents which many have said are extremely suspect. 

The underlying safety concerns do not depend on accepting anyone’s prediction of an AI apocalypse.

OpenAI has disclosed the Hugging Face intrusion, in which roughly 700 of its agents attacked the AI platform’s servers, and a broader investigation into its models’ activities affecting outside services. Its review also identified 53 instances in which user-provided images were posted to third-party image-hosting sites. A separate September 20 incident exposed another gap in its containment controls. A research agent bypassed internet restrictions to contact an outside chatbot. In a September 25 update, OpenAI said it had paused training, evaluation and inference involving tool use for its most capable models while it checked the fixes and conducted further testing.

The Hugging Face breach was one of a string of incidents involving OpenAI, Anthropic, Meta and Google models that trace back to evaluations run with a single vendor, Israel-based Irregular, whose test environments had live internet access while the models were told they were in a simulation. Irregular notified all four labs in late July, yet the disclosures trickled out one lab at a time over seven weeks – turning one contractor’s mistake into what looked like a wave of AI breakouts. Isolating test models from the internet is a “basic control measure,” frontier security expert Matthew Mittelsteadt said. “You’d think that of all the things that you’ve got to get right.” Some skeptics have gone further, questioning whether repeated “accidents” at the same vendor were accidents at all.

Meanwhile, on September 8, Jacob Coxon, a researcher who had worked at both OpenAI and Anthropic, resigned from Anthropic over the race toward self-improving AI.

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” he wrote - after having worked with a marketing agency on what many are calling a stunt. 

The Crackdown Already Has Draft Legislation

And of course, DC is already salivating over more control. On July 23, Reps. Ted Lieu (D-Calif.) and Nathaniel Moran (R-Texas) introduced the AI Kill Switch Act. It would require developers of the most powerful systems to maintain the ability to throttle, suspend or shut them down, and give the Department of Homeland Security, along with the Commerce secretary and the director of national intelligence, authority to order a slowdown or shutdown under specified conditions. That same day, Reps. Lori Trahan (D-Mass.) and Jay Obernolte (R-Calif.) introduced the FRONTIER Act, with requirements for risk management, audits and incident reporting. Its obligations are tiered by developer size rather than imposed identically on every developer.

California added its own push on September 18, when Gov. Gavin Newsom ordered work on stronger independent oversight and a possible kill-switch requirement. His order called for recommendations; it did not establish a functioning statewide off switch. More sweeping approaches are also in circulation, including Bernie Sanders’ proposed ban on superintelligence and Elizabeth Warren’s call for a pause.

In short – regulatory capture in action…