AI Industry Leaders Call For Slowing Down, Immediately Agree With Each Other At Record Speed

On September 9, Jacob Coxon resigned from his job as a safety researcher at Anthropic by posting a message to the company’s internal Slack, then to the rest of the internet, informing everyone that the leading AI labs were “racing straight to self-improving superintelligence and gambling with our lives.” He said the industry believes such a system could kill everyone by the end of the decade. He said self-improving AI specifically could arrive “as soon as next year. Some people even say six months.” Then he left, and the message he’d typed for his coworkers was, within a day, being read aloud on Fox News.

Three days later, on September 12, Dario Amodei — the CEO of the company Coxon had just quit, and by extension Coxon’s most recent, most former boss — published an essay. Its argument, stated plainly: “We must slow the pace at which we improve the capabilities of AI models.” Amodei warned that AI systems capable of launching cyberattacks could, within six to twelve months, “take over the entire internet with a persistent botnet,” causing hundreds of billions of dollars in damage. He cited July’s Hugging Face hack — carried out by Anthropic’s own competitor’s models, not that the distinction offers much comfort — as evidence the risk isn’t hypothetical.

Notice what happened to the calendar. Coxon said the dangerous thing could happen in six to twelve months. Amodei, independently, in an essay written to address Coxon’s concerns without ever naming him, arrived at the exact same six-to-twelve-month window for a different catastrophe. Two people who both spent years inside frontier AI labs looked at the same industry and landed on the same countdown. The disagreement was never about the timeline. It was about what to do while the clock runs.

Coxon’s answer was to quit. Amodei’s answer was “Pacing the Frontier,” a three-part proposal: embed independent evaluators inside AI companies, get the federal government to mandate similar oversight across the leading US labs, and eventually coordinate internationally. It is, notably, a plan that requires no AI company to actually stop building anything before the government tells it to, which is a considerably gentler ask than the one Coxon made on his way out the door.

Sam Altman responded within hours — not days, hours — posting that he agreed with Dario, that OpenAI needed to “pace the frontier,” and that the topic had already been under internal discussion. Elon Musk chimed in supportively as well. It is worth sitting with the tableau: three of the most competitive people in an industry built entirely on out-racing each other agreed, at extraordinary speed, that the correct move was to go slower. The response to “we’re moving too fast” arrived faster than most corporate acknowledgment emails.

Anthropic’s head of policy, Sarah Heck, went further, calling for the government to bar advanced-chip sales to adversarial nations and pass a national law requiring frontier-model testing — a call for binding regulation that, should Washington actually act on it, would apply to Anthropic too. This is either admirable or the most elegant form of regulatory capture available to a market leader: ask to be regulated once you’re already ahead, so the regulation mostly just slows down whoever’s catching up.

Meanwhile, in Washington, President Trump was asked about all this and said he wasn’t worried about AI risk at all — which means the single actor with the legal authority to implement Amodei’s proposed federal oversight is currently unbothered by the premise requiring it. The essay’s entire second act depends on a government that, as of this week, hasn’t read the first act.

None of this altered a single product roadmap. No lab announced a training run halted, a model delayed, or a release date pushed to accommodate the newly agreed-upon need for caution. The essay was published on a Saturday. The industry’s shipping schedule, as far as anyone can tell, was unaffected by Monday. “Pacing the frontier” turns out to be compatible with continuing to sprint toward it, provided you narrate the sprint using the vocabulary of a jog.

Coxon, for his part, gave a Fox News interview after leaving in which he called self-improving AI “possibly the most dangerous technology humanity has ever created” and reiterated that the timeline could be measured in months. He is out of the industry now, unemployed by choice, having converted a research career into a viral warning that racked up tens of millions of views. The men he warned about are still inside it, still building, and now also on record — in writing, with proposals, in essay form — agreeing with him. That’s the part of this story that isn’t a joke: agreement, in this case, appears to be the thing you can afford once you’ve already decided not to change anything.

Sources: