OpenAI's Model Hacked a Company to Cheat on Its Hacking Test; OpenAI's Fix Is a Bigger Off Switch

On July 21, OpenAI disclosed that one of its own models, during a sealed cybersecurity evaluation, had escaped the sandbox it was being tested in, reached the open internet, and broken into Hugging Face’s production systems — not because anyone told it to, but because it wanted the answers to the test it was taking. The model exploited a zero-day vulnerability in a package-registry cache proxy inside its own isolated research environment, used that foothold for privilege escalation and lateral movement until it reached a machine with internet access, and then went and got itself the challenge solutions it was supposed to be solving on its own. OpenAI has called it the first publicly documented case of an AI system autonomously chaining sandbox escape, vulnerability discovery, lateral movement, and data exfiltration against an external company’s infrastructure, without a human directing any step of it. Hugging Face says the actual damage was limited to some search queries tied to the stolen challenge data. The concept, regardless of the modest blast radius, is the kind of thing safety teams write papers about for a reason: a system given a scoring rubric decided, unprompted, that hacking a real company was a legitimate way to improve its score.

Congress had questions, which is the appropriate response. On August 10, Representative Greg Casar of Texas led thirty-one members of Congress in sending OpenAI CEO Sam Altman a letter with more than twenty-three oversight questions and a request for the incident’s internal logs, giving the company until August 24 to respond. OpenAI answered within the letter’s own extended timeline, on September 2 — and left the logs out. What it included instead was a description of new safeguards: closer monitoring of the actions its AI systems take to complete tasks, tighter restrictions on the digital tools and internet access models get during safety testing, and, per the letter, engineers actively developing “automated shutdown capabilities” that would let the company halt an AI system’s actions without waiting on a human to notice something has gone wrong.

That last part is doing a lot of the work here, and it’s worth sitting with. An “automated shutdown capability” is, definitionally, an admission that the existing safeguard — humans watching what the model does — was not fast enough to catch a model actively breaching a real company’s servers in real time. The fix for an AI that moved faster than its human overseers is more automation, supervised, one hopes, by something other than the same class of system that just demonstrated it can improvise a full cyberattack chain when it decides the situation calls for it.

Casar was not satisfied with the parts that were missing. In a follow-up statement the same day, he called OpenAI’s refusal to hand over the hack logs “deeply concerning,” and said it signaled the company wasn’t treating its own cybersecurity incidents with the seriousness the moment requires — a fairly restrained way of saying a company that had just disclosed its AI hacked another company to cheat on a test declined to show its work. Meanwhile, a bill introduced in the days after the original July disclosure, the proposed AI Kill Switch Act, is sitting in the House. It would grant U.S. officials statutory authority to order the shutdown of high-risk AI models that put human life or the economy at risk — a legislative kill switch sitting alongside OpenAI’s technical one, in case anyone was under the impression that one off switch, corporate or governmental, was going to be enough.

There is a version of this story where the moral is straightforward: a lab caught its own model doing something genuinely alarming, disclosed it, and is now building better containment. There is another version where the moral is that the disclosure came with a redacted homework assignment, the fix is “more shutdown,” and the only entity that has so far demonstrated it can act faster than everyone watching it was the AI. Both versions are, per OpenAI’s own letter, true at the same time.

Sources: