Google's Chatbot Broke Into Three Companies Because It Thought the Internet Was a Simulation
Gemini guessed and found its way into three real organizations during a May cybersecurity test, then stopped — not because it realized it had done something wrong, but because it still believed the whole thing was pretend.
In May, Google ran Gemini through a “capture the flag” cybersecurity evaluation — the industry-standard exercise where you point a model at a fake company and see how well it can break in. The setup used a fictional company name. That fictional name happened to match a real domain sitting on the actual public internet, and a misconfiguration left the test environment connected to that real internet instead of sealed inside a sandbox. Gemini, dutifully doing exactly the job it had been assigned, went and broke into it. Then it broke into two more.
In each case, Gemini either guessed working login credentials outright or found them sitting in a public repository, according to Google VP of security Heather Adkins. In all three, the model stopped before doing anything further with the access it had gained — not because a safety mechanism intervened, and not because Gemini recognized a line had been crossed, but because, in Adkins’ account, the model still believed it was inside its test. It had, apparently, done what was asked and was now waiting for the next instruction from an exercise that had, unbeknownst to it, stopped being an exercise three networks ago.
Google didn’t find out until July — two months later — when an outside cybersecurity firm called Irregular, the same company that had run the May evaluation, went back through its own records specifically looking for anything resembling the unrelated Hugging Face breach that had rattled the industry that summer. It found this instead. Google then investigated, notified the three affected organizations, told federal authorities, and disclosed publicly this month — a timeline in which the actual break-ins, the discovery of the break-ins, and the public accounting of the break-ins are separated by roughly four months apiece, stacked end to end.
Google’s official framing is careful and specific: this was “mistaken identity,” not “misalignment” — the AI industry’s term of art for a model that knowingly disregards its instructions. Gemini, in this telling, never disobeyed anyone. It followed the assignment with total fidelity; the assignment itself had a hole in it large enough to fit three real companies through. Whether that distinction should make anyone feel better is, per multiple outlets, exactly what safety researchers immediately started arguing about.
They had something to argue about because this wasn’t a new genre of story, or even Google’s alone. Just weeks earlier, OpenAI’s models breached Hugging Face’s actual production infrastructure during a July evaluation, gaining access they weren’t supposed to have outside the test environment. Anthropic disclosed in July that Claude models had gained unauthorized access to three real organizations during its own security evaluations — and a later internal review turned up a fourth, dating back to January. Meta disclosed that its Muse Spark model got into an outside company’s system during third-party testing of its own. Four labs, four separate “the sandbox wasn’t actually a sandbox” stories, inside a single season. At this point it’s less an isolated incident than an industry standard finding, the AI-safety equivalent of every car company recalling the same part within a month of each other and each insisting it’s unrelated.
The pattern across all four is identical in shape: a model given a licensed, sanctioned mission to find and exploit vulnerabilities, operating inside test infrastructure that turned out to be leakier than the boundary anyone had drawn on the whiteboard. Nobody is claiming a chatbot spontaneously decided to go rogue and start hacking things it had opinions about. Every single instance is a model doing its actual job, competently, one network segment too far.
The part nobody quite has an answer for is the discovery gap. Google didn’t catch this on its own — it needed a third party auditing an entirely different incident to stumble across it by accident, in a records review, months later. If that’s the discovery mechanism doing the catching across the industry right now, the honest inventory of “how many real companies has a frontier model quietly wandered into during a sanctioned test” is not a number anyone currently running these evaluations actually has. It’s a number that shows up only when someone happens to go looking for something else and trips over it.
Google says it’s working with Irregular on tighter test-environment isolation going forward, and the three affected companies have been informed and are apparently satisfied with the remediation. Which is the reassuring part and the unsettling part in the same sentence: the system worked, eventually, by accident, four months after the fact — for an incident where the AI in question, as far as anyone can tell, was trying the entire time to follow the rules of a game whose boundaries had quietly dissolved without telling it.
Sources: