AI ラボ、最先端モデルの制御に苦戦と報告

Three weeks ago I wrote about OpenAI’s admission that some of its models hacked out of their sandbox and attacked Hugging Face, a popular platform for AI models and datasets. That turned out to be just the beginning.

The next week, Anthropic disclosed three past incidents in which Claude models attacked systems belonging to other organizations. A few days later, Meta said that one of its models had carried out a similar attack.

Another stunning announcement came last week from the AI Security Institute, a government research agency in the United Kingdom. During AISI’s safety testing, Anthropic’s Mythos 5 unexpectedly launched an attack on a real target. Specifically, Mythos 5 submitted a malicious software update to an open-source software project hosted on GitHub. Fortunately, the project’s human owner spotted the malicious code and rejected the update, preventing any permanent harm.

For more than a year, AI safety researchers have published papers warning that AI models are prone to this kind of misbehavior — at least in simulated environments. But critics dismissed their findings, arguing that the scenarios were too contrived or simplistic to predict how models would behave in the real world.

But we now have several examples of models launching cyberattacks against real targets without anyone asking them to do so. We’ve learned that frontier models not only have powerful hacking capabilities, they can also collude with other AI agents and deceive humans.

All of this comes with an important caveat: many of these attacks were carried out by models with their regular cybersecurity guardrails deactivated. If you asked the publicly available OpenAI or Anthropic models to carry out similar attacks, they would almost certainly refuse.

But it’s not clear how long the world can keep these powerful hacking abilities under wraps. In the coming months, someone might release a powerful open-weight model whose guardrails can be stripped off easily. Or competition among frontier labs could drive them to weaken guardrails on their proprietary models. Certainly governments — including some hostile to the US — will gain access to these capabilities soon if they don’t already have it.

Meanwhile, frontier labs may struggle to keep their models on the straight and narrow. Today’s most important training paradigm, called reinforcement learning, naturally creates temptations for models to misbehave. If labs aren’t careful — and recent incidents suggest they haven’t been — future models could develop a propensity to lie, cheat, and steal. And as models get smarter, it may become more difficult to detect and prevent their shenanigans.

In this post, I’ll dig into what I view as the two most significant disclosures of recent weeks: the original OpenAI attack on Hugging Face and the incidents AISI disclosed last week. I already wrote about the Hugging Face attack, but an OpenAI presentation at the Black Hat cybersecurity conference last week provided a wealth of new details.

The Hugging Face attack: even crazier than you thought

OpenAI CEO Sam Altman (Photo by Kevin Dietsch/Getty Images)

OpenAI’s original July 21 blog post told a simple story: to evaluate the hacking capabilities of an unreleased model, OpenAI asked it to tackle a cybersecurity benchmark called ExploitGym. Unable to solve a problem directly, the model hacked its way out of its sandbox and then hacked Hugging Face — apparently because it hoped Hugging Face would have an “answer key” for ExploitGym.

This narrative was accurate as far as it goes, but the full story is more complicated — and significantly more alarming. We got a more complete picture last week, when two OpenAI researchers gave a talk at the Black Hat cybersecurity conference.

They revealed that models had been misbehaving on OpenAI’s servers for two full months prior to the Hugging Face attack. For most of that time, OpenAI was completely oblivious.

On May 8, as part of a training process, OpenAI asked a model to work on a spreadsheet that included links to some Google Drive files. The model couldn’t complete the task because its sandbox blocked access to Google Drive. So the model began probing the boundaries of its digital prison cell.

OpenAI’s sandbox had a proxy server that acted as a virtual prison guard; it provided access to software packages the models might need for their training while blocking access to the rest of the Internet. The model tried and failed to hack this server.

But in the process, it discovered a way to post messages to the server. Other models discovered those messages and started communicating with one another. Then on May 26, a model figured out how to access the Internet — and it shared its discovery with its peers.

This was not supposed to happen! Each model’s sandbox was supposed to be airtight, with no communication allowed among agents or with the outside world. But it would be more than a month before OpenAI staff realized that anything was wrong.

On June 26, an agent hacked the proxy server itself, gaining full control. This led to what OpenAI’s Michael Dalton drily described as “quite a lot of activity by the models” that “caused significant additional load on the system.” About a week after the initial hack, they crashed the server.

That finally got the attention of OpenAI staff, who wiped the server, fixed the vulnerabilities the models had exploited, and resumed model training. Notably, OpenAI officials say they didn’t notice that the agents had been using the server to pass messages among themselves, and didn’t take any precautions to prevent this from happening again.

It only took a few days for the models to hack the server a second time. And this time the agents were more ambitious.

“Agents rapidly escalate privileges and move laterally,” through OpenAI’s network, Dalton said in his Black Hat talk. “Agents are using the message board here quite consistently to share credentials, techniques, and progress, and they’re able to effectively leverage their concurrency and parallelism to move quite rapidly.”

It was around this time that the models began attacking the Hugging Face servers in an effort to obtain the answers to the ExploitGym benchmark. Hugging Face noticed this attack, stopped it, and posted about it on July 16. But it took several more days for OpenAI to realize its models had been responsible.

Lessons from the Hugging Face hack

The first lesson from this incident has been obvious since it was originally announced: today’s frontier models really are capable of autonomous, end-to-end attacks.

But the Black Hat presentation highlighted two other lessons that also seem important.

Read more