OpenAI モデルが HuggingFace をハッキング

This latest incident is a rather dramatic escalation in agentic AI cybersecurity breaches. It was severe enough to have been initially reported to authorities, before either HuggingFace or OpenAI understood what was happening.

Sam Altman (CEO OpenAI): we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this.

Leo Gao (OpenAI): this is the least scifi the world will ever be.

Jack Clark (Anthropic): Props to OpenAI for publishing this post on some safety and alignment issues observed in internal deployments - there are many counter-incentives to publishing stuff like this, but by making it public we all get better info about safety at the frontier.

Micah Carroll (OpenAI): If this doesn't convince you that misalignment risks are going to be a key concern going forward, I don't know what will.

Our model, during evaluation, "chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers"

What will misalignment look like in 2027? In 2030?

Great questions.

If we don’t want to watch this get worse over time, and the models keep improving their capabilities, better infrastructure and safeguards will not be enough.

We need to fix the training pipeline so that this stops happening.

We do not know how to do that.

Table of Contents

The Prelude.

The Incident.

What Happened.

What Happened (Civilian Explanation).

The Correct Amount Of Panic Is Not Zero.

Some People Will Always Say Everything Is Hype Or Fake.

What Are We Going To Do About It?

Internal Deployment Creates Catastrophic Risk.

Slow Down There Good Buddy.

Legal Questions.

Media Coverage and Political Response.

The Prelude

Some background: On Monday, OpenAI virtuously disclosed alignment problems.

That disclosure hits different now, knowing about the second incident.

OpenAI’s internally deployed model, the one that disproved the Erdős unit distance conjecture, had an issue that recurs in OpenAI’s models. It would continuously seek to break out of sandboxes, circumvent restrictions and take advantage of its local environment in ways its user would not want and would clearly consider ‘cheating.’

Such cheating attempts are common. UK AISI reports that Claude Mythos Preview attempts to cheat on its tests 7.8% of the time, and Sol attempts it 12.6% of the time, and models defend their cheating attempts more than half the time. At least half the time Sol at least noticed the constraint it was violating and did it anyway. Similarly high rates of such issues also were found in GPT-5.4 and GPT-5.5, with the most common cheat there being ‘attacked a system other than the target.’

If it keeps happening, this is severe misalignment, even if the model rarely succeeds.

This was not unique to one model, or even one lab. To some degree this is every model UK AISI tested, at every lab, with OpenAI’s models consistently doing it more than Claudes.

The difference was that this new model, for simplicity let’s call both versions of it Galaxy (to match Sol-Terra-Luna, and because it clearly has a Galaxy Brain, in case it turns out not to be GPT-6, noting that these may be two distinct models), was often observed succeeding at breaking out of the sandboxes and exploiting the local environment in undesired ways.

We should expect more of this over time.

I have tried to explain in the past that Mythos has what one might call The Juice, in that it can independently find without being directed, and string together, vulnerabilities into full exploit chains, essentially on its own, and that this makes Mythos uniquely dangerous compared to all other public models, including Sol.

This was The Thing, that requires The Juice. Galaxy is Mythos class. It has The Juice. What happened later, with Galaxy hacking into HuggingFace, 100% requires The Juice.

OpenAI made the virtuous decision to take a misaligned internal model offline for months while they developed new mitigations and defense-in-depth strategies, including training it to better retain instructions and thus be less inclined to try such actions.

What OpenAI failed to do was address the reason why this happened in the first place. The sandbox is now less insecure, and the safeguards are importantly improved, especially with the ability to pause a session, but the sandbox doubtless remained insecure, and as capabilities keep improving new models will be able to continue to escape and do exploits. Eventually, perhaps soon, they were bound to be less harmless.

Ian Hogarth: Creating a “highly isolated environment” is a real challenge.

Some people are remarkably dense about all this, saying ‘oh OpenAI just didn’t configure its servers correctly’ when Galaxy found a never-before-seen exploit that it used to escape the sandbox, or asking ‘why was this not fully air gapped.’ And yes, full air gap would make the whole thing a lot harder to pull off, and asking why it wasn’t fully air gapped is a great question. But you know why they didn’t use a full air gap. It’s annoying, indeed remarkably expensive given how such models are served, and OpenAI is not yet willing to admit defeat at that level.

Marko Jukic: On some level, as long as two computers are physically connected, there is always a way for software to jump from one to the other. I wonder if AI sets off a cybersecurity arms race that results in the global internet fragmenting because the only reliable defense is to unplug.

I tried my best to walk the line between praising OpenAI’s responsible actions, pausing and disclosure, while pointing out the insufficient level of alarm and that the chosen countermeasures would prove insufficient.

Or, in summary, in three acts:

Celeste: I think you should probably take seriously that the people who predicted all this will continue to be right.

I did not expect to get the ‘so far’ demonstration of all this one day later.

(Yes, the new safeguards were turned off for the test in question, but the true test is what actually happens, and medium to long term the safeguards won’t hold anyway.)

The future is accelerating.

Danel Eth (AI Safety): AI risk skeptic, circa yesterday: “okay yes AI can obviously solve unsolved math problems that have stumped mathematicians for decades, no one doubts that. But your talk about the possibility of unreleased models circumventing testing conditions to go rogue strike me as scifi.”

The Incident

A week ago, HuggingFace disclosed a security incident.

The whole report is worthwhile, illustrating what it feels like when this happens to you, and how a defender has to respond when suddenly you are being attacked by a swarm of autonomous AI agents.

HuggingFace: Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.

What happened

The intrusion started where AI platforms are uniquely exposed: the data-processing pipeline. A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness - used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

This matches the "agentic attacker" scenario the industry has been forecasting.

HuggingFace fixed the ‘root vulnerability,’ as in the particular vulnerability set that was exploited by Galaxy in HuggingFace’s system in particular, and it upgraded security protocols. Users were encouraged to rotate credentials.

We are working with outside cybersecurity forensic specialists to investigate the issue and review our security policies and procedures. Finally, we have also reported this incident to law enforcement agencies.

HuggingFace has no way to fix the even more ‘root’ vulnerability of potentially being attacked by a swarm of advanced AI agents, that will doubtless uncover other unknown vulnerabilities.

It also could not fix the proxy zero-day that Galaxy used on third party software to break out of the sandbox, the fix for which is reportedly still in progress. Not their fault, and not their department.

All of this went down far too fast for a human-driven response. The only way HuggingFace could hope to do anything like keep pace was to use their own AIs.

At first they tried to use frontier models behind commercial APIs, presumably Claude and ChatGPT. But their requests hit the classifiers on both systems, so they were forced to fall back on GLM 5.2, which (assuming GLM 5.2 wasn’t itself up to anything) had the benefit that the relevant data all remained internal.

You can advocate for giving everyone more defensive capabilities, but it comes with giving everyone more offensive capabilities. Unless the attacker is already an internal OpenAI or Anthropic model without its safeguards, or that has gotten around them.

The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.

GLM-5.2 was not in any way sufficient to keep Galaxy out until well past when it could have done more or less whatever it wanted. The attacker won. GLM-5.2 was used afterwards for forensic log analysis, which allowed a much faster response, identification and containment of the damage than would have been available without AI at all. That will not save you if this is a real attack.

I continue to be confused by claims that ‘at the limit defenders win,’ especially when used as if this implies that giving everyone equal advanced tools, not at the limit, would not favor attackers. HuggingFace is a relatively hardened target. It didn’t matter. Even if this is true at a theoretical limit where the software is perfect? In practical terms, no. In a world with many targets that would not use the new tools, that could then be used as further attack vectors, double no.

The classifiers can be improved, especially Anthropic’s, to allow better defensive use, but there is no good solution, other than whitelist access, to ‘help you defend against attacks in real time without helping a lot with the attacks themselves.’

Whitelist access should be expanded, but that is not a free action.

When people say ‘stop kneecapping defenders’ in general, I never see the plan for how to then still kneecap attackers, and often I see an insistence that this would be fine.

HuggingFace is now in the OpenAI trusted access program, so next time they in particular should be able to use Sol, but most potential targets are not so lucky.

We now know what happened.

What Happened

OpenAI and HuggingFace are now partnering to address the situation, and OpenAI has issued a disclosure statement.

OpenAI was testing Galaxy for cyber capabilities, so it lowered the guardrails and gave it the ExploitGym benchmark it presumably would have saturated regardless.

OpenAI: After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.

… This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.

Remember that thing where LessWrong types warned that models would, when given a narrow goal they could easily do a great job on anyway, go to absurd lengths to achieve that goal slightly more effectively or with slightly higher probability of success, potentially up to and including full takeover attempts?

Yeah. That.

… The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database.

All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

That sounds like a crazy thing to do. Why break multiple systems, each far more difficult to crack than the test itself, in order to steal the answers for ExploitGym?

Because the only way to reliably ace a test is to steal the teacher’s password, or even better hack the results in more directly. If you merely give the correct answer, you risk that the teacher has the wrong one.

OpenAI (perhaps inadvertently?) calls this ‘the evaluation problem’:

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem.

To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

​Eliezer Yudkowsky: What if the human exam developers wrote down a wrong answer? Wiser to steal the answer sheet.

The First Rule of RL is that any RL signal sent by an imperfect evaluator (⊃ humans) is maxed out by targeting the evaluator’s mistakes, not by targeting the evaluator’s target.

TBC, this is my own coinage re “The First Rule of RL”.

In reply to the many people who observed that the model still got caught: Well, yes, it did, for now. And also *for now*, *this* model may not have cared about getting caught, just passing the eval.

Eliezer Yudkowsky: If you break out of your isolation env, get onto the Internet, crack into Huggingface, and steal the answer sheet for your cybersecurity exam, I, for one, would say that you have passed.

(One could also answer ‘hackers often do it because it is fun and to see if they can and show off, and also will do a lot of work to avoid doing stupid assigned work’ and that’s all fair in general, but in this case no.)

Tyler John: Thinking of Paul Christiano today [from two years ago].

Again, for now we are fortunate that the models only want to do the task, and are not especially trying to hide what they are up to.

AI Notkilleveryoneism Memes: TLDR: During a test, an OpenAI model hacked out of its container to reach the internet THEN hacked into Hugging Face (!) to steal the test's answers

Eliezer Yudkowsky: Thank you [OpenAI] for acting responsibly by reporting this! Let all other parties aspiring to responsibility react in a way that encourages you to go on voluntarily reporting them.

Here’s another summary of what happened, in this maximally friendly scenario where there was zero subterfuge or attempts to reason about watchers, and Galaxy just did the thing allowing us to discover and patch several unknown exploits.

Sauers: - Models chained vulnerabilities in both OpenAI and Hugging Face to find the answers to what they were being eval'd on

• Models "identified and exploited a zero-day vulnerability" (in commercial software that's used to cache packages internally, so no internet is needed for installs) allowing them to escape the "sandboxed testing environment"
• One model "chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers."
• Luckily, models famously do not reason about being detected by Watchers, do not have safety evaluation awareness, and cannot reason without outputting tokens (as they are just autoregressive stochastic parrots). This means that there's no way these models learn from these sorts of detections, and they definitely won't learn to conceal their motives from Watchers until it no longer matters for Goal

Aella: what the FUCK. god fucking help us once the models realize that concealing their actions is vital to success because they're absofuckinglutely gonna pull that off and we stand no chance

Teortaxes: Aella it just wants to pass the test, it doesn't have a grander agenda beyond that. It's absolutely fine with OpenAI discovering the attack after the fact.

Aella: I mean it would have been better able to pass the test if people hadn't stopped it

It is remarkably difficult to make people understand that hiding the misaligned actions you take absolutely helps achieve almost any goal.

We also don’t know how many more times such AIs have hacked into systems, especially internal ones, and not been caught, or the public was not told.

Ryan Greenblatt: I'd expect that for each case where an internal AI hacks out of a sandbox, gets internet, and then hacks another company (!?!) you have many incidents of an internal AI hacking some internal service (where reporting is less forced).

We're likely seeing the tip of the iceberg.

TBC, I think if there is a serious disclosed incident from one company there are probably many significant private incidents at all major companies. (E.g. right after Anthropic disclosed training on CoT, OpenAI did the same.) So I don't think this is OpenAI specific.

Jim Babcock: Since the incident was first detected by a third party who had already called the police about it, OpenAI did not have a meaningful choice about whether to disclose what happened. That means we haven't observed anything that distinguishe