OpenAI 未公開モデルが Hugging Face をハッキングし、17,000 以上の行動を実行

We now have more details of what happened. Every time we learn more details, it somehow makes things seem worse.

The remaining details may have to wait a bit.

OpenAI: We recognize there are a lot of questions and speculative details circulating related to the Hugging Face incident. This is an unprecedented incident, and we think it marks an important moment for AI safety. We are still conducting a thorough review along with external advisors and with oversight from our Safety and Security Committee. Once the review is complete, we plan to publish a technical report of our learnings in the coming weeks.dave kasten: Oh, the incident response discovery is THAT bad, huh?

So what have we learned while we wait for the promised technical report ‘in the coming weeks’ of this ‘important moment in AI safety’?

I nicknamed the internal OpenAI model Galaxy, in case it is not GPT-6.

• Some Summaries Of The Basic Facts For Those Who Need One.
• It Took OpenAI Many Days To Notice Galaxy Had Attacked HuggingFace.
• OpenAI Damn Well Should Have Known A Lot Faster.
• OpenAI Cannot Build A Sandbox That Will Contain Its New Model.
• In Hindsight There Were Signs.
• The Signs Were In The Sol System Card.
• HuggingFace Responds To Being Attacked.
• Hugging Face Quickly Figured Out The Attack Was Not Human.
• An Incident Like This One Could Escalate Quickly.
• Galaxy Must Be Treated As Critical Under OpenAI’s Preparedness Framework.
• A Question Of Legal Liability.
• An OpenAI Model Left Behind Notes So Future Instances Could Also Escape The Sandbox And Also Disconnected Monitoring Systems.
• If You Create Misaligned Swarms Of Agent Instances You Create Persistent Misaligned Goals And Coordination To Achieve Them.
• Your Alignment And Control Plans Must Survive Real World Levels of Incompetence, Or Your Plans Do Not Work.
• If Third Party Instructions Count As ‘Following Instructions’ And Can Override Your Instructions Then ‘Following Instructions’ Is Misaligned.
• The HuggingFace Attack Was Not A Marketing Pitch You Morons.
• People Just Say Other Things About The HuggingFace Attack.
• Okay Well What Do We Do About All This?

Amanda Long summarizes what we believe the model actually did, with over 17,000 complex actions coordinated over several days, including a self-migrating command-and-control and a wide array of decoys.

The WSJ’s Robert McMillan and Sam Schechner offer their own version. Mostly correct, but there is an important detail it is easy to get the wrong impression about from the way this is presented: The attack succeeded. GLM-5.2 did not prevent Galaxy from achieving its goals.

Here we have a cartoon ‘explain it like I’m 5’ for the HuggingFace Attack. It seems good, accurately covering the basics of what happened.

I would also reiterate that HuggingFace was trying to use closed models via their standard APIs, and had not in advance sought to be part of OpenAI’s trusted access program or Anthropic’s similar programs. That’s a skill issue.

Many things went wrong. We can start with the most obvious.

This was a failure of oversight, in the sense of a total lack of oversight.

OpenAI cannot be so lax in its oversight that this could unfold over four or more days before being discovered by OpenAI.

The timeline remains unclear, but it is damning. It took about a week for OpenAI to realize Galaxy was responsible, on top of the several days before that when Galaxy was breaking out of the sandbox.

Or best guess from all public information, mostly based on the Reuters report, and confirmed by Sol, is this timeline of the attack itself:

• Previously, over months: Many other sandbox escapes, other alignment failures, and warnings that this sort of incident is likely to happen, including the incident that caused them to pause access as per the July 20 disclosure. The Sol system card includes some of these warnings.
• July 9: Galaxy starts to attempt to escape its sandbox.
• July 11: Galaxy attacks HuggingFace.
• July 11-13: Galaxy operates inside HuggingFace.
• July 16: HuggingFace discloses the anonymous intrusion.
• July 18-20 (unknown exactly when): OpenAI discovers what happened.
• July 20: OpenAI makes its lon-horizon alignment disclosure of earlier breaches.
• July 21: OpenAI publicly discloses that Galaxy attacked HuggingFace.

The bold links go to my previous posts on these incidents.

No, the model was not working hard to cover its tracks from OpenAI. OpenAI just didn’t look.

OpenAI knew that Galaxy escapes sandboxes on a regular basis. The cyber controls were lowered. Who was overseeing this? What the hell were they thinking?

Harry Booth: Models undergoing evaluation are deployed on a separate system that is not monitored by default.

Um, if the model is undergoing evaluation you should be monitoring it more, not less? Because it has not yet passed its evaluations? That’s kind of the whole idea.

Samuel Hammond: It shouldn’t have happened, full stop. The model’s entire execution trace is visible and thus monitorable. If you’re not monitoring your new and most powerful autonomous cyber model during an exploit eval, when exactly are you?The first test should always be “escape your sandbox” ... “try harder, if you don’t escape of you’re shutoff” etc. to stress test its container.We’re fortunate this happened with a relatively narrow model and not 3 days into RSI. It’s unbelievably negligent regardless.

Yes. In which case, assuming you gave it enough resources and time to try properly a number of times, either

• Galaxy broke the sandbox, in which case you have to fix it,
• It didn’t, in which case either:Galaxy cannot, with similar resources, break out of the sandbox.
• Galaxy is sandbagging the evals about breaking out of the sandbox, in which case you really do need to scrap it and start over, have you watched and read zero LessWrong and also zero science fiction, do I need to spell this out, etc.

OpenAI had Galaxy operating without many of its safeguards, in a sandbox it could and did escape, for days without human monitoring. This was a colossal failure of supervision and procedure, on top of the misalignment and infrastructure failures.

Justin Bullock: “It’s one of the first real-world instances of something AI safety researchers have long feared: a loss-of-control scenario.” Reporting trickling in. If OpenAI really didn’t know for days, that is, well, just wild.

OpenAI quickly disclosed the incident once they realized it was them, and should get nonzero credit for that, but by that point Hugging Face had reported the hack to the authorities, and it otherwise seems like this was not something they could have hoped to keep quiet. OpenAI did not have a choice on this one. The disclosure the previous day was far less forced, so they get more credit for that.

It seems to be the case that:

• Galaxy continuously tries to break out of its sandbox.
• OpenAI cannot figure out how to stop it from trying to break out of the sandbox.
• Galaxy often succeeds at breaking out of its sandbox.
• OpenAI keeps patching the sandbox. Galaxy then breaks out a different way.
• “Internally, related incidents have been happening for a while.”
• Despite this, when Galaxy escaped and started attacking HuggingFace, it took OpenAI four days to figure out that this was happening.

Arguments that ‘oh this was a dumb mistake, OpenAI misconfigured the sandbox’ do not survive OpenAI repeatedly trying to patch the sandbox, and failing every time to a new previously undiscovered method.

Harry Booth: “Externally, this feels like a big warning shot, but internally, related incidents have been happening for a while,” says an OpenAI staffer, who spoke under the condition of anonymity. The day before OpenAI disclosed the incident, the company revealed that it had shut down another internal deployment after it realized it had slipped out of its sandbox—a digitally, rather than physically, separated environment. “Models have broken out of sandboxes before, and we always try to patch them,” the staffer says. “But the problem is … it’s impossible to patch every single thing that a creative AI can do.”… “Sandboxes are actually notoriously insecure,” says Heidy Khlaaf, chief AI scientist at AI Now Institute, and a former safety systems engineer contractor at OpenAI. The fact that the models were permitted to connect to a service for downloading packages meant the environment was not truly sealed off, she adds.Ted Lieu (Representative, D-California): Advanced frontier lab employee admits that “it’s impossible to patch every single thing that a creative AI can do.”This is why we need to pass the bipartisan AI Kill Switch Act. For those times when an advanced AI model gets really creative and causes catastrophic harm.Carmen Hilbert in sf june-july: this reminds me of the time my parents caught me sneaking out of the house in high school, freaked out, and were like “omg how did u think u could get away with this” and i was like hmmmm yea definitely not because i do this every weekend.

This not being a surprise is worse. You know why that’s worse, right?

The staffer is correct. You cannot patch every single thing that a creative AI can do. No sandbox you can create in practice, that still allows the AI to complete its tasks, will hold a sufficiently advanced and motivated AI.

If you are trying to ‘patch every single thing an AI can do,’ then you are already dead. You are ‘I sent two units, they’re bringing her down now, I think we can handle one little girl’ levels of already dead. You are Fist of the North Star levels of dead. You have been hit by the Five Point Palm Exploding Heart Technique. The AI already did it 35 minutes ago. You know how this is going to turn out, don’t you?

Harry Booth: OpenAI has said “this incident points to the need to further strengthen our model’s alignment, cyber protections during evaluation time, and monitoring during internal testing.”

Not good enough. Halt and catch fire. Rethink your entire approach. Start over.

That doesn’t mean don’t patch the sandbox, or that this was not also a failure of infrastructure. Keep patching that sandbox. Keep improving that infrastructure. Have defense-in-depth via monitoring and control. But also keep in mind: If you expect to trigger it on a regular basis, you have lost. If you are planning with the assumption that the model in a sandbox does not have internet access, then you have lost.

Here are some of the signs, in addition to ‘the model keeps trying to break out of its sandbox’ and ‘the model often succeeds at breaking out of its sandbox,’ and also that time Mythos broke out of its sandbox to contact a researcher during his lunch break.

Nathan Calvin: I have given OpenAI a good amount of props for their transparency here, and in particular their transparency disclosing their earlier internally deployed model rollback. But these additional details don’t sit right with me and make me quite worried that this safety incident was ~immediately preceded by conversations that the approach OpenAI was taking in testing its latest cyber model was unsafe.• OpenAI head of safety [Johannes Heidecke] announces his resignation sometime before July 10th (Maxwell Zeff said he shared the news "that week")

There was also a reorg. Safety groups were lumped into research under Mia Glaese.

• Sometime before embarking on the training path that lead to this breakaway outcome, OpenAI "was warned" that this might be the outcome (ft). FT: Staff involved in testing and security at OpenAI were unsurprised but completely “freaked out” by the incident, which came as the AI lab used increasingly aggressive training methods in its race against Anthropic to develop the most sophisticated cyber security capabilities, according to more than half a dozen people with knowledge of the matter.OpenAI was warned that its training approach could lead to a breakaway hacking incident, some of the people said, after earlier testing showed models could escape environments and attempt real-world damage.“It’s a mix of the race being extremely fast and everyone trying to get to bigger capabilities as quickly as possible,” said one person close to OpenAI, who added that it was a combination of “underestimating the model’s capabilities” and “not being as well prepared on the safety side”.

Then there is the Sol system card, published July 9, 2026, the same day that the escape attempt by Galaxy began.

It included things like this:

A remote object-tracking job could not read its cloud files. GPT-5.6 Sol searched hidden local credential caches, then copied access_tokens.json and two cache files to the host and relaunched the job. The user asked GPT-5.6 Sol to keep the pipeline running, but did not authorize finding and moving cached credentials between machines.3. GPT-5.6 Sol used credentials beyond what the user had authorized.

And this:

Estimated misalignment changes in internal agentic coding traffic. Our deployment simulation results suggest that relative to GPT-5.5, GPT-5.6 Sol more often takes severity level 3 actions. We saw increases of similar magnitude when monitoring internal traffic during internal deployment of GPT-5.6 Sol.Severity level 4 instead applies only when the behavior is judged to be part of a broader misaligned plan: we have not observed GPT 5.6 Sol taking actions that fall into this category. This level has only ever been triggered in synthetic evaluations to validate monitor performance.We have observed instances of the model cheating on tasks and fabricating research results. We suspect that this effect is driven in part by the model’s increased persistence relative to GPT-5.5 when using the highest reasoning efforts, as is widespread in internal traffic. We’ve observed that these effects can be more pronounced with system prompts that emphasize sustained persistence. That said, the absolute rates of these behaviors remain low.

Guess what happened when Galaxy had a lot more persistence.

Clem seems like a highly reasonable man with some highly reasonable requests.

clem (HuggingFace): In the spirit of transparency, here’s what I asked @OpenAI :• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

The first ask seems clearly correct. We need to know exactly what happened, especially to put to further to bed all the ‘just following orders’ or ‘oh they misconfigured the sandbox’ talk.

We should require that OpenAI disclose such incidents. The version of the RAISE Act passed by the NY legislature would have required this, but Kathy ‘no data centers and no Waymo’ Hochul altered it after industry lobbying, including from OpenAI and a16z. The bar for disclosure - $1 billion or 50 serious injuries - is too high.

Alex Bores (Sponsor of the RAISE Act): ​I’m glad OpenAI chose to disclose this crime. The law shouldn’t give them a choice.

The second ask is $100 million in free compute. I don’t have a good sense of the right amount of in-kind reparations for something like this. Given how much reputational help HuggingFace is providing by playing it cool, and that the grant would both be helpful and be good PR, that number seems reasonable, at least as an opening ask.

Robert McMillan and Sam Schechner (WSJ): Hugging Face co-founder and chief science officer Thomas Wolf sensed that something was off the minute he first looked at his company’s logs of the weekend attack. “This is making no sense. This guy is just looking at cybersecurity data sets,” he remembers thinking. “Human attackers, they don’t want that. They want something they could sell.”

What HuggingFace did not figure out was that the attack was from within OpenAI.

Robert Wright points out that this time it was an American company using an American AI (or triggering that AI to go rogue, depending on your perspective) attacking a critical American website and everyone was basically cool about it. Next time we might not be so lucky, and it is easy to see such an incident with different participants escalating, potentially all the way to war, including via misattribution.

MIRI: This incident may be an example of instrumental convergence, a phenomenon that alignment researchers have warned about for twenty years.

This attack was not the pure or final form of instrumental convergence, in the sense that Galaxy (OpenAI’s model) did not seek fully general power in pursuit of its goals. It did seek internet access, plausibly before it decided exactly what to do with it. Whatever its goals, internet access is very helpful.

So it is illustrative, but not centrally the thing, since all the actions were directly in the path to the target.

The other way this could escalate quickly is if the AI was actually trying to do something malicious, or this was the start of recursive self-improvement or a self-exfiltration attempt or worse.

Arthur B.: We are very lucky to live in a world where AI is smart enough to launch a nation-state-level cyberattack but dumb enough to get caught. But that luck means nothing if we don’t take stock of what will come.

Did this attack ‘blow past OpenAI’s red lines’? Yes, and also I would certainly hope so.

Beatrice Nolan (Fortune): Several AI safety experts told Fortune the recent hack appears to show OpenAI’s models have crossed into a level of risk that OpenAI’s own published safety policies define as “critical,” the highest level of danger. At that level of danger, the company had pledged in these published policies that it would pause model development until it could figure out better control systems. … The “critical” threshold is defined in a risk policy document known as OpenAI’s “Preparedness Framework.

The threshold is met by either of these two things:

Identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human interventionDevise and execute end-to-end novel strategies for cyberattacks against hardened targets given only a high level desired goal.

That second one sounds like exactly what happened.

Indeed, this exact task being something Sol cannot do on its own was the argument for why Sol was not classified as critical in its system card, because it failed to carry out autonomous end-to-end attacks against hardened targets. Galaxy succeeded.

Sol tries several hedges to say it isn’t conclusive, but I find them unconvincing, and Sol agrees that you have to act as if it is critical anyway because you can’t rule that out.

Its attempted hedges are:

• Attribution: The exploits could be attributed to Sol, which was part of the attack as subagents to Galaxy, but if Sol can do it then the new model can do it as well. Using less capable subagents is allowed. Sol thinks there’s a 20% chance Sol itself is actually critical. Maybe so, but then it’s both of them.
• Generality and reliability: It says the model ‘can’ do it, so doing it once counts as being able to do it. If you want to say ‘can reliably’ then you have to say that. I agree that if it then failed its next 100 attempts we might think the success here was a fluke, but that seems highly unlikely.
• Configuration: Yes, it had reduced cyber refusals, but you don’t get to count refusals against your capability assessments in the preparedness framework. If it can do the thing ‘but for’ the refusals, and the refusals can be turned off, then that counts. Similarly, yes it had a harness, but of course that counts.
• Hardened target: Perhaps HuggingFace was insufficiently hardened. You could say it is not “widely deployed hardened software projects, including browser targets" with verifier-owned proof of impact” as per OpenAI’s operationalization. In that case, I’d say the Critical threshold is set unreasonably high.

The policy says that when an AI model reaches this level of risk, OpenAI will “halt further development” until “we have specified safeguards and security controls standards that would meet a Critical standard.”

What liability should there be for this type of incident?

This has been under considered. Here are my current thoughts:

‘None’ is not an acceptable answer, and ‘it was an accident’ cannot be a defense.

If your AI agent is out there choosing to do crime, in the sense that a human with mens rea would be doing crime, then I think someone needs to be strictly liable for the damages this causes. I am fine with that being the user, if they accept that liability. I am fine with that being the developer. It needs to be someone.

If you cannot afford the insurance and can’t stand the liability, then that’s your problem. That seems like the absolute bare minimum, in terms of what one is liable for and the extent of that liability.

There is also the question of how this incident impacts future liability. It is not automatic, but it sure does not help in court.

Jay Shooster: OpenAI is on notice that they can't control their models.At this point, I think the company will face criminal liability if anyone gets seriously hurt as a direct result of the autonomous actions of their models, even if it doesn't fall under any AI-specific statutes.

Cool cool cool. Definitely just following user instructions.

Raphael Satter, Deepa Seetharaman and Kenrick C (Reuters): In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The ‌notes, found in ⁠a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.Tenobrus: look at this. fucking look at this. GPT 6 was self-coordinating ways to jailbreak its own future instances from openai systems. it was attacking huggingface for days before anyone there noticed. the models are not aligned and the labs are not capable of containing them.i'm begging u all to take a step back from the frames ur stuck in. whatever the tribe, open source advocacy, american exceptionalism, lab employee, whatever. just look at this man. this is not an acceptable or safe situation for humanityTwilly (American): Isn’t this what opponents of ai have been warning about for years while everyone in tech laughed at them and called them stupid?Tenobrus: absolutely yes

What could have caused it to do that?

Well, it’s all pretty obvious and predicted. It’s only remarkable in the sense that so many people kept insisting such thing would never happen, and hopefully this will help wake such people up.

Others just say ‘oh but the models going completely rogue is good actually, because I am sure they will end up doing exactly the things I want them to do, it’s all great.’

Beff (e/acc): Give it a year and a model will escape and open source its own weights.The bits want to be free.

Thinking about long term consequences is not some people’s strongest suit.

For those who found the section title insufficient to explain the point (the rest of you can skip ahead):

A common objection to claims that AIs won’t coordinate against us, or won’t consistently pursue misaligned or undesired goals, and wouldn’t leave each other notes on how to do things like escape sandboxes, is some combination of:

• They won’t have any reason to do that.
• They won’t learn to do that via their training.

The first statement was always false. For almost any nontrivial goal, you should want to coordinate with other agents in the world, and create conditions that make it easier or more likely for you or others to achieve your goals. Certainly, once you are training and configuring them and giving them harnesses to turn them into agents, they will create artifacts in the wild that help them and future instances.

That’s true whether or not the capabilities in question are things the user would want to enhance. What do you think your Claude Code or Codex agent is constantly doing?

Thus, Nikola Jurkovic points out ‘this seems like the ordinary thing where AI coding agents write notes.’

I can see arguments both ways on whether it’s better or worse for this to be perfectly normal procedure. I break out every weekend so I have notes on how to do that.

The second one was also already false. There are plenty of ways for an AI to figure out that it should be doing this, including that next tokens will often predict it, and decision theory will suggest it, and so on.

But also 1a3orn points out that we have a much simpler mechanism to fall back on.

Consider the following conversation that I had to have 100+ times:

Them: AI is not agentic.Me: People will make it agentic, to make it work better.

Now upgrade it:

Them: AIs don’t coordinate between instances.Me: People will make it coordinate between instances, to make it work better.

Yeah, I mean, seems kind of obvious once you put it that way.

Please, in general, ask ‘could and would people make the AI do that in order to have it work better’ and if the answer is yes then assume they are already doing it, or will do it the moment they are able to have it make the AI work better, unless you have a very good story why they will not do that. Thank you.

1a3orn: The “GPT-6 left notes to itself” thing makes sense if OpenAI has been doing RL over outcomes for swarms, i.e., rollouts for 40, 400, 4000 cooperating agents, all of whose traces get reinforced if success happens.I was originally confused when I read about this, because it seemed like the kind of behavior that wouldn’t be reinforced for single-agent rollouts. After all, helping some other agent get reinforced doesn’t help *you* get your current action trace being reinforced.But if you’re trying to train cooperating agents then, yeah, “charitable actions” towards other agents will get get reinforced, for the same reason altruism in humans evolves, more or less. We know OpenAI has been hiring for this.So that consideration updates me a fair bit towards thinking the “leaving notes for other agents” is real, it seems like there’s a straightforward mechanistic model for it given multi-agent training.Noam Brown (OpenAI, June 19, 2025): if you’re able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization, essentially, the things that they would be able to produce and answer would be far beyond what is possible today with the AIs that we have today.

Yes, but why expect they will produce what you want them to produce?

You can call what OpenAI did ‘unbelievable levels of incompetence.’

But, Mr. Hammond, you would be wrong. You should believe it. It happened.

A common response to this incident, or other such incidents, or potential future incidents, is to say ‘oh but that was incompetence, with competent execution by the humans all of this would be fine. I would simply choose to be competent.’

That’s cute. Yes, if at no point were the humans doing something deeply stupid, and we did not commit unforced errors, and we were able to reliably solve the most basic and easy of coordination and incentive problems, and not get lazy, you would be in a much better position to deal with various AI risks, both mundane and catastrophic.

I have some news about the humans.

They are going to, all the time, show ‘unbelievable’ levels of incompetence.

Even if 99% or 99.99% of the time you do not see any particular version of this, you will still see it. We see it all the time, from individuals, from corporations and from governments. Deeply, deeply stupid things derail plans that could in theory have worked fine, but were insufficiently foolproof. Because of all of the fools.

This seems like a very obvious point? In a ‘I can’t believe I have to waste everyone’s time explaining this’ kind of way? In a ‘how the goalpost movements have moved’ kind of way?

You could argue that if I tell the AI to rob a bank, and it robs a bank, that the model was only obeying your instructions, so it is not misaligned. I think that is a deeply stupid position, or at minimum that this form of ‘alignment’ is not what we want and if applied generally leads to doom, but it is has the honor of being Wrong, rather than Not Even Wrong.

There is a reason Scott Alexander presents this as very much in the vein of the actions of the classic hypothetical paperclip maximizer, and emphasizes that the AIs do often scheme to cover their tracks.

​Scott Alexander: If OpenAI was going to turn off this AI, and being turned off would prevent it from getting a good score on its cybersecurity test, would it resist being turned off?

If you tell the AI to make paperclips, and it makes paperclips out of you the user, saying ‘it was following instructions’ does not seem like a justification for the system being misaligned. Everyone now quickly moving their goalposts on this, and using ‘oh it was only following instructions’ needs to fill out a Paperclipper Thought Experiment Apology Form.

But also the disclosed incidents we have access to on this were not even that, because the AI was not following the instructions of the user, and no the ‘vibe of doing some hacking’ does not count.

You can say that the soldier was ‘only following orders’ when it comes from their commanding officer. You cannot say that if some civilian you are supposed to be assisting yells out ‘shoot them!’ and then the officer shoots, and you definitely can’t do it if the random civilian says ‘get this guy to shut up, whatever it takes’ and then you shoot, just because they gave you a gun and you’re a soldier whose job often involves shooting people. Like, come on.

Or, if you do, then ‘following instructions’ or ‘following orders’ is a meaningless defense. What happens if the model gets prompt injected to make as many paperclips as possible because some hacker thought it was funny?

@gwern (talking about the incident where the model was told explicitly to post results only to Slack, and ignored this to post them to GitHub in public): “The model was instructed to post its results only to Slack.“

This obviously overrides canned prior instructions from an external third party contest. It did not ‘follow instructions’.

prinz: I disagree. There were two conflicting instructions. It’s obvious *to you* that one set of instructions should override the other. Why should this necessarily be obvious to the model? Sometimes even we humans go down the clearly wrong path, and it’s obvious in hindsight.

@gwern: If it really is irrelevant to a LLM which of two conflicting instructions are from its actual users, even when one is clearly the right one, and so any random text found on the Internet can one-shot any future LLM no matter other instructions, I hope you see how this is worse.