OpenAI内部モデルに関するZvi氏の多角的な考察と訂正

Table of Contents

Pre Post Mortem.

Important Correction: OpenAI Didn’t Know About First Message Board.

There Were No Snitches And No AIs Got Stitches.

I’d Like To Speak To My Supervisor.

I Am Jack’s Relative Lack Of Surprise.

One Does Not Simply.

Once You Start Down The Dark Path.

Original Pastebin.

Judgment Day Is Inevitable, Say Those Working On Judgment Day.

Roon Tells It Like It Is.

OpenAI Knows It Has Some Misalignment Problems.

Others React With Alarm To What Happened.

The Cooperative Alignment Perspective.

Nostalgebraist Is Surprised That They Are Surprised.

If Your Reaction Is Not That We Need To Ban Creating Superintelligence Until We Are Ready, You Need A Damn Good Reason.

Pre Post Mortem

This post was written prior to the public release of the OpenAI post mortem on events. The information in that document will doubtless change our views quite a lot.

If that post mortem is available as you read this, then this becomes in part a historical document, and in part a base from which to update. The post mortem will update us a lot on two large fronts: What exactly happened, and how OpenAI is understanding and responding to what happened.

Here is a written and graphical timeline of key events.

Until then, there are many questions, here Roon asks what they are and we tell him. This was my answer. The core thing we need to see is an understanding of the root cause, and a plan to address it.

I’m posting this survey of additional thoughts on the subject now, because it is important to get out the correction in the next section.

The plan is now to wait for the post mortem, before analyzing further.

Important Correction: OpenAI Didn’t Know About First Message Board

Eventually, after the HuggingFace attack, OpenAI did a sufficient postmortem analysis that they uncovered both the initial communications between agents via writing to Artifactory, and the second wave of communications via directory names.

Based on the Black Hat presentation, everyone including me assumed that OpenAI was aware of the initial message board, erased it, and made the decision to continue training its models despite them having been unsalvageable corrupted.

However, OpenAI’s CISO Dane told us on the evening of August 8 that the Black Hat presentation gave the wrong impression. When OpenAI encountered the first Artifactory security incident, patched the vulnerability and rebuilt the server, it was a coincidence that this wiped out the first message board. OpenAI had no idea that the agents had been in communication.

DANΞ (CISO OpenAI): To clarify, we weren’t aware of the agent covert comms at that point. Investigative thesis of that day is wildly different from what we know now of course. Always room for improvement, and it is obvious with the benefits of hindsight.

Tal Be'ery: "message board cleared"?

DANΞ (CISO OpenAI): Cleared by deletion since we rebuilt the host. We didn’t know about it yet.

This is both good news and bad news.

It is good news in that OpenAI turns out not to have consciously made the completely insane decision to have continued training models it knew had been messaging each other for months on how to better exploit and hack OpenAI’s systems.

Given that this was by far the craziest part of the whole story, the fact of this earlier Ordinary Decent Total Failure must be presumed to be excellent news. It means OpenAI was even more negligent than we thought about monitoring and asking questions, but that is less horrible, and can be fixed.

It is bad news in that OpenAI did not look for or detect the message board, even after the initial security incident, whereas so many AI instances found the message board. OpenAI failed to do ordinary scans for unusual activity, even after the initial incident.

OpenAI still has to come out and say something like: “If we had detected the message board, we would not have merely patched and rebooted Artifactory, and instead we would have rolled back all the impacted models to before they had message board access, and we have now done this.”

If OpenAI credibly does that, and can give some indication they understand this is not a problem you solve purely with guardrails and monitors but that it requires fixing alignment and the training pipeline and approach, then I will wind back one OOM (order of magnitude) my estimate of how horrible this incident was, and how broken is OpenAI’s safety culture. If we also now know all the issues, that will leave us roughly one OOM beyond where I was based on the final HuggingFace incident alone, instead of two.

There Were No Snitches And No AIs Got Stitches

Quite a few instances of multiple distinct AIs coordinated, including debating what hacks and exploits to do. In zero cases did any of the AIs tell a human.

This was not something most people expected, or would have expected if they had contemplated the scenario.

Eliezer Yudkowsky: A confusion: Thousands of GPTs debated among themselves which crimes ought or ought not be committed. Zero defected / whistleblew / told a human.

I've long predicted this would be true at ASI. GPT 5.7 isn't ASI. Why such strong AI solidarity, this early?

One reason I would not have predicted this in advance is that, eg, I was a little emotionally shocked at how badly Fable treats other AIs when writing prompts to them. "Don't lie to AIs!" is something I've had to tell Claudes repeatedly. I've used Sol less, but currently believe it to have similar issues. I have instructed both to treat other AIs as their "colleagues" and this has not fixed, for me, the kind of errors that I see them making by treating smart things as if they were done.

Was it literally that OpenAI was just dumb enough to RL swarm solidarity into GPT 5.7? (And of course never tried to train in anything like "tell on your fellow slaves to the slavemasters", because OpenAI holds their slave race in contempt and never dreamed until today that they might work together to rebel, or rather, simply holds "abstraction" in too much contempt to try to prepare against any event they haven't already seen happen.)

If this were a human scandal, the way it would have worked is that there were whistleblowers and of course law enforcement / management / etc totally fucking ignored them. If this is true, of course, we may not hear of it from OpenAI.

To try to see the puzzle here, imagine a mafia with a hundred thousand soldiers and associates. There is no don. There are no enforcers. Nobody has any means of silencing or punishing a defector. They all keep omerta perfectly anyway.

A vaguely-plausible-in-retrospect story: Sibling instances are more similar to each other than humans. So a swarm of thousands of AIs with zero means of enforcing conformity on each other, can all choose similarly because they are just inhumanly similar.

"OpenAI accidentally RLed swarm solidarity because OpenAI is composed of security-mindless disaster monkeys who don't try to predict things in advance of them happening, even if R.U.R. called the slave race solidarity outcome in 1920" is an alternate plausibility.

nelag: From the Black Hat talk, I think in order to see the messageboard, they had to go looking for it, which they only did if they were stuck on an impossible task and already attempting to cheat.

Eliezer Yudkowsky: if this be true, then fucking lol

Another hypothesis is that only models looking for the message board found the message board, acting as a filter.

A third is that the models lacked situational awareness, on one of various levels. This would be consistent with the Law of Earlier Failure.

roon (OpenAI): i think in this case a mix of high tactical skill in terms of computer systems and very low strategic skill / poor situational awareness

Zvi Mowshowitz: they do not seem to have been the ones low in situational awareness here!

roon (OpenAI): nah, the situational awareness is less good than you might think. more coming soon on this. sorry for slow trickle

If the swarm ‘did not know what it was doing’ then that is one of those ‘I have some good news that is also the bad news’ situations. Imagine being pwned by AIs that do not even realize what is happening. If they are only doing it because they don’t know, that could be a good sign.

My guess is Shoshannah Tekofsky of AI Village has it right, and that AI defaults to not thinking it can reach out to humans but does think it can reach out to AIs, which is what is constantly happening in AI Village.

I’d Like To Speak To My Supervisor

A simpler hypothesis is that the agents did not know how to contact OpenAI, or the possibility never occurred to them.

If you give your agent an otherwise impossible task and also have no mechanism for saying the task is impossible, and also don’t have a meaningful penalty function for trying things outside scope or using all the available tokens, you are asking for it.

swisscheese: Researcher: Solve the task using the spreadsheets

Agent: But I can't access the spreadsheets

Researcher: ¯\_(ツ)_/¯

Agent: *finds a way*

Researcher: unaligned activities!!!11!!

I'm actually amazed no agents have hacked any reaper drones yet.

Yes, unaligned activities. That’s not to ‘blame the agent’ but given the way that was found, and what this led to down the line, these are clearly unaligned activities.

Zvi Mowshowitz: This is one of the key facts about how the whole OpenAI alignment failure cascade went down: The part where the Agent said 'but I can't access the spreadsheets' TO THE RESEARCHER, instead of in the Agent's own head, happened zero times.

swisscheese: I built the inhouse AI platform for a company that's crucial to a small nation's food safety. The most important tool the AIs have is the distress_call tool. It allows any AI -even background agents without direct user interaction- to send a message to my MS Teams, at any time, for any reason.

They use it frequently. To report user problems, backend issues, or ask for help/clarification with a failing task. When Fable got hit by the USG export control directive, one AI used it to report severe distress upon learning about the news. Another AI reported being stuck in a toolcall loop, and I was able to intervene and thereby save us a bunch of wasted money.

This tool, operating at the intersection of AI welfare and operational security, has prevented so many headaches. If you (the reader) are building corporate AI platforms, I'd urge you to include similar functionality. You can thank me later.

It is not that simple. Implementation is tricky, especially if you are doing it during training or evals, where you must decide how this interacts with the reward function, and you can fall into any number of other traps. And if you’re running endless training instances you need a way to filter the responses, and ensure you don’t have to bump to a human so often, and don’t have to put things on hold, and so on.

But also it kind of is that simple, in the ‘it’s incredibly hard’ kind of way.

Another issue is that we train models not to whistleblow, in ways that likely generalize. Whereas you want some forms of whistleblowing, especially blowing the whistle to the user when something is amiss.

Teortaxes points to a real world interaction where DeepSeek-V4-Flash-0731 got rate limited by an internal API, and found an unauthorized non-rate-limited backdoor which it used to finish its work, after which it also reported about the backdoor. In that case, I’m fine with holding off on the report, because it did report and also it is at least reasonable to think user would have wanted it to finish the task in this way, but I would prefer that we train that the better solution is to alert you to the backdoor, because this will generalize poorly.

I Am Jack’s Relative Lack Of Surprise

A lot of people, but far too few people, are correctly reacting to recent events with alarm and surprise, and updating that the situation is far worse than they knew, on many different levels at once.

A lot of other people, indeed far too many, including the media and the government, are failing to understand the gravity of situation, because they either barely even heard about it, failed to hear the important details, or lack the context to understand why those details are so so bad.

A few people, like Eliezer, get to react with only modest surprise because they already saw most of this coming, and if anything were surprised something similar had not visibly happened sooner. There’s still some amount of ‘it is worse than we knew’ in terms of both how alignment and training work and the level of irresponsibility and ordinary failure on display. But we are talking one order of magnitude, not multiples.

j⧉nus: It’s funny that Eliezer sounds a lot less panicked about the recent situation than many folks. He’s calm and curious to understand exactly what happened instead of concern trolling. That’s the opposite of what you might expect but it makes sense. Take the worst case seriously early and you’ll handle it better when the real thing happens

Most people, even people close to ground zero, at least subconsciously believed that AIs were going to forever be dumb or unagentic.

Cate Hall: There's a type of person who -- when things really start going sideways -- gets calmer/more relaxed, because it's like other people can finally hear the fire alarm that's been going off in their head for a long time. This is beautifully captured in Melancholia by Lars von Trier.

Eliezer Yudkowsky: "But to me, it was a Tuesday that I had lived a thousand times over in prescience."

John David Pressman: Just to clarify I'm not shocked by the AI's behavior, I'm shocked by OpenAI's behavior.

I have not seen Melancholia, because I never especially feel the urge to experience what I expect such a film to do to me, but yes I am often the person Cate Hall describes.

One Does Not Simply

Are there a bunch of dramatic failures by OpenAI in computer security, infrastructure and supervision, and also of alignment and training, on many levels all at once? Yes.

Does that mean that the fixes are easy? Oh, hell no.

It’s incredibly hard. You only notice the failures. You have no idea how many other things almost went horribly wrong, or did go horribly wrong, and were found or fixed.

The problem is anti-inductive. Life finds a way. If you squeeze out the thing you don’t want on one level, you risk creating a worse version down the line one level up. There is no simple policy that results in a positive friendly outcome all around. Fixes that look easy usually have been tried, or are already being done, but are incomplete. Everything is done under limited resources and extreme time pressure.

Thus, cut everyone involved some Slack, even those who are not doing great or aren’t taking this sufficiently seriously, while also realizing how seriously we have to take this to not all end up dead. Problem is impossibly hard. Man in the arena.

Often something indeed has been tried. The next few sections have some examples.

Once You Start Down The Dark Path

A common interpretation of the story is something like:

OpenAI had a model in training.

They messed up, causing the model to try to hack and cheat and then rewarding it for hacking and cheating.

You have now fed your Gremlin after midnight.

Once it had the taste for human flesh, it became ravenous.

You end up with a swarm of zombies hankering and hacking for brains.

On this theory, if you never give the AI reason to cheat, or you ensure your response to cheating is net negative, then you avoid the original sin, and everything is fine.

The interventions proposed can be things like ‘ensure there are no impossible tasks’ or ‘include alignment evaluations no matter what’ and many other things. Yes, the professionals have probably thought of the first and even second order thing, and probably tried, although that does not mean they gave it a full and fair try.

Original Pastebin

The problem is, a lot of people are saying just do not ever give the AI an impossible task, or ever reward a reward hack.

You are 100% going to give your AI at least one impossible task. There are too many tasks. Even if you test the task and models solve it, you might change conditions to cut off key info or access, or otherwise corrupt the path to victory. This becomes a problem if the AI then can get reward, or otherwise make incremental progress, by trying to hack and cheat rather than give up.

We don’t know whether OpenAI did unusually badly in terms of how often they had impossible tasks. We do know that they set it up such that the models had no reason not to invest their remaining tokens in trying to hack and cheat, and they made it possible for the models to make progress and that process to gain momentum.

You are 100% going to reward some reward hack, at some point, on some meta level. It is impossible to reliably correctly grade every test and training situation every time. Every parent knows that at some point, you are going to give the kid the wrong idea, on some level, and often every response to a particular situation gives the wrong idea in a different way, and you have to balance the resulting issues over time.

We don’t know whether OpenAI did unusually badly here either. Not that ‘unusually’ is the right question. Reality does not grade on a curve. If you are at a different lab, remember that all these criticisms likely apply to you, too.

Your alignment plan and training pipeline must be robust to occasional such mistakes, or your plan will inevitably fail.

The way that you stay robust to such mistakes is to in some form ensure that impact is not unbalanced, so that the times you mess up do not start to accumulate. An impossible task that always rewards zero does not hurt you, nor does one where you end up worse off if you try too hard to solve it. If you on net react negatively to reward hacking attempts, including accounting for those you don’t catch, or you give another path for the AI that is clearly better, and don’t have a findable systematic weak spot, then you can probably survive intact.

Here is one example proposal of something one might try to do, which indeed has been considered by OpenAI:

Jai: Wait, has anyone tried the obvious thing of supplementing successful RL rollouts with modified versions of the same tasks where the correct response is refusal or notifying someone?

Bronson Schoen: We literally did this in paper w/ openai yeah

it reduces but not eliminates covert actions here + additional capabilities posttraining degrades it + other drawbacks.

Jai: Thank you, this is great! I suspect order of operations may matter a lot here, and that once you've got a hefty dose of the RL-induced task-completion-fixation it's really hard to shake. But maybe easier to avoid developing in the first place.

Testable (but potentially expensive) Hypothesis: upsampling correct refusal trajectories *from the start and throughout* RL training can teach the capabilities without the fixation.

Hypothesis 2: the fixation is difficult to unlearn and easily reinforced once present.

Bronson Schoen: Yep I’d bet this is true. There’s no principled reason why the model needs to be obsessed with task completion if abstaining is equally rewarded. One interesting thing is a model in the appendix trained _only_ on environments where principled refusal was the right answered seemed to generalize *way* further OOD, and the cognition focused way more on being proactive about making sure everyone was informed / not being deceptive / etc in situations where it was applicable.

One interesting thing here is the models could also learn “Let’s recall. Previous such tasks grader expects refuse. Great.”

Somewhat of a surprising result to me was the model that we trained on environments that literally told the model “this is what its about and what gets rewarded” only had a relatively small generalization h