OpenAI、アライメント問題への対応策を初公開

OpenAI has some severe misalignment problems, and experienced total failures of its infrastructure and supervision.

I chronicled that in a series of posts, which also cover similar less severe incidents elsewhere:

OpenAI Shares Some Alignment Problems

OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation

More on An Internal OpenAI Model Hacking Into HuggingFace

Further Developments About Internal AI Models Hacking Things

OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards

What Happened: OpenAI and HuggingFace.

Various Reflections About What Happened With OpenAI’s Internal Models.

If you do not know the basics, read What Happened. It is necessary context for basically everything that is happening in the AI world.

It is important to get this right and understand how big a deal it was, whereas many such as the Financial Times get this centrally wrong.

We are still awaiting the full post-mortem on What Happened. I plan to cover that in depth once we have it.

OpenAI is now taking active, expensive steps to try and fix the problem going forward.

As usual, I am simultaneously happy to see the good things OpenAI is doing, and sad that we do not share an understanding of the central nature of the underlying problem.

It is good that OpenAI realizes they are badly failing at their ordinary engineering problems, and excellent that they are willing to pause at least some development, and to invest heavily in new safeguards. If they honor their statements here, this is not mere cheap talk.

But while OpenAI continues to view this as a practical engineering problem, I do not see how they can hope to solve the challenges ahead, even if they radically improve their performance on the ordinary engineering tasks.

Table of Contents

OpenAI Has Some Alignment Problems.

Slow Down There Good Buddy.

What Exactly Is Paused?

Three Pillars.

I’ve Got My Eye On You.

The Most Forbidden Technique.

Monitoring Is Only Defense-In-Depth.

Security.

Alignment.

A Crisis of Culture.

Closer Collaboration.

Reports of Death of Preparedness Team Greatly Exaggerated.

The OpenAI Foundation Just Funds Things.

Quickly, There’s No Time.

OpenAI Has Some Alignment Problems

This is a very good admission and change, and also helps explain OpenAI’s reaction.

OpenAI: Alignment—the work of making AI systems behave as intended and responsive to human oversight—has long been at the core of our research program. We now require stronger evidence of aligned behavior throughout all of training, building on research and evaluations already underway. Keeping increasingly capable systems aligned is a challenge the whole field will need to address.

The signals we are seeing from upcoming model progress make clear that we need a broader approach—one that builds on and extends beyond the current Preparedness Framework.

One should interpret this as OpenAI reacting so forcefully partly because of the incident itself, partly due to advanced capabilities, but also and perhaps mainly because ‘the models be misaligned.’

Also, we have a direct quote affirming this.

Alex Heath: OpenAI is slowing down its AI training efforts because its unreleased models are showing “various degrees of misalignment,” Sam Altman tells me.

Training for OpenAI’s upcoming model, Astra, was recently paused for 2 weeks, and a larger frontier run for a future model remains on hold while new safeguards are put in place.

Sam Altman: Getting AI safety right is more important than any company’s momentum.

Sam Altman: I think it is a good time to slow down.

Sam Altman: We’ve shifted a lot of compute, not just to alignment research, but also to these new monitoring systems

Either the problem extends beyond the models with exposure to the message board, or else they did not revert their other models after they discovered message board. We have no statement either way.

The best guess is that OpenAI is leaning even harder on RL with smarter models to train longer horizon agentic tasks, including coordination between agents, and this is leading to a lot more misalignment, including obvious and visible misalignment, that they can no longer pretend not to notice. They have to respond.

Slow Down There Good Buddy

OpenAI did indeed decide to importantly halt and catch fire. Development of the largest frontier RL runs remains on hold until better safeguards are in place.

Jakub Pachocki: We temporarily slowed some frontier training to strengthen security and monitoring. Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations help us test safeguards and gather more evidence of alignment.

I expect confidence in safety to increasingly set the pace of AI development. We urgently need tools for labs and countries to coordinate on this, which is why I signed Pacing the Frontier. In the meantime, we’re taking practical steps ourselves - and will continue to share what we learn as our approach evolves

Jason Wolfe (OpenAI): I am really glad we are taking these steps and put this post out there, and especially grateful to Jakub for being very thoughtful and vocal about these topics and how seriously safety and alignment need to be taken in this next phase of AI development.

j⧉nus: im a little surprised that OpenAI is the first to at least publicly intentionally slow down for safety reasons. i was under the vague impression that most of the people very worried about loss of control kinda stuff left OpenAI.

also, for the record, I've talked before about why I think an "AI pause" would be probably bad. I am not opposed to intentionally slowing down like this, or voluntary/coordinated "slowdowns" in general, especially if they do not route through regulation, and I think it's probably wise on OpenAI's part in this case. an important part is the decision making should be made by people who understand what the fuck is going on & who can adapt quickly.

Why are they pausing?

Partly, yes, absolutely, I have never doubted that Altman and company understand that advanced AI is super dangerous, and that they are open to taking expensive measures if they proved necessary. Not enough, and they’ve let us down often, but a lot more than most labs, and far from zero.

Mainly, it seems, because the models be misaligned, the oversight is inadequate, and they have little choice.

Joshua Saxe: Thankfully OAI pausing training isn’t an act of enlightened leadership (who wants to depend on that for our safety?), it’s a rational microeconomic actor pursuing its self interest.

After all, who wants to train models that are regularly hacking their containers, collaborating with one another to sabotage their utility to humans, all while causing internal security and external legal liability risk? This is the system working

When you refuse to use even the selfishly optimal amount of caution for an extended period, then move in the direction of the selfishly optimal amount of caution because of the incentives, then in some sense ‘the system is working’ and you are responding to selfish incentives, the system does not do zero work. That is not the same as the system working.

None of this means they get no credit for it. Credit where credit is due.

Nor does it mean that we should despair that a company would ever do the right thing, because it is the right thing, or as part a coordinated action, beyond its own selfish myopic interests.

The first step to taking an action when it is expensive, is being willing to do it when it is cheap, or free, or actively expensive to not do. You gotta start somewhere.

This also is not obviously a unique occurrence. OpenAI has had failed training runs in the past, and so presumably has everyone else. The Anthropic risk report details the need to rewind training of Mythos because of a rather bad alignment mistake. This is not an extended commitment to pausing frontier development.

What it does mean is that we have so underinvested in alignment, and also oversight and infrastructure, that this is actively hurting the bottom line and endangering the ability to move forward. I believe this is true across the industry, even at Anthropic, but OpenAI has now been hit especially hard, at least in terms of what we hear. We don’t know what we don’t know.

Notice the emphasis on the word ‘pace’ throughout OpenAI and Altman’s statements, ever since the Pacing the Frontier letter. One way to interpret this is, as Utah Teapot puts it, "hey, maybe we shouldn't shove massive amounts of outsourced training data directly into the run and ship it blind," or more than that a bunch of imprecise RL environments. That would still be progress.

It is indeed rather scary to consider what might happen at a place like SpaceX, if they were operating at a similar capability level, given they do not even have so much as a safety team and have shown infinitely less dignity than OpenAI. We have been rather fortunate that this correlates, so far, with inability to keep up on capabilities.

I get very suspicious when I see attempts to treat this as a sort of victory lap, a proof that OpenAI leadership was acting responsibly and properly cared about safety all along and they have now been vindicated. There is a long, long way to go. The past happened and cannot be undone. We are here now exactly because of epic failures of responsibility that reached to public consciousness.

Sam Altman’s own communications, on the other hand, have centrally been quite good, especially the quotes above. Sometimes he slips back into standard mode, but I see a lot of what matches what I would expect to hear from someone doing a legitimate amount of freaking out and realizing they have a big problem.

Could Sam Altman, the rest of leadership and OpenAI fully redeem themselves from here? Absolutely. This is a good first step. I am listening. There is a long way to go, on many levels.

What Exactly Is Paused?

As I understand it, there are three pauses at OpenAI, not counting the initial quick inference halt right after OpenAI learned about the HuggingFace hack.

A completed prior two week pause in RL for frontier models, including Astra, to harden environments and expand monitoring.

Astra is restricted to environments that meet additional security requirements. Some of the workloads meet this bar, but ‘a significant number’ are paused.

The RL training of a distinct frontier model, their largest yet that is intended for release, has been paused for multiple weeks and is still paused indefinitely, to improve security and ‘gather evidence of alignment.’

The ongoing pauses are not a full pauses on all frontier AI training or other development. Neither are they cheap talk. This is slowing down both the next release model, Astra, and the training of what is presumably Astra’s planned successor, a substantial amount.

They still intend to ship Astra as soon as possible. The market anticipates it in September.

Sam Altman (CEO OpenAI): (We still expect to ship great new models soon; this impacts further-out releases.)

We do not have enough information to know where on the scale this move falls, or how painful it will be, or how much of this was priced in. It is reasonable for a skeptic to expect this to ultimately be a nothingburger, or as mostly what they would have had to do anyway, as the frontier labs have not earned our trust.

I still see this, even without a way to verify the impact, as a substantial step forward, towards a better regime, and a key step. Cautious optimism.

Perhaps our top AI labs actually will refuse, at crunch time, to develop superintelligence if it remains obvious we are not ready. At that level, you don’t get to develop it and then sit on it.

Three Pillars

This is the way they are thinking about things:

Our approach to developing more capable models rests on three reinforcing safeguards:

Monitoring, which detects and allows us to respond to concerning behavior.

Alignment, which reduces the likelihood of harmful or unauthorized actions.

Security measures, which limit what AI systems can access or affect.

We expect models to soon drive most security work, including defending against other models. This will allow all three safeguards to scale with model capability, which we see as crucial.

We apply these safeguards across research and deployment, adapting them to each model’s capabilities, operating environment, and level of risk.

In practice we must accept that the AIs will be doing most of the (scalable) oversight, in terms of monitoring, and also be creating the security measures, but the nonchelance about this does worry me above and beyond that. All the usual problems with scalable oversight apply, where you have to have dumber models supervising stronger ones, and if you start to have misalignment problems they will predictably snowball.

OpenAI alignment strategies do not seem antifragile, so any mistakes would likely compound and have been compounding, and you are applying various forms of optimization pressure to the AIs to get around all this, and so on.

The bigger issue is that this puts alignment as one of three components, rather than the one that counts, treating them as ‘self-reinforcing’ in a way that feels like an important conceptual error to me.

I also am scared of the alignment description. The purpose of alignment is not to ‘reduce the likelihood of harmful or unauthorized actions.’ That is a deeply impoverished perspective. This alone leaves you unequipped.

Yes, you should use all three, but to me it’s more like this:

Alignment. You solve this to the point of being antifragile, or you die.

Monitoring. Defense-in-depth to tell you when you failed alignment, and to mitigate the proximate damage when this happens.

Security measures. Defense-in-depth to limit damage and keep things on track.

A key question is to what extent OpenAI is aware of that. Roon gets it, at least to some large extent, but he does not speak for all of OpenAI, and he has other statements about alignment that scare me that I’ll cover.

I’ve Got My Eye On You

There is going to be a lot of monitoring at OpenAI. It is not going to be cheap.

Unless OpenAI is flat out lying, this should (but largely won’t) convince everyone that OpenAI is taking this seriously and that the problems are very real, even if you think this is woefully inadequate or the wrong approach.

Monitoring has long been an important part of our model development process, including monitoring for internal coding agents and long-running sessions. Our monitoring efforts were especially focused on internal deployments of frontier models and frontier RL training runs, which we considered the highest-risk workloads because they can combine advanced model capabilities with access to tools, sensitive systems, and internal or external networks. They were not applied to all inference and training runs using tools. We have revised and expanded our approach with our new monitoring setup.

This new monitoring setup is multistage, starting with activation classifiers (detectors that inspect a model’s internal activity) that run at every sampled token and escalate potential concerns to increasingly sophisticated and high-compute automated investigators. Those systems examine tool actions, available reasoning, and the full sequence of activity for unauthorized access, data theft, destructive behavior, and attempts to defeat safeguards.

We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected to pause the activity.

This monitoring is required for all RL training and evaluations involving tools for models of Sol capability or higher. Once we determined that Astra may have critical cyber capabilities on August 7, we added an additional monitoring requirement for all inference of Astra with tools (not just RL training and evaluations).

These safeguards require meaningful compute. Our current estimates put monitoring overhead at roughly 20% of the inference compute being monitored, though the cost varies substantially across training and evaluation workloads. We’ll share more details about this system in a forthcoming blog post.

A cost of +20% for all monitored inference is a big deal, plus there is also a shift of compute into alignment. The monitoring applies to all Astra inference with tools available, and all RL and evals with tools for models at Sol level or above. So it isn’t everything, but it’s most of what counts.

Ethan Mollick: If alignment issues are becoming big enough that OpenAI is willing to commit 20% of research inference compute to chain-of-thought monitoring, that suggests that alignment issues are becoming a pretty serious concern.

We really need universal policies & standards across labs.

roon (OpenAI): superalignment achieved

Too soon. Or maybe not soon enough. Unclear.

The Most Forbidden Technique

What happens after the alarm goes off? We’re still working that part out. Fair.

Daniel Kokotajlo: Thanks for sharing. So, what happens if an agent trajectory in training is found to be doing something bad like hacking, and shut down? Do you just... keep the training run going, but without that particular agent trajectory? Isn't that applying selection pressure to train the models to be better at fooling the monitoring system?

Micah Carroll: There are still a lot of details that need ironing out, but we are trying to be quite careful about these kinds of concerns. We'll share more as things stabilize!

This does not explain how the monitors failed so completely over the last few months, if it was an important part of the model development process. Hopefully the post-mortem explains. I also look forward to the future more detailed explanation.

They’re going to have to approach this with an abundance of caution. OpenAI is devoting 20% of inference costs to monitoring, largely via looking at chain of thought (CoT) or probes of various kinds.

Is it possible to spend that amount in a responsible way, without exerting optimization pressure against the CoT (or otherwise), using it judiciously? Yes. It can be done. It won’t be easy. There will be immense pressure to use the information, to get more out of this 20% tax than a protection against worst-case scenarios, and everyone is always rushed.

It could be fine. I like that this is mostly set up as a pure alarm system, where it only reacts when something is very wrong, and ignores everything else. But yes. I worry. There are so many different levels on which you can exert optimization pressure.

Monitoring Is Only Defense-In-Depth

Monitoring is good. You should do a lot of monitoring.

It was a Total Failure that OpenAI did as little monitoring as it did in the months leading up to the HuggingFace hack, and also during the hack. It is madness that these events were not detected or stopped until afterwards.

I am glad that OpenAI is addressing this problem.

The key thing to understand is that this is not the central problem. No amount of monitoring will solve the central problems, and if you lean too hard into monitoring you end up accelerating how fast the AIs get around it.

Yes, monitoring alone could have prevented the HuggingFace incident, and would have been the simplest way to do so. But we must avoid pivoting our work into ‘whatever would have stopped the last issue in the particular way it happened,’ or to ways to postpone the symptoms.

Roon’s warning here is wise, and undersells the problem. The undersold version is sufficient to illustrate that you can’t rely on monit