OpenAI の AI によるハッキング事件を踏まえたフロンティア管理の議論

In the wake of the letter calling on us to prepare to potentially Pace the Frontier, there has been much discussion of when pacing the frontier would be prudent, and whether it makes sense to prepare to do so.

This has now been informed by the events surrounding OpenAI training models for months while they had access to a joint de facto message board, which was detected only in the wake of the hacking of HuggingFace by OpenAI’s AIs models during a cybersecurity eval. As we find out more about that, a lot of people have grown far more alarmed, as they should given what they previously believed about the difficulty of alignment, about the state of capabilities and about the level of operational supervision, infrastructure, safety and safety culture at the frontier labs.

This post will not go further into the details of that incident. It treats that as background to keep in mind, and mostly involves perspectives from before the Black Hat talk. This was originally scheduled for Friday and got bumped.

A lot of the disagreements about the need to pace tie into expectations about the default pace of capability advancements. As I wrote recently in The Three AI Pills, sincere disagreements about AI policy usually boil down to disagreements about the expected pace of progress, and what we expect future AIs will be able to do.

Table of Contents

Danger, Will Robinson.

Progress Fast and Slow.

Statements of Support For Pacing the Frontier.

No One In Charge.

Pacing The Frontier.

Pausing the Frontier.

Senator Sanders Demands A Pause.

Moderate Prudence.

That Escalated Quickly.

If You Are In Mundane Alignment Pivot To Scalable Alignment.

Taking It Fast.

Full Speed Ahead.

Suicide Squad.

Prepare To Adjust Your Pace.

Danger, Will Robinson

There are also disagreements about how dangerous a given level of capability would be, or how it would physically impact the world, and disagreements about what options we have, the nature of various coordination mechanisms or government interventions, and balancing different sacred values. Often people only properly see one half of a key trade-off.

Mostly the reason people often sincerely only see one half of such key tradeoffs is that they anticipate so little AI progress that mitigating existential risks or worrying about humans losing control is unnecessary.

Or they anticipate so much AI progress that mitigating existential and catastrophic risks and maintaining human control has to be the priority, and that necessarily is going to mean some group of people collectively choosing and charting, in some way, a deliberate path through causal space towards outcomes that allow us to survive.

Yes, that necessarily means enabling some group of people to have some collective mechanism to chart some aspects of Earth’s path through causal space, and yes there are reasons to worry about that, but that is why we should work to figure out the best way to do that.

Progress Fast and Slow

Those opposing the Pacing the Frontier letter do not want to ‘slow down’ and instead want AI to go ‘fast,’ but their vision of fast is, while super fast by historic standards, not all that fast.

Those who signed the Pacing the Frontier letter mostly want AI to go at least as fast as the opposition. What they want to avoid is AI going super duper ultra hyper fast.

Daniel Eth (AI Safety): Here’s how I’m interpreting the words:

Pause: step on the brakes

Pace: don’t attach a giant fucking rocket engine on the back of the car that will accelerate us from 65mph to 10,000 mph, at least not unless we can turn it off. also, don’t disable the brakes.

Almost everyone signing the letter wants better chatbots and doctors and science and other forms of diffusion. They don’t want superintelligence and a singularity in 2027.

Nick: a lot of the anti crowd wants better chatbots and doctors and stuff, a lot of the pro crowd expects like way crazier worlds in the short term and wants them to come in the just slightly less short term when we’ve figured out how to control these things better. plenty of overlap

how to do it no idea, and also how to measure the speed limit no idea, but i feel like this framing avoids some of the issues with pause, which also has roughly the same questions

Dean W. Ball: This is what most people I know who signed the “pacing” letter (myself included) think. The slowdown we have in mind is temporary, and to a rate of progress that is still much faster than even today’s rate.

Augmented Fifth: It’s Dec 19, school is out, and the Christmas presents are under the tree. “Let’s wait until February” is not going to fly with the kids.

I quote that last one because what is happening is that the people who are arguing against the Pacing the Frontier letter are saying they won’t wait until February and demanding they get the presents on December 25 and they’d better get that BB gun.

Whereas those signing the letter are saying maybe we should wait until at least tomorrow before we order even more presents and there is no more room left in the house, plus maybe keep the toys reasonable, and only get you the BB gun that’ll shoot your eye out kid and not an AK-47 or tactical nuke or that sexy Von Neumann probe.

Statements of Support For Pacing the Frontier

Another strong comment from one of those who signed the Pacing the Frontier letter:

Drake Thomas (Anthropic): Very happy to have signed the pacing the frontier letter; its existence gives me a lot of hope for humanity’s survival!

I endorse the letter as written (and think, given the constraints, it’s probably close to the best it could be for this level of consensus), but some places where I differ from its connotational tone:

(1) Not only is AI “not guaranteed” to make a dramatically better future, the odds of failure are terrifyingly high: I think* there’s something like a 40% chance we get an outcome around as bad as human extinction or worse, and another 30% chance we get a future that, while containing some good things, falls radically short of what a wiser civilization could have obtained (say, <5% of the value of a truly great future).

(2) I don’t really endorse the vibes of “to realize AI’s potential”; I think a much more immediate and pressing motivation is “to avoid catastrophically bad outcomes from AI that will kill a lot of people”. (The potential is also very important, ofc, but I think most moral theories would view it as being of secondary importance when risks are this high and there’s little that would do more than temporarily delay that potential anyway.)

(3) “address emerging risks, develop security measures, and strengthen oversight” is fine so far as it goes but a little vague. Concrete things I’d like to do with slower AI development: way better interpretability, build up a robust and well-resourced third party ecosystem for independent auditing of AI companies and get lots of reps in for their oversight, put tons of effort into the automation of alignment research, work on governance mechanisms for ASI, develop a vastly better science of the nature and development of AI behavior, build much more powerful control mechanisms, point lots of powerful AI labor at ambitious scalable alignment projects (eg work like ARC’s), deep dives (including external audits) of individual AI behavior incidents, etc. Also getting civilizational biosecurity preparedness in order.

*epistemic status very approximate vibes, my numbers will change day to day and depending on the exact operationalization.

And a very good statement about the actual thing we may be on the verge of doing, and need to prepare to handle, from Samuel Hammond. You get results like this if you keep drawing straight lines on logarithmic graphs and follow the logic of everything involved. You can invoke the absurdity heuristic or ‘bottlenecks’ or what not all you want, but this is what the labs actually expect. The variation Hammond offers has things accelerating quite a bit but is not even fully ASI (superintelligence) pilled about the ultimate ends of this cycle:

Samuel Hammond: On the contrary, I’d argue liberalism originated in the Hobbesian moment when we jointly deferred to a higher power to preserve our agency and avoid killing each other.

Before succumbing to the temptation to naval gaze into the political theory abyss, it’s worth stepping back and clarifying what exactly is happening and being proposed.

Several US companies are on the precipice of fully automating the AI R&D loop, inclusive of pre/post training, env creation, data generation, evals, algorithm and kernel design, systems engineering, architecture search, etc. -- the full stack.

We are already in a regime of weak RSI via partially automated SWEs, but closing the loop altogether represents a difference in degree becoming a difference in kind. The pace of progress will be explosive and potentially uncontrollable.

The US companies closest to this threshold are warning that they are unprepared for a runaway intelligence explosion, and yet feel locked into a prisoners dilemma vis a vis each other and to a lesser extent vis a vis China.

We’ve already seen how rapid and comparatively unbounded progress is in verifiable RL domains, leading to spikey forms of superintelligence in math and cyber, including models that can prove open math conjectures, discover massive speed-ups for breaking encryption, and execute sophisticated multi-step exploits. We’ve also recently seen several severe examples of “loss of control” / misalignment incidents given inadequate monitoring and sandboxing practices relative to model capability. Moreover, these new capabilities mostly stem from scaling-up long-horizon post-training on legacy clusters, with OOMs of new compute about come online / in construction.

In the pre-RSI regime, human frictions created automatic buffers between new model releases, giving researchers and society time to probe emergent capabilities, design better evals, develop novel alignment techniques, and adapt / harden their infrastructure. As progress has accelerated, capability improvements have already started outstripping our adaptive capacity, as manifest in METR’s inability to evaluate model autonomy beyond 13 hours, and narrow window for cyber defenders to prepare for open weight versions of Mythos.

RSI will exacerbate all these issues and create all new ones. At minimum, we should anticipate

• the equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months),
• concurrent algorithmic improvements densifying models to ultra-efficient sizes at any given capability level
• 100x Mythos-like capabilities across most verifiable domains, including chem, nuclear and bio
• new forms of multi-agent misalignment risk
• “company in a box” agents trained to stand-up whole organizations / corporations
• “cyber nuke”-like capabilities that require de minimis infra
• several transformer-scale breakthroughs, such as for long-term memory / continual learning, open-ended domains, and/or all-new training techniques for idealized “GPT-zero”-esque metalearners
• concurrent speedups in any complementary technical domain, i.e. explosive rates of R&D and novel discoveries

It seems to me there is little to lose, and much to gain, from having the social technology to “pace” these developments rather than to let them rip with zero industry / gov’t coordination, particularly as there is technically no law explicitly prohibiting a company from letting an RSI loop run indefinitely and unleashing whatever comes out the other end into the world.

There are innumerable ways an uncoordinated intelligence explosion could become an unmitigated disaster for the cause of liberalism, including runaway power concentration, rapid societal destabilization, rogue AIs / loss of control scenarios, WMD mass proliferation, vulnerable world technologies, and beyond.

Human civilization is about to be forever changed regardless, however if were possible to coordinate the handful of key actors and create artificial “buffers” between each step-change in model capability to enable adaptation, mitigation and alignment research to catch-up, it’s worth a shot.

Given the short-timeline, I think a DPA 708-style agreement is probably our best bet, i.e. an industry consortia with narrow antitrust carveouts for sharing safety and security practices, funding an assurance nonprofit / independent verification organization for 3rd party evals, incident reporting, internal deployment monitoring, standards setting, and enforcing a protocol for coordinated delays / slowdowns, among other things. This still leaves open the China question but that’s a bridge we won’t cross until after solving the collective action problem at home.

I’m open to other approaches / coordination frameworks but this is the object level issue we’re facing. Political theory is great, and I would love to use our limited steering capacity to guide AI development toward a future that maximizes individual liberty, but as a discussion baseline, gesturing at philosophical abstractions is simply non-responsive to the crisis at hand. A red-herring at best, a suicidal circlejerk at worst.

Samuel Hammond: If you do a simple linear regression on model release cadence it predicts a new frontier model will be produced roughly every day by January 2027. Whether a daily release cadence makes any sense is another question. By that point I suspect new models will be private by default.

The most important point here is up top. If things accelerate as described above, I think humanity is probably toast. Even if humanity survives that scenario, your liberal order is most definitely toast. It will seem absurd or incoherent to suggest otherwise, in a world with AIs that are more capable than humans across basically everything and growing more so every day, that have not been meaningfully constrained and cannot be collectively steered.

No One In Charge

Do you think that by ensuring no person is in charge of what the labs do, when the top lab is many current cycles ahead of the next one and the AIs outperform the humans across the board, that you will get a nice, friendly, liberal order of humans?

My actual read is that the thinking is ‘I don’t need to think about that, I just know that what you are proposing sounds bad, so I am against it.’

That reaction is a rejection of the premise of the question. It is refusing to be ASI pilled, even within a hypothetical.

In which case, one should state they are rejecting the premise, but also still answer the hypothetical. If such AIs did come to pass relatively soon, what do you think would happen by default? If you think the answer is still ‘nice, friendly, liberal order of humans,’ then what limitations are you still counting on to ensure this? What would cause it to break down?

Pacing The Frontier

AI Futures Project, the creators of AI 2027 and Plan A, lay out some options for how one might Pace the Future of the Frontier of American AI.

There is extensive analysis of these options at the link. I see the logic in these proposals. I am most interested in option 4, to require safety cases, even though it is harder to implement. Option 2 also appeals (with ideal numbers TBD), to have a minimum compute allocation for alignment and safety efforts, and can be a complement, with the caveat of obvious problems pinning down what that means.

Pausing the Frontier

One can also take the full position, not taken by the Pacing the Frontier letter, that given recent events the frontier capabilities development should be paused now.

This is importantly not what the Pacing the Frontier letter calls for. The Pacing the Frontier letter calls for gaining the capability to pace development later. Those calling for a pause want to set the pace of development to zero, right now.

As always there are three basic objections to this:

Coordination is too hard, incentives do not work, we cannot do it.

Think of the potential, we cannot afford to do it.

There is not enough potential, we do not need to do it.

Plausibly we are now entering the phase where it becomes so dangerous to continue that it is better to pause individually, even without coordination. Which would then make it far easier to coordinate.

Geoffrey Irving: I don’t think it is rational for anyone to be doing capabilities research at a frontier lab right now. We are not in a Prisoners Dilemma: the situation is very dangerous, and if one person or lab stops it makes it easier and more peer-compatible for other people or labs to stop.

A useful clarification is that this conclusion that unilateral stopping is rational is not obvious nor some kind of tautology: it depends on how dangerous the situation is. We’ve seen a lot of danger recently!

Industry coordination is way better than that! U.S. federal laws could be better still! International treaties are even better than that! But those might take time, even years, and it is not clear we have that time.

At some point, ‘you cannot afford to continue’ overwhelms ‘you cannot afford to stop,’ even if both are importantly true. It is now a lot more arguable that we are there.

I do not think we are at that point yet where I agree with Irving, but I am a lot less confident about this than I was two weeks ago. I would not be okay working on resuming capabilities development at OpenAI without assurances well beyond what we have seen in public, and am very glad they are consciously slowing development down.

This is distinct from not releasing things already developed, including GPT-5.6-Cyber and Project Daybreak, which is fine so long as GPT-5.6-Cyber at no point was trained with the message board active.

Senator Sanders Demands A Pause

Senator Bernie Sanders sends a letter to the leaders of the frontier labs, calling outright for them to pause AI development, on account of their promise to do so if they proved unable to control AI. He cites the HuggingFace attack and the recent use of AI to create new viruses.

Dear Mr. Altman, Mr. Amodei and Mr. Zuckerberg:

Almost every day, there is a new story about how your companies are losing control of the AI technology you are developing, with potentially cataclysmic results.

This week we learned, frighteningly, that AI has been used for the first time ever to create new viruses. As you know this type of development, in the wrong hands, could lead to new bioweapons that result in the deaths of tens of millions of people.

Last month, the world found out OpenAI lost control of an AI model. The result? The model hacked into another company’s computers—a clear violation of federal law. After conducting internal reviews, Anthropic and Meta reported their models similarly escaped their control.

One of the targeted companies called the AI hack “an unprecedented event” that deserves an “unprecedented response.” Yoshua Bengio, the most cited living scientist in the world, said these incidents “should serve as a wake-up call.”

I agree.

So do the top scientists at the companies you lead, the very people building this technology. As you know, these technology leaders recently called for the international community to create a safety mechanism—a pause button—to avoid catastrophe. They warned there is a “real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”

And yet, at a moment when we have seen human loss of control and the creation of potentially dangerous viruses, your companies are still racing ahead, investing tens of billions of dollars into a technology that nobody can fully understand, predict or control.

That is absurd, irresponsible and extremely dangerous.