AI #176 パート 2：プラン B

This is part 2 of the weekly, broadly covering speculation, rhetoric and policy, along with alignment research.

This does not cover the release of GPT-5.6-Sol. As always, I will be taking a few days to digest what the new model has to offer and to allow others to try it and react. I will cover Sol and its capabilities early next week. I covered the GPT-5.6 system card back on June 28.

This also does not cover the release of Plan A, the follow-up to AI 2027. This new scenario is a positive vision of what its authors think we should do going forwards.

I do not endorse all of the recommendations or predictions of Plan A, but I do endorse reading Plan A and taking it seriously. Scott Alexander, one of those who worked on it, writes an introduction and justification here. I will have full coverage soon.

Table of Contents

Quiet Speculations. Will our AI regulations be ad hoc indefinitely?

The Goalposts Are Dyson Spheres. This might take a little longer.

People Just Say Things.

Three Pills. Unpilled, AI, AGI, ASI.

The Quest for Sane Regulations. You’ve got to have guardrails.

OpenAI National Security Principles. Good ideas, but how to enforce them?

Greetings From The Department Of War. New primary documents.

Chip City. Nvidia keeps lying right to the government’s face. Also yours.

Open Weight Models Are Unsafe And Nothing Can Fix This. It is time.

Their AI Propaganda Bots. Oh, I’m sorry. This is abuse.

Rhetorical Innovation. Security mindset.

You Learn. But did it really count?

You May Be Tan And Thin And Rich But You’re a Tool. I, on the other hand…

Train Those Thoughts. Yay finding the error, boo the actual error. So confusing.

Train Out Those Thoughts. GRAM as a new training technique.

My Own Private Idaho. You have nothing to hide. Privacy is still worthwhile.

Aligning a Smarter Than Human Intelligence is Difficult. How’m I doin?

No Space Like J-Space. Let’s see, what do we have here? Oh.

Cooperative Alignments. Claims about Sonnet 5.

The Lighter Side. You have not been a good user.

Quiet Speculations

Outgoing White House advisor Sriram Krishnan says Donald Trump will never support a formal licensing regime for AI.

Joe Miller: Donald Trump will not establish a formal licensing regime for AI, the president’s departing AI adviser has said, even as the White House wields emergency powers to stall the most advanced models.

… “This administration, [the] president, from day one has been against burdensome, onerous, bureaucratic red tape,” he added. “We are not in the business of picking winners and losers.”

If true, this means they will instead continue a fully ad hoc regime (e.g. ‘there will be guardrails’), where winners and losers are chosen according to the whims of the administration. Presumably they think that approach has its advantages.

Despite this, Sriram’s statement does not end with And That’s Terrible.

Joe Miller: Krishnan said setting up a centralised agency requiring “a team of lawyers before you can get a model out” would put “sand in the gears” of the AI revolution. “That is never, never going to happen under President Trump,” he said.

Instead, you will have to go through a fully opaque, arbitrary ad hoc process that involves the whims of power. Much better, you see. Krishnan also supports extorting equity from major AI companies, as part of this regime.

Joe Miller: Krishnan warned that if cutting-edge AI tools were held back by the government for several weeks, “that would probably be bad for American innovation”.

Some of us are worried about much worse outcomes than a few weeks of delay. GPT-5.6 is available as of yesterday, so the delay was annoying but not an epic deal.

Joe Miller: Asked whether a future Democratic government could use the Trump administration’s unilateral use of export controls as a pretext to stall the rollout of AI, Krishnan said: “I don’t think about future governments. I think about this government and this moment in time.”

I think we may have found part of the problem, right there.

Tyler Cowen, never stop Tyler Cowening, as he speculates on how AI and the fertility crisis will intersect in ways only he could, holding the world constant so he can center things he finds interesting at the expense of many more important questions. He imagines a world of depopulated cities where people talk to each other and focus on looking and being good, unique and interesting to compete with AI interactions. The first comment claims Tyler is biased in favor of predicting social change, but actually this is Tyler predicting remarkably little social change.

Carlo Cordasco claims that transformative innovations have their costs clear right away, but their benefits only become clear over time. This seems not right, even historically. Instead, Carlo tells the story of various warnings about technology downsides (e.g. Plato on writing) that did not pan out. Such wrong warnings are common, but also often the biggest downsides are missed. That doesn’t mean we are sad about the printing press or radio, or social media and smartphones, or agriculture, or birth control, or burning fossil fuels, and so on, but no we do not see the costs up front. I think it’s better to say that transformational technological impacts are historically hard to predict.

The Goalposts Are Dyson Spheres

Once again we get another entry into ‘oh but you didn’t realize that physical actions take nonzero amounts of time to figure out and test, now did you?’

Epoch AI: Will we get Dyson Spheres a few years after automating AI R&D? Most AI futurism debates answer this by looking at AI capabilities, but miss half the picture: how intrinsically hard it is to build futuristic tech in the first place.

New essay by @datagenproc and @ansonwhho .

Once AI research is automated, AI could rapidly surpass human experts across almost all cognitive domains.

However, that doesn’t tell us how long after that we’d have “sci-fi” tech like dyson spheres, nanotech, near light-speed travel, brain uploading, or interstellar probes.

That doesn’t mean that Dyson Spheres are impossible within a few years of automating AI R&D. It just means that forecasts of speculative technologies should be sensitive to the specifics of how hard they are to build.

They suggest that you think about, given assumptions about how capable is your AI, you look at how hard a specific technology is to build and estimate how long it would take.

Their example is a drop-in worker replacement that runs on an H100.

I mean, yes, sure, that is a good exercise to do, but as framed it presumes the AI is a fixed level of capable. The whole point of recursive self-improvement (RSI) is that the AI gets a lot better.

If you wanted to use H100 drop-in worker replacements to build a Dyson sphere as quickly as possible, any gamer would know that - aside from maybe some long lead time experiments or other groundwork you need to start well in advance - that you don’t start building or even researching the Dyson sphere right away. You keep running up the intelligence tech tree first, well past that point, and only then try to build the sphere. That’s the point of RSI.

They give their third objection, that it’s hard to model superintelligence and that the AIs will start doing things we didn’t anticipate and perhaps cannot imagine, short shrift. It’s clear that the idea is, focus on what we know non-ASI AIs will be able to do, using methods we know about, and then base our plans largely on that, knowing that maybe ASIs could do more.

JS Denain: For example, suppose you had a billion AIs that were each at least as good as top human experts at virtually all cognitive tasks.

If you could show that these AIs could build molecular nanotechnology within a few years, then surely a billion much smarter AIs could too, even if you don’t know how to model them. And if you can show that these AIs probably can’t do this, then at least this helps us identify cruxes, focusing debates on more concrete AI capabilities.

That’s a lower bound, and I would say in this case not such a useful one, and it leads to reinforcing of the idea that the future consists only of implementations of things we already know we can do.

But yes, sure, this kind of thing is sufficiently valuable to decision making that someone should do the estimation work here. It is especially useful because we do not know that we will get superintelligence, and we would like to know what we could still do without it, if either it is too hard to do or we find a way to choose not to build superintelligence for a while.

Or alternatively ask Fable or Sol to do the estimations for you.

People Just Say Things

Fact checker Megan Herbst gets the facts wrong in Wired, claims AI fact checking cannot be trusted based exclusively on AI overviews. If you want to know exactly what the studies found and what it actually means Dan Williams asked Fable for you. Megan then concludes ‘AI is wrong about half the time’ and that all fact checking must be human. This is how a lot of our intellectual elite is handling AI.

Three Pills

The three are: AI pill, AGI pill, ASI pill.

Roughly:

AI pill: AI can do the cognitive things it can already do.

AGI pill: AI will in the future do more of the cognitive things.

ASI pill: AI will in the future do all of the cognitive things.

To count as pilled you must also grapple with the implications. If you then treat this as an isolated curiosity and assume nothing much changes, it doesn’t count.

If you don’t take the ASI pill, you are responding to the wrong long term future.

If you don’t take at least the AGI pill, your AI policy won’t make sense now.

If you don’t take at least the AI pill, you might be most of the government.

One must understand that those who have strongly influenced White House policy flat out refuse to acknowledge the capabilities of existing closed frontier models, and have been playing that game for years.

Dean W. Ball: Basically I think that, back in 2023 or so, the “consistently wrong about AI” VC and SaaS community was operating under the assumption that AI’s trajectory would mean model capabilities peaking around GPT 5.5/Opus 4.8 capabilities somewhere around 2030, plus robots.

And if that was your assumption, I can totally understand why you think everything commodifies/frontier AI isn’t a legitimate business model, etc.

That is a nice world to believe in! In the real world, however, that community has been wildly wrong for three years, and I would expect them to continue being wrong for more years to come.

They may not be wrong forever! Things eventually commodify. But people have been saying “the models are good enough” since GPT-4, and it’s been untrue. I suspect that will continue to be the case because I think that we remain in the earlier stages of the AI industry, and along the steep part of the trajectory.

More broadly: the notion of “good enough” should gross you out, a little bit. The economy of the future will be about heavy-tailed excellence, not middle-of-the-bell-curve, loser-premise, “good enough”-ness.

Daniel Eth (AI Safety): Also explains why they’ve taken this insane, radically anti-regulatory stance towards frontier AI risk

Dean W. Ball: yep. increasingly their discourse feels like it requires disbelieving your eyes, imagining eg a political economy where open-source models profoundly more capable than Mythos/fable are encouraged by governments just as governments begin hardcore regulation.

If you take recursive self-improvement (RSI) and superintelligence (ASI) seriously, this changes everything. We don’t know what level of seriously we need to take them how fast, which contrasts this pill with the first two pills.

prinz: In the age of RSI, the claim that models will commoditize looks increasingly dubious. The gap between the frontier and the second tier is already huge (much larger than the benchmarks suggest), is clearly growing, and will continue to grow at an accelerating pace.

Many will ask: but what about the plethora of enterprise tasks that don’t need a frontier model? What if a fast/cheap model really is good enough for most knowledge work? The answer: RSI implies that the frontier labs will capture the *entirety of the pareto frontier*. They’ll be SOTA on intelligence, but also on speed, and - if competitive forces so dictate - also on cost.

Fully automated AI R&D also likely means that tomorrow’s models will look nothing like the LLMs of today. Some of the gap will consist of novel architectures or techniques, which the second-tier labs will struggle to independently discover and timely implement.

All of the above doesn’t hold if RSI doesn’t work! But if you believe that RSI will work, then model commoditization is likely the wrong bet.

Similarly, people react to this in a lot of different ways:

Jared Friedman: Fable is so insanely good. Deserves the hype.

Paul Graham: Imagine what it will be like if 5 years from now models have improved on Fable as much as Fable has improved on GPT3.

Not AI pilled: Fable is not really thinking or creating anything, who cares?

AI pilled: Fable is good but it is basically the same as other models, whatever.

AGI pilled: Imagine what you could do with an army of even better Fables.

ASI pilled: Imagine what that future Fable will do to you, or to the world.

To illustrate the range of what pace we might expect, here are the first four replies:

Sascha Jürgens: at some point we will not notice the difference any more, at least subjectively, from a human point of view, even though the stats will keep growing exponentially

Dr. Dad, PhD: Even just Fable-level but cheaper and faster would be incredible.

Plastic Soldier: Your base case should be that models will improve even more than that.

Derya Unutmaz, MD: We will see as much advancement in 2 years. There will be more AI progress in the next two years than in the past 5 years, and that’s being conservative.

Derya’s response is correct. AI improvements are accelerating, and we have entered the range where those improvements have a lot more real world impact. Exponentials come at you fast.

Sascha’s response is narrowly correct, at the full limit, or in the context of any given compact task or in a normal conversation. But the impacts on the world won’t stop.

The Quest for Sane Regulations

We need guardrails. You’ve got to have guardrails. Or, in Trump’s language:

Donald Trump (President of the United States): We’re going to have guardrails. We have guardrails. You saw that a couple of weeks ago. We were able to stop something that we didn’t like. And by the way the company [Anthropic] was very good. They were very good, you know that.

But it can be used for tremendous good. Mostly good and some bad, and the bad we have to stop. But it’s a massive industry. And I think things are going to happen.

You know what you are seeing here, I think something could happen in that regard too. With a contribution to the people of our country. I’ll give you a little inside information. Am I allowed to do that? [laughter] Scott, you know what I’m talking about. No, I think you are going to see a contribution made by those, because they’re making tremendous amounts of money.

This could end up meaning pretty much anything. The natural read is that yes, there will be at least some rules involved, but I like to hope that we knew that.

Over a long enough time horizon, yes, AI is kind of a big deal. Our elected representatives are starting to get that, with ‘losing control of AI’ the third biggest issue listed here.

OpenAI National Security Principles

They have them now, with an official statement.

Based on these principles, we will not support use of OpenAI tools for:

● mass domestic surveillance

● high-stakes decisions — including decisions over the use of force — without appropriate human judgment and accountability

● uses that evade legal obligations, oversight, or accountability.​

Those are good principles, indeed the first two are echos of Anthropic’s red lines, but as we saw in the DoW-Anthropic clash, you need to precisely define what you will and won’t do, and you need that to have teeth.

Remember that ‘mass domestic surveillance’ is not defined in American law, and that de facto ‘high stakes’ decisions already get made via AI and other automated systems all the time.

The third clause is interesting and welcome, and again requires careful definition.

We will have a set process to apply this framework to national security and law enforcement opportunities. Over time, these recommendations will establish precedent. Like a common law system, the principles will become clearer through documented applications, creating a body of “case law” over time.​

That is good, but OpenAI does not make the law. If you engage with the government, or if the government wants to engage with you, this fact will loom large.

OpenAI: Our Principles.

We support government uses of AI that benefit people.

We prioritize partners who share our values.

​We have an obligation to ensure AI strengthens democratic institutions and avoids concentration of power.

In national security and law enforcement, AI should support human judgment and maintain human accountability.

‘Share our values’ means aiding the United States and what at least used to be its democratic allies, and empowering them, so they don’t ‘lose’ against authoritarians.

How do they plan to avoid concentration of power and strengthen democratic institutions, in contrast with what seemed to be happening with them and DoW?

Queue the usual talk of all the standard things.

OpenAI will help build that understanding through contractual negotiations, ongoing operational work, knowledge sharing, and public policy debates.

Engagement with select government partners can serve the public good when it helps agencies, policymakers, and oversight institutions understand AI capabilities, limitations, and risks. It should also help identify uses for which AI is not yet ready, reliable enough, or consistent with critical human accountability.​

That engagement should include the democratic institutions responsible for overseeing agencies that use our models, including in law enforcement and national security.

OpenAI is not the right institution to oversee government use of AI. But we can help regulators, auditors, courts, lawmakers, and oversight bodies build the understanding and practical capacity to oversee AI use at speed and scale, including through the use of AI itself. And we can help the public, who are responsible for electing officials, understand the benefits and limitations of AI.

Where legal, compliance, or oversight frameworks are underdeveloped, we should help identify the gaps and advocate for improvements. In some cases, those frameworks may need to change before deployment is appropriate. In others, carefully structured engagement can help agencies identify risks, develop better guidance, and build capacity for responsible use, including in highly sensitive environments where our visibility may appropriately be limited.

Helping authorities and the public understand is good. This still overall sounds, in practice, a lot like beyond that this is more muddling through and passing the buck.

In practice, what has happened so far? OpenAI has cooperated with executive authority, and not done much visible negotiation on the merits, and seems to have signed up for clashes with its values. This is without a regulatory framework, and without anything that looks like proper supervision of the supervisors, and every national security expert I talked to expects OpenAI to not have legal defense if DoW tries to use OpenAI models