Zvi が提唱する「AI に関する三つの薬」：未来能力への見解を分類

Sincere disagreements about AI are usually disagreements about future AI capabilities.

There are roughly four positions people take. Two are reasonable. Two are not.

I distinguish these via the Three AI Pills. You can take zero, one, two or three.

Three Pills

The three pills are, roughly, taking each of the following three things seriously:

AI pilled. AI exists and can do the things it can already do.

AGI pilled. AI will be able to do a lot more of the things.

ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

The Unpill People

I see unpilled people.

Where do I see them? Everywhere. The majority of people have not taken the first pill.

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure anywhere as ‘what AI can do.’

They dismiss AI as worthless because it pointed them to a closed store or recommended the wrong number of pizzas. They cite old studies that were obsolete before they were published and used terrible prompting techniques.

They often still talk about ‘stochastic parrots’ or how AI can never possibly think and everything must be stolen from the training data. And so on.

Some versions of this are wrong. Some are Not Even Wrong. None are reasonable.

When you discuss AI with people who are fully unpilled, your goal is usually to first give them the AI pill. Show them that AI can do the things it can already do.

The AI Pill

Even fully taking the first AI pill is a big deal.

Existing AI unlocks, today, in practice, tons of cool things. It is, in many ways, already smarter and more capable than you.

So many things that you used to do by hand, or some other way, are now better done by typing a quick request into a text box.

So many things that previously were not worth doing are now worth doing.

So many questions previously not worth asking are now worth asking.

The marginal cost of seeing what the AI can do for you is often very close to zero.

AI can also do a variety of harmful things, or do things that are useful for you but make others worse off or disrupt or invalidate norms or systems. People don’t like that.

Most economists, and most people who work in policy and government, have taken at most this first pill, and underestimate even the impacts of the first pill alone.

Often they say things like ‘AI will be too expensive to use on [X]’ because they don’t realize it will soon be orders of magnitude cheaper for the same level of intelligence. Or they point to particular details where AI does poorly, and presume this will not be fixed. They see AI as ‘uncompetitive’ without realizing the situation is temporary.

When you discuss AI with someone who has taken only the first pill, you typically have three basic options.

You can try to ‘fully AI pill’ them and explain the things AI can already do and the implications of what that means, even if things stop here.

You can try to explain that we will get better at using what AI we have, and that there is a lot of ‘unhobbling’ left for us to do, even if things stop here.

You can explain that things will not stop here, and you need to be thinking about what future AIs will be able to do. Get them to take at least the AGI pill.

Even if AI could permanently only do the things it can currently do, that would be Internet big, and radically change the world, mostly for the better.

AI capabilities will not permanently stop here. It is wrong to not take the second pill.

Stuck At The First Pill

Our debates about AI remain largely stuck on settled questions, because so many people cannot even take the first pill.

Dean W. Ball: A couple years ago, the AI debate was centered, rightfully, on whether crazy-sounding things like “AIs autonomously making math breakthroughs” and “AIs breaking from their sandbox and hacking on the internet” would be real things in the near term. Sometimes it feels like that’s still the debate we’re having. This can be frustrating, because in my view, that debate is settled and was settled quite a while ago.

To have good discussions, we need to at least take the second pill.

Whereas, yes, many people, even many who work with AI, really do say current AI is ‘good enough’ and can’t imagine what a better one can do. As in, someone tweeting at Sam Altman saying ‘Sol does everything I want it to do, this is all I ever need’ and Altman retweeting saying they were wrong. Which they obviously are.

The AGI Pill

The AGI pill is a much bigger deal than the AI pill.

If you take the AGI pill, you understand that AI is advancing its capabilities rapidly.

Even if you think that such AGIs will remain fully under human control, and remain ‘mere tools,’ and you expect the lived experience of most people’s everyday lives to not change so radically, you understand that their capabilities will ‘change everything.’

You see that we will face times of great transition and uncertainty, that have the potential to go extremely badly, and that those who succeed at AI will leave those who do not behind in the dust.

The world in the future will be very different from our own. AIs will be able to do most digital work, most of the time, along with the inevitable robots and self-driving cars and so on. Lots of current jobs will go away, whether or not they are replaced by new and potentially better ones. Economic growth and productivity will accelerate.

You see some of the dangers of what would happen if we empowered misuse of such advanced AI systems before we were ready, especially in places like cyber and bio risk.

You see the potential for centralization of power, or inequality, and also for some forms of runaway gradual disempowerment.

You see the potential for mass unemployment, either transitional or permanent.

You understand that our legal and regulatory regimes are not ready, either to protect against and mitigate the risks and harms, or to allow for the opportunities and remove the bottlenecks to diffusion and mundane utility.

The Need To Be Prepared

Those who expect AI to quickly become sufficiently advanced to greatly impact the physical world usually see great danger. They notice that as a result everyone may soon die. Usually they think this is bad, actually.

Thus such folks call to take coordinated action to mitigate the downside risks of such impacts, keep us all from dying, and ideally also to help capture the upside benefits.

Those who expect AI to become importantly more advanced, but with a slower and smaller impact on the physical world, and who think the practical value of more intelligence will cap out.

For different values of ‘sufficiently advanced,’ as in AGI versus ASI, you would see different degrees of danger.

The AGI pill is still sufficient for most things in the Overton window or under serious consideration as of August 2026. We are almost entirely considering overdetermined, low cost, high benefit interventions.

There is no good case for not doing radically more investment in alignment, infrastructure and oversight, state capacity, transparency, liability, disclosures, safety testing including of internal models, red teaming, auditing, enforcement of export controls and laying the groundwork for diplomacy.

This includes laying the groundwork to Pace the Frontier should that prove necessary.

If you are fully ASI pilled, and realistic about the current state of alignment and how superintelligence likely plays out if it arrives soon, then you will want to go further. You will want to do things that have real downsides, and require real tradeoffs.

Some such people want to do a full international pause of frontier AI development. If you took the full ASI pill and believed what they do about superintelligence, in terms of how fast it might arrive and what it can do, you might well agree with them.

The ASI Pill

The ASI pill is the understanding that AI is on pace to be able to do approximately all of the things better than you.

I said ‘approximately.’ As I go over in detail, that does not mean literally all of the things. There are some things that inherently require or greatly benefit from being a human. And there may be weird corner cases where the AI won’t be good enough.

It does not mean omnipotence or omniscience, although one should expect it to look a lot like that to an unaided human.

It does mean the AI takes your job, and then takes the new job that you switch into, unless you pivot to ‘requires literal human.’ You will be uncompetitive at essentially any other task.

It does mean that it will use this capability to figure out approximately all of the things, remarkably quickly, until you hit the physical limits.

It does mean that those who rely more on such AIs will reliably outcompete, in all senses including for resources, those that rely on such AIs less.

It does mean that, in a ‘fair fight’ or sufficiently open competition, the AI wins.

It also means the AIs often figuring out and doing things you did not imagine or anticipate.

It means realizing that intelligence does not stop anywhere near the human level, nor does its ability to chart paths through causal space towards preferred arrangements of atoms.

It also means not pretending that its superior intellect can be matched by your puny weapons, or your pieces of ink on paper, or your entries in a database, or your regulatory capture and rent seeking.

And Then Nothing Much Changes For You

Despite all that, the sign of the AGI pill, as opposed to the ASI pill, is the belief that day to day life will continue to look similar to how it looks now, in the sense that we see day to day life in 1926 as not that different from life in 2026.

Sometimes this is clearly disingenuous, as it is coming from someone ASI pilled enough to know better. I read Sam Altman saying various forms of ‘life will not much change after the singularity’ as basically unjustified Obvious Nonsense to reassure his audiences, rather than a coherent position. Sometimes the person such folks want to reassure most is themselves.

Others sometimes do have something more concrete in mind as their vision of the future, and why it will not change so much, with varying degrees of coherence.

Those with only the AGI pill believe in bottlenecks that hold back change.

They often believe that our ability to exponentially grow AI’s capacity and capabilities will hit various physical limits. There can only be so many chips. Actions take time. Things too far out there are often pejoratively dismissed as ‘magic.’

They often believe there is not that much left to physically discover, in the classic ‘close the patent office’ kind of way, even in theory. Your steak can only be so tender, your lobster so buttery, your lifespan so long, and your status so high, so why does it matter. I strongly disagree on lifespan and health, and expect we have a long way to go in so many other ways in terms of finding value, although they may have a point about moment-to-moment maximal hedonic experiences of a physical human brain.

They often believe that the upside of intelligence is importantly limited. That no mind, however advanced, could be all that persuasive, or that economically valuable, or that capable of creating innovations in the physical world, or of running sufficiently accurate simulations, or making sufficiently strong predictions, or even able to do things like overcome red tape and regulatory capture.

Intelligence Denialism

I sometimes call this Intelligence Denialism: The idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.

Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.

But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.

This is, usually, the central good reason people who are AGI pilled do not take the ASI pill. They are unable to understand that superintelligence is a thing.

This confusion is also often behind the instinct that models will commoditize. There is an ideal thing, ‘intelligence,’ and you approach it via an asymptote. More is impossible, or more won’t help you, depending on how you frame it.

Superintelligence Versus Omniscience and Omnipotence

A common error is to think that anything superintelligent would be omniscient or even omnipotent, and therefore superintelligence is impossible.

Whereas, again, there is a lot of ‘space above humans’ without becoming omni.

There are many other similar confusions, where ‘there is some upper bound to [X]’ is confused with ‘we are at or near the upper bound to [X].’

Here Adi gives us a way into explaining what’s happening, by doing what from my perspective is a mirror image of the central motte-and-bailey.

Both versions do exist.

Timothy B. Lee: There’s an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don’t.

Adi: There’s a motte-and-bailey trick being played between two definitions of “superintelligence”

“Big-S Superintelligence”, which is definitionally omnipotent

And “small-s superintelligence”, which is the hypothesized end state of the current thread of AI research

It’s surprising how uncritically the AI safety community treats these dubious equivalences, especially since one of the most famous treatments of the motte-and-bailey fallacy comes from none other than Scott Alexander, who’s quite prominent in the community.

As in, I see the following a lot:

Zvi Mowshowitz: The motte-and-bailey here is real, but more commonly it is in reverse. as in a form of:

1. You claim superintelligence [S] soon will run circles around humans and do absurd stuff.
2. But [S] would not be fully omniscient/omnipotent.
3. Therefore [S] would be a ~normal tech.

No.

Adi: I’ve seen this too, and I agree that it’s equally wrong

There will still be things that require non-trivial real world time to achieve. There will be particular actions and paths and methods that have their limitations. These limitations matter. Fully abstracting them away can be a large mistake.

I constantly see versions of ‘the mind that is maximally good still could not do [thing it seems to me it could obviously do]’ or even [thing it can already do] and I think this form of conjunction is a lot of why, and how people justify themselves. When AI gains capabilities, and more things fall, they move the goalposts but don’t change the game.

Adi’s comment above was in the context of discussing Plan A, where the AIs are very obviously not ‘Big-S Superintelligence’ in Adi’s lexicon. There are practical things that these AIs cannot do, or can only do up to a point.

The mistake was not being made. It sometimes gets made, but this is rare.

Persuasion Persuasion (A Worked Example)

At most one can claim some particular capability, typically very effective persuasion (often ‘super-persuasion’), would not be possible. Indeed, a common argument states that even an idealized AI would be unable to be superior to the best human persuader, or even a typical good human persuader (who is much worse than the historical best human persuaders), despite it having access to many advantages over humans, such as much faster thinking speed and access to information, including every little reaction of its target including body language.

Often this is done by citing that context matters for persuasion. Many talk as if this context would trump everything, despite this not being true for humans. They also believe that highly capable AIs could not engineer favorable contexts.

Often it’s a form of ‘I would simply choose not to be persuaded’ or ‘if you can’t do it cold and one shot everyone with only text it does not count’ or something like that.

If it needs a human presence to do the persuasion, or a social context, it can acquire one easily enough in various ways.

I flat out cannot understand why someone would think advanced AIs will remain relatively unpersuasive, other than to think that AI will not get much more capable than it already is, and that’s if we fully restrict the AI to using known standard persuasion techniques and rule out any wizardry.

Yet I am not a superhuman persuader, so many find my arguments unconvincing.

No, seriously, as in Adi is citing the highlighted passage as if this was not very obviously true, my lord, how can you think this is not going to happen in a scenario where AI capabilities continue to develop apace, if anything 2035 seems crazy slow:

This just isn’t much of a threshold. Nor does it require that the AI do this from pure text box versus what a human can do in person. It just says that, given access to similar tools, the AI will be able to do considerably better than any human. How could you not think this, given the rest of the scenario, even if all it does is have all the advantages and skills of the various different humans, plus the ability to process a lot more information a lot faster?

And like, worst case, the AI gets an earpiece and tells a human what to do, Jesus.

Another way of thinking about this:

Adrien Ecoffet: Could a superpersuasive AI get a high speed rail line between SF and LA constructed for under $50B and under 10 years with existing construction technology?

Dean W. Ball: No

Nat McAleese: @deanwball

I think this is an important question not to be overconfident on! And if it feels obvious, you are surely overconfident.

Dean W. Ball: obviously there are no guarantees, but “frontier ai systems reforming ceqa and then cleanly funneling tens of billions of dollars through California’s government” doesn’t seem realistic + like a thing to prioritize compared to many other things

My answer, for a sufficiently persuasive AI is essentially: Yes, but not without incidentally taking over at least the California government. Instrumental convergence.

Not that you would then choose to focus on High Speed Rail, as Dean replies. This is not a case of would, so much as could. The point is the could, in theory.

An example that is almost too perfect an example of ‘you will never be able to do [thing that you can already do]’:

Ramez Naam: “No matter how high quality your tokens are, they cannot turn lead into gold.” Pleasantly unhinged and overall correct.

The Singularity is Nearer: In The Metamorphosis of Prime Intellect, the hard takeoff works because AI discovers the correlation effect, some quantum trick to manipulate matter. In reality, there is no correlation effect. No matter how high quality your tokens are, they cannot turn lead into gold.

Ryan Greenblatt: I get the idea, but this is a funny example because it is totally feasible to turn lead into gold. Extremely capable AI systems (after tech R&D and infrastructure build out) will have the capacity to very quickly build infrastructure for turning lead to gold.

We already have successfully turned lead into