AI #161 第2部：AIをめぐるすべての議論

AI discourse. AI discourse never changes.

That’s not actually true. But it is true to a rather frustrating degree, for those of us who need to be in the thick of it all the time. It is especially true if someone says the word ‘pause,’ whether or not they would actually support one. Play it again, Sam.

Meanwhile, you know what changes a lot? Actual AI capabilities. Also, war.

In any case, here’s the policy, discourse and alignment side of the week that was.

Table of Contents

The OpenAI Foundation Exists. It continues not to focus on its supposed purpose.

Congress Exists. There is some attempt to focus on its supposed purpose.

China Self-Owns. China prevents Manus founders from leaving the country.

The Quest for Survival. Attempts to use procurement law as general AI policy.

Alex Bores Watch. He does an interview with Vanity Fair.

You Received The Federal Framework. Business as usual, except for the GSA.

Chip City. We finally catch a major chip smuggling operation.

Water Water Everywhere. Water use as Gell-Mann Amnesia.

Senator Bernie Sanders Acts Authentically. He is rather alarmed by AI.

Pick Up The Phone. Kathleen Hicks tells Axios that China would answer the call.

Rhetorical Innovation. Every debate on pausing AI, or on trying not to die.

I’m A Conscious Robot. Owain Evans finds consciousness claims change behavior.

How To Get Zvi To Read Your Paper. Call it ‘Metagame Matters for Training.’

People Really Hate AI. Will Manidis offers the latest version.

Greetings From The Stop AI Protest. Photos, summarizes and reactions.

Models Have Goals. They don’t appear reliably in every context. But yes.

If I Was Two-Faced Would I Be Wearing This One. Contrasts in quotes.

Other People Are Not As Worried About AI Killing Everyone. Here’s the mic.

The OpenAI Foundation Exists

After OpenAI successfully stole most of the company from its nonprofit, OpenAI’s nonprofit is still left with quite a lot of money. What are they doing with it?

OpenAI Foundation: This work is just beginning. Over the next year, as we quickly ramp up, the Foundation expects to invest at least $1 billion across life sciences and curing diseases, jobs and economic impact, AI resilience, and community programs. This includes early investments toward our previously announced $25 billion commitment to curing diseases and AI resilience.

They picked someone, Jacob Trefethen, to head up the health efforts, who everyone says is highly qualified and an excellent pick for a project like this, although he was plausibly hired in large part exactly to make it harder to call out the dramatic lack of funding for AI safety.

The pick for the ‘AI resilience’ effort is Wojciech Zaremba, an OpenAI cofounder. He has a known history of being dismissive of existential risks early on, but is at least willing to say the risks ‘have to be taken seriously’ now.

David Krueger: Wojciech is one of the various people who called me crazy for being worried about x-risk back “before it was cool”; ICML 2016, New York.

The people left at OpenAI are not the ones who are worried about safety.

Wojciech Zaremba: AI is clearly becoming very advanced and x-risks have to be taken seriously. Tons of important work to be done!

Health is a good cause, and I have hope that within the cause area they will execute well, but it is not the cause for which this foundation exists. You don’t ensure humanity survives AGI, or benefits from AGI, by trying to cure Alzheimer’s.

I’m not saying don’t cure Alzheimer’s, but I am saying You Had One Job, it was rather an important job, and there are a lot of foundations that can work on curing diseases, on public data for health or on accelerating progress on high-mortality and high-burden diseases.

If the OpenAI foundation was spending a lot on AI Safety, and was then also spending this billion on health, I would think that was great. But that’s not how this shapes up.

They then mention jobs and economic impact, and ‘AI resilience’ which they divide into AI impact on children & youth, biosecurity and finally we get AI model safety.

Dean W. Ball: Great news! If this foundation does even a B/B+ job and can avoid being co-opted by the various governments across America and the world that by default will try to plunder its wealth, it will probably do more good than all government action on AI put together.

Well, it’s already had most of its wealth plundered by OpenAI, and it’s already redirecting its remaining wealth mostly to sounds-good generic causes with ‘AI’ stapled onto them. So the track record does not look great so far.

If you read Sam Altman’s announcement post, you see that same old generic ‘AI will bring challenges and complex emergent effects’ language, whereas no, what AI will bring is the risk that it will kill everyone via various paths but you cannot say that out loud even in the context of the foundation’s work.

Sam Altman (CEO OpenAI): AI will help discover new science, such as cures for diseases, which is perhaps the most important way to increase quality of life long-term.

AI will also present new threats to society that we have to address. No company can sufficiently mitigate these on their own; we will need a society-wide response to things like novel bio threats, a massive and fast change to the economy, extremely capable models causing complex emergent effects across society, and more.

These are the areas the OpenAI Foundation will initially focus on, and in my opinion are some of the most important ones for us to get right. The Foundation will spend at least $1 billion over the next year.

That’s a lot of words to say essentially nothing.

It’s also not spending that much money relative to its total assets. Yes, we all know that a billion dollars, unlike a million dollars, is cool. But the foundation, even after it was robbed of the majority of its value, is still worth over $100 billion, so this is promising to spend less than 1% of its value in 2026, mostly on causes unrelated to an impending singularity or the associated risks, whereas OpenAI is aiming to automate AI R&D, and kick off a singularity, in 2028.

Their offer is, once again, mostly nothing.

Kelsey Piper is saying this is good, actually, and the explanation is that she expected an offer of actual nothing, so mostly nothing is an improvement.

Kelsey Piper: I guess I never expected them to fund a penny of safety research but this feels too meta or something to me. I dislike the idea that people doing things that are good is *worse* than if they were doing useless things because it ‘makes it harder to call them out’

Curing diseases is really good! the AI resilience stuff may or may not be good, and can be criticized inasmuch as it is bad. (registering a prediction that it will be not only useless but actively counterproductive/harmful is perfectly valid, though).

Oliver Habryka: I mean, what did you think that opposition-capture would look like? Did you expect the labs would not offer people genuinely good things and do genuinely good things in exchange for immunity from critics?

Like, this is just obviously the level Sam Altman plays at, and it has worked quite well so far. I don’t think I am being too paranoid in thinking that this is a non-trivial fraction of what’s going on.

I agree it’s important to recognize good things as good, and I am not downplaying that here! But in this case the downside of this seeming to me like effective opposition-capture seems just much bigger.

If Sam Altman announced that OpenAI was funding these operations out of its own pocket, in addition to all other efforts, or some other billionaire announced this as a new project, then I would say yes, this is a great thing. Good things are good.

But that’s not what the OpenAI nonprofit was for, nor does the size here reflect the assets and capabilities of the nonprofit. Standard procedure is a legal minimum of spending 5% of assets per year, and if you think superintelligence is a few years away you’ll want to be picking up that pace quite a lot.

David Manheim (Home): The @OpenAI Foundation promised State AGs and the public that in exchange for becoming a for-profit, they would spend 27% of the company’s value charitably.

They need to actually do that.

Here’s a post explaining why they need to go much bigger, quickly.

Most of the foundation’s assets, it seems, have effectively been co-opted and effectively stolen a second time, to be used on things that look good and indeed are good, but that mostly don’t help humanity transition to AGI without us all ceasing to exist, and mostly the assets are not being spent until it is too late to matter. For shame.

Congress Exists

Senator Elissa Slotkin introduces the AI Guardrails Act, to address exactly the issues raised in the clash between Anthropic and DoW, in exactly the correct way to address it, which is that you pass a law setting down rules, in particular to mandate that:

There must always be a human in the kill chain of nuclear weapons.

It codifies DoD Directive 3000.09, and bars DoD from using autonomous weapon systems to employ lethal force without “appropriate levels of human judgment and supervision.”

But this can be waived for up o a year at a time by the Secretary of War if ‘extraordinary national security circumstances’ require it. At this point, we should assume that such emergency powers will be used in the regular course of government, with or without a plausible emergency.

DoW is barred from using AI for domestic monitoring, tracking, profiling, or targeting of people or groups in the United States without an “individualized, articulable legal basis,” regardless of where the data came from.

The ban on nuclear weapon launches looks ironclad, and hopefully that’s one place that it is never relevant since you wouldn’t want to do that anyway. There’s still nothing stopping the humans from essentially listening to AI advice, and that is a highly plausible future scenario.

The other rules look a lot easier to get around.

The core prohibition on surveillance is only on monitoring, tracking, profiling, or targeting people in the United States “without an individualized, articulable legal basis.” One should assume that courts and lawyers will interpret such provisions perversely to favor permissiveness. Similarly, the first amendment protections include ‘solely for the purpose of’ which is the kind of thing very easy to drive a truck through.

For autonomous weapons, DoD would simply declare their levels of judgment ‘appropriate,’ including saying none is an appropriate level, or they can use the waiver. To justify use, the error rate for AI merely has to be as low as the error rate for humans, which is not that high a bar.

I am not a lawyer let alone a national security lawyer, so I am not the right person to suggest alternative legal language. I am confident that, while helpful on the margin, this is not it, which is fine for a mostly symbolic first attempt but not for a real law.

The problem is that you can’t have it both ways, in that you have to pick one:

EITHER the law actually binds the government for real even when expensive…

OR it doesn’t, and the government can do what it wants.

The Constitution, and our entire system of government, is based on door number one. The government often runs up against the First, Second, Fourth and Fifth Amendments, among many other constraints, and finds this super annoying. Good.

China Self-Owns

Meta tried to pay a large amount of money for Manus. China said not so fast.

Bloomberg: Chinese authorities barred two Manus co-founders from leaving the country, the FT reported, heightening scrutiny over Meta’s acquisition of the fast-rising agentic AI startup

Cate Cadell and Nitasha Tiku (WaPo): Authorities in Beijing have barred two executives from a Singapore-based AI firm from leaving China amid a review of the company’s $2 billion acquisition by U.S. social media giant Meta, according to a report by the Financial Times on Wednesday.

Xiao Hong and Ji Yichao — the CEO and chief scientist, respectively, of Manus — were summoned to Beijing this month and questioned over a possible violation of foreign direct investment reporting rules related to the acquisition before being told they could not leave the country, the report said.

Tyler Cowen: Here is more from The Washington Post. In my view, the American lead in AI is somewhat larger than a model comparison alone might suggest.

Dean W. Ball: If we were smart we’d see this as a major self own by China, as natsec-brained public policy so often is. The message the government is sending is: if you ever want to found a company, especially one that makes money on software, move to Singapore first (easier to get GPUs too!).

If you were a bright young Chinese citizen looking to found a technology company, would you want to do it in a place that, were you to succeed, would not let you leave? Would not let you do a highly profitable exit?

It would be one thing if this were a frontier lab, although that would still have a dramatic chilling effect. It’s not even that. It’s Manus. So now every potential founder, no matter what they are building, should think long and hard about this. And yes, I would heed Dean’s advice if I could, and do my building elsewhere.

Dean W. Ball: To be clear, self-defeating, simian natsec policy is one of the things the U.S. and Chinese civilizations share in common. If I were an AI policy planner in Beijing, I would be popping champagne (or the domestic equivalent) over the DoW-Anthropic stuff.

I am always extremely frustrated when Chinese self-sabotage is used to justify American self-sabotage, as if the CCP always makes great decisions.

The Quest for Survival

You can propose a ‘lightweight Federal framework’ for AI while using procurement to regulate by proxy and very clearly violate the framework, especially agenda item 5.

If they don’t regulate AI via law they’re going to end up doing it without law.

Jessica Tillipman: (whispers) the same document that gives you no regulator and broad preemption is also what gives you the Anthropic/Pentagon fiasco and the proposed GSA clause.

The framework is silent on procurement, which is an invitation for more ad hoc governance by contract, with less transparency, less consistency, and less recourse than the regulatory alternative.

Don’t be surprised when the entity with the most leverage in the next transaction produces the exact opposite of this “light touch” deregulatory framework.

I’ll show myself out. Have a good weekend, folks.

Jessica Tillipman: The deadline to submit comments on GSA’s draft AI clause has been extended until April 3, 2026.

Madison Alder: General Services Administration pushed back the timeline for draft contract language that would define the government’s relationship with AI service providers in a major federal acquisition program, citing industry requests for an extension.

… “I don’t think we’ve ever seen something as substantive as this tried to be pushed through a MAS mod,” Kelsey Hayes, a partner with Burr & Foreman with a focus on government contracting advising, told FedScoop.

… “I have worked in GSA consulting for nearly 20 years, and this is the first time I have seen a change of this magnitude introduced in this way,” Aubel said. “Typically, a policy shift of this scope would proceed through formal rulemaking and would not appear unexpectedly in a MAS solicitation refresh.”

As seen last week, Jessica Tillipman explains why the current draft clause is ‘governance by sledgehammer’ and would plausibly make generative AI unprovidable to the Federal Government by any but the worst actors.

Jason Miller points out that both the attempt to murder Anthropic and the new GSA language are interfering with the Trump Administration’s attempt to get industry and federal agencies on board with integrating AI into government. The AI Action Plan looked great for this, but the combination of these actions is scaring potential venders, as well it should.

Jason Miller: “At a time when demonstrated by the Iran war, where they are using AI for all kinds of targeting and it’s been pretty remarkable, the administration is creating an environment where if someone disagrees with you on the terms of a contract and you can’t negotiate a resolution, the answer is to drive the company out of the government and possibly business? What message is that to all American AI firms?” said one industry observer, who requested anonymity for fear of retribution.

No one is under any illusions that the attacks against Anthropic are anything other than retaliation. The GSA extends even more extreme demands to every procurement across government. Why would your company want to risk being next on that list, in the face of a government making logistically impossible demands?

Jason Miller: “It would be very difficult for organizations to comply with the draft AI terms of service as currently written for any number of reasons,” said a second industry observer, who too requested anonymity for fear of retribution. “What we would probably see is if this overarching message from administration continues is vendors and agencies would move away from the Schedules as a pathway to sell or buy AI tools and both would look for other vehicles to consume and sell AI tools.”

Meanwhile, the State Department is launching the Bureau of Emerging Threats. What are the emerging threats? Weaponization of technology, including AI, by Iran, China, Russia, North Korea or terrorists. It’s a full enemy-and-misuse orientation.

That’s a much better idea than not doing it, so I’m glad Marco Rubio has thrown yet another ball into the air.

Alex Bores Watch

Here is an interview in Vanity Fair with Alex Bores, the candidate Meta and a16z are spending untold millions to attack.

If they hadn’t been attacking he wouldn’t be in Vanity Fair. They’re not good at this.

Joshua Achiam: I am pretty sure the effort by the pro-AI lobby to torpedo Alex Bores will later on be widely understood as a pointless own-goal. The ads are Kathryn Hahn in Parks & Rec tier self-parody; there is nothing serious in them.

The politics are so bad on this. AI is unpopular so let’s... double down on making him look like The People’s Champion on fighting AI? Yeah, that’s gonna work in a D+Infinity district in a year where Bernie is telling people we have to stop building datacenters.

The policymaking is also terrible. Dear AI advocates: do you know what the alternative is to an Alex Bores, who seems to actually be informed, thoughtful, credible, and productive on this? You will not like it!

(And I say this as someone currently mildly miffed at him for mischaracterizing OpenAI’s safety position. He is still ten thousand times more likely to be reasonable than other folks who will come along after him!)

I get that the big bone of contention is that he has quasi-EA affiliation vibes through AI safety but we have got to stop treating EA as the ultimate boogeyman. Also politically no one knows what EA is or who SBF is. This fails to work on basically every level.

I feel fated to watch smart well-intentioned people on all sides do the “People’s Front of Judea” sketch 24/7 and designate highly-compatible colleagues who are one degree left or right of them as eternal foes who must be fought to the death.

Daniel Eth (yes, Eth is my actual last name): Major respect to Joshua for this thread calling out the efforts to attack Alex Bores, being led by OpenAI & a16z. The thread is right on its merits re policymaking, & also has wise points regarding the politics of AI. I encourage more OpenAI employees to speak out against this!

Peter Wildeford: Completely agree. The SuperPAC run by OpenAI’s Greg Brockman, a16z’s Marc Andreessen, and Palantir’s Joe Londsale is shamelessly brazen, terrible at both politics and policy. It doesn’t take an OpenAI Chief Futurist to see this but it’s