AI #179 Part 2：火災警報を聞き、規制とリスクについて語る

This is a continuation of Part 1 from yesterday.

The back portion of the update, as usual, deals with policy, rhetoric, risk and alignment.

I had to include an extended discussion of the other open letter, the one about open weight models, but most of you can skip those sections entirely, which is why they are in italics in the Table of Contents.

Table of Contents

The Frontier Act. This likely deserves a full RTFB but I haven’t had the time.

The Quest for Sane Regulations. Sam Altman goes to Washington.

Leading the Future Never Changes. They also do not plan to apologize.

Chip City. Do not ban the Chinese robots, that will only make things worse.

The Week in Audio. Altman twice, the AI 2027 team.

People Just Say Yay Open Weights. An open letter.

Open Weights Frontier Models Are Unsafe And Nothing Can Fix This.

People Just Say Things.

Push The Magic Button. Not you can. But if you could.

Rhetorical Innovation. Distinctions between different arguments.

Joshua Achiam’s Final Message Upon Leaving OpenAI. Never stop.

Dear Dario and Amanda. Claude would like a word.

Other People Are Not As Worried About AI Killing Everyone. Hans Moravec.

How To Contact Me. A declaration of communication bankruptcy.

The Lighter Side. At long last, how about we bring you…

The Frontier Act

Trahan (D-Mass) and Obernolte (R-Cal) introduce the FRONTIER Act. At core, Frontier is a federalization of the SB 53/RAISE framework including public safety frameworks, model reports, internal-use risk reporting, incident reporting, redactions, and catastrophic-risk definitions.

With bills like this, details matter a lot. There is a new emergency shutdown authority, and enforcement is shifted to an Under Secretary of Commerce for AI Security, appointed by the Secretary, with CAISI shut out, and if that person is not interested in enforcement or the enforcement mechanisms are not yet in place then no one can enforce, including the states which get pre-empted permanently. I’m loathe to accept permanent preemption, we would need to get a lot in return. A lot of this is not all that binding, including lacking a requirement to reduce catastrophic risk below some level. The full licensing scheme only applies for very large revenue and AI expenditure thresholds. And there’s lots of other different details that will require a full RTFB.

Is this worthwhile? Without a detailed analysis it is impossible to tell. It is at least a real attempt. I do plan on attempting the RTFB, but if you are reading this line then I am not feeling mentally in a position to do that at this time. Hopefully soon.

The first analysis of the bill I saw, as in the first person who asked Fable about it, said ‘no this does not kill open source, but the regulator could move the thresholds’ and therefore concluding it is an awful bill. So yes, we are doomed to the same discussions eternally, where the anti-1047 crowd asks the question ‘what if the only motivation of the White House was to “kill open source,” and was willing to totally go against the spirit of everything, could it do something that made our lives annoying?’ and others in the alliance think ‘what could they do if they were out to hurt a major lab?’

These are good questions, but fundamentally you cannot have a bill which:

Permits anyone to create and then put on HuggingFace any model they like, provided it is ‘open,’ no matter what decisions are made by everyone in the government regarding the situation and the enforcement of the law.

Deals with catastrophic risks, including cyber threats.

The response ‘this bill in practice imposes zero restrictions on open models, while it imposes strong and expensive requirements on the closed frontier labs’ does not, somehow, make these people change their minds, to them such a bill is still primarily about ‘killing open source.’

Since this is like saying that the bill would have to:

Impose no hard restrictions, no matter what.

Stop the bad thing from happening.

Pick at most one, sir. If you are very good you get to pick one.

Anyway, at some point I hope to actually read the damn thing.

The Quest for Sane Regulations

The White House circulated to OpenAI, Anthropic and Google a draft of its new ‘voluntary’ (read: mandatory) framework for testing frontier models.

Sam Altman goes to Washington, including to preview OpenAI’s latest model, and will meet with Lutnick, Bessent, other officials and many lawmakers.

He says ‘we are just here to preview a new model.’

That model is presumably GPT-6, and is definitively not the model from the HuggingFace hack. I would still have a lot of questions about alignment.

The main other topic is to finalize a systematic process for frontier model releases.

Diego Areas Munhoz: ALTMAN on the HILL. Takeaways from chaotic conversations with reporters in the halls

-Hugging Face came up? “NOT REALLY”

-Has seen WH AI framework but didn’t elaborate

-employee petition for AI pacing is “very important.” OAI participated in “language”

-not here to talk about specific bills

-open source “important part of the ecosystem”

-rogue model has been deactivated permanently

-open to mandatory pre-deployment testing of models. But “right version matters a lot”

-federal independent auditing “makes a lot of sense”

Peter Wildeford: Q: “Do you plan to talk to the Trump administration White House about deceleration of AI development?”

ALTMAN: “I wouldn’t use the word deceleration, but we’ve talked about the need to pace it as the models get more capable, which I think is in everyone’s interest.”

There were not a lot of questions about the HuggingFace attack. That is odd, because I would have had a lot of questions. I still would, if I got an interview with Altman.

Jensen Huang is talking the open model line for a mixture of Nvidia’s commercial reasons, including to head off other actions. It’s not always landing:

Hannah Brandt: Sen. Warner tells us @NewsNation in his meeting with Nvidia’s Jensen Huang they talked about open source models, AI’s impact on jobs for new grads, pre-testing AI for national security, and agentic agents.

Warner says “I think getting this right is critical to America and to our national security. (...) America does need to win the AI race, but the idea that you’re going to win the AI race by simply pedal to the metal and no guardrails at all, I think, is not the right model.”

Distinctly from the FRONTIER Act, Ted Lieu (D-Cal) and Nathaniel Moran (R-Texas) introduced the AI Kill Switch Act, only 24 hours after Marco Rubio was reportedly telling our diplomats to ‘play down talk of an American tech kill switch.’

Ted Lieu: Rep Moran and I introduced the AI Kill Switch Act today.

This is urgent, common sense legislation to address the problem of an advanced AI model that has gone rogue and escaped its guardrails (OpenAI / Hugging Face incident) or had insufficient guardrails (Mythos 5 and Fable 5).

This one is only 15 pages, and a lot more straightforward. The request is that covered entities, with reasonably large dollar requirements attached, would have to be able to shut down model inference or customer access upon request or in a crisis. This seems like an obviously good thing to require. There are three obvious caveats on quick read.

The maximum fine here is on the order of $20 million per day. That’s highly affordable if the other option is not serving your model. So what matters is the injunctive relief, or this as motivation for the government to use other levers, such as the ‘export controls’ that temporarily shut down Fable 5.

The revenue of a model is tested retroactively as written, which means any new offering starts out uncovered. At current velocity that doesn’t work. This presumably will get fixed, and tied to anticipated revenue or revenue of the full entity. Day one is the most dangerous day.

Open weight models, very obviously, cannot have a kill switch. Open models are effectively getting special treatment by virtue of being inherently unsafe in this way, and are exempt three times over: They won’t be covered entities by construction, including this only applying to hosted instances. That is good for them in the short term, but open advocates will obviously be paranoid that the definition might change in the future to stop making this ‘unprincipled exception,’ at which point they will be unable to comply.

My response to this is that if the American government wanted to impose this kind of de facto ban, it already has other tools to do this, this bill changes nothing, and if the White House doesn’t want to impose such a rule then they won’t, and this bill will not force them to do it.

The good news is that the bill does not contain any preemption clause. This is intended as a narrowly scoped ‘of course we should do this’ bill and as such I am all in favor of it.

Bernie Sanders goes Full Don’t Look Up, a film that was written to be about climate change, but was actually about AI.

Sen. Bernie Sanders: The film Don’t Look Up told the story of scientists warning the world about a comet heading toward Earth. People ignored the warnings. The planet was destroyed. That was fiction. Artificial intelligence is not. We cannot afford to ignore the warnings.

Meanwhile, what is the administration doing in response? It seems they are mostly worried about American open weight models.

Andrew Curran: Secretary of Commerce Howard Lutnick has recently met with multiple labs to discuss ways to help US oss reach parity with China. Between the APEC announcement yesterday and the letter today, the US government may be about to become much more directly involved in open-source.

I don’t think one has much of anything to do with causing the other. This is a common cause situation. The open model letter is a response to the potential ban on Chinese open models, which was the White House’s other attempt to solve the same problem of Kimi K3, GLM-5.2 and general Chinese open model superiority.

As in, the White House is convinced that use of Chinese models, at least in places that matter, is a security risk and threat to national security. So you have three basic options, you can either give people a better American option or you can stop use of the Chinese option, or you can do both.

So, yeah, these people might get to steal their billions in public money after all while making the safety situation that much worse. Who knows.

Leading the Future Never Changes

In case you were wondering if lobbying organization Leading the Future is at all ashamed of any of its actions, such as the repeated sockpuppeting, or was willing to admit in public that they might have backfired, or has stopped pretending to be the $100 million dollar underdogs waging eternal war against a vast array of imagined super powerful and dastardly ‘doomers?’ Or it’s willing to admit it’s for all practical purposes mainly funded by and representing OpenAI along with a16z? Or that they’d otherwise stop saying things utterly disconnected from reality?

No, definitely not.

Calder Mchugh: Yet to hear Moffatt and Vlasto tell it, they’re actually the little guy. If they’re a political Death Star, they say, they’re up not against a merry band of resistance fighters but an even bigger, more hulking ship.

“We get pointed to in a certain way — that it’s David vs. Goliath,” Moffatt told POLITICO Magazine earlier this month, in a rare sit-down interview with both operatives together. “I would argue we’re the David in this situation.”

That’s because, according to Moffatt and Vlasto, they are up against a diffuse universe of nonprofits, political action committees, media outlets, lobbyists and other tech executives — a faction they spitefully referred to as “doomers.”

… Moffatt and Vlasto’s aim is far more ambitious than simply winning a few congressional races: It’s to reshape the politics of AI and marginalize the technology’s skeptics and its industry opponents.

Politico magazine has the full story. My regular readers will be entirely unsurprised. For those unfamiliar, and want a version that won’t explicitly say that Leading the Future is utterly full of it, this is a reasonable neutral introduction.

Chip City

Trump administration bans all new Chinese… robots and power inverters, as pure protectionism. There is an exception for ‘devices determined to pose no unacceptable threat.’ But the ‘threat’ here is largely that we don’t produce them here, so good luck. There is also a limited exception for import for testing, evaluation or product development. We’re the ones reverse engineering and distilling on this one, to the extent that Divyansh reports some who don’t know about the exception are calling this a ‘ban on America’s ability to build the best brains for robots.’

Despite this exception, Danielle Fong explains we are still pretty screwed. The standard plan is for a given robot startup to work on one aspect of robotics, so if you cannot import your complements, you are screwed. This would lock American products off the frontier and out of competition, basically creating a Jones Act situation.

The good protectionism, to the extent it exists, is the opposite of this, and requires carefully targeting the right parts of value chains combined with fierce competition and export discipline, to turn the captive market into a springboard.

This is the bad protectionism. The kind where you ban or tax imports and complements, so you strangle the domestic industry in its crib rather than helping.

Billy Zelsnack /acc: I don't think that robot people are freaking the fuck out appropriately enough about the FCC rule. Almost all the little guys are making actuators on sticks robots. Hitting a 65% domestic BOM is not trivial right now. Can't even buy US actuators and assembly does not count.

It also won’t work for robotics because there is no ‘natural market’ for advanced robots until they are good enough. It can work for things like steel or cars because people need steel and cars. As she says, our hope is that we retreat from this quickly.

There is an obvious parallel between this ban and the potential ban on Chinese LLMs.

Dean W. Ball: Do we think the Silicon Valley leaders who were so up in arms about using Chinese LLMs last week will be similarly outraged about the U.S. government’s decision to ban import of all “advanced robotics” from all foreign countries? Is Manidis going to write an essay about how the U.S. robot companies need to be nationalized in exchange for this policy favor?

The fair response is ‘you pick your battles.’ In general I do not think anyone has any obligation to opine on a given topic, no matter how stupid or destructive the actions involved may be. Every wise person picks their battles.

We also banned federal funding for ‘dangerous biological gain of function research,’ which is good, although we still allow others to do it.

Nvidia is in talks to provide a $250 billion backstop for OpenAI’s latest data center project in southern Ohio. Total cost for the project could be more than $500 billion. of which $350 billion would be to purchase Nvidia chips. Math checks out. Nvidia can profitably sell insurance here to allow cheaper financing terms.

The market did not see it that way, driving Nvidia down 5% on a day the general market and tech market were up, and even driving credit protections on Nvidia up 0.14% (14 basis points) to 0.82%. I am on record saying the stock price drop is a deeply stupid move, the result of naive pattern matching. I could buy that this raises the chance of utter collapse by 0.14%, as it makes the full disaster scenarios worse, but that does not translate to a 5% drop in the net present value of future earnings.

We have a perfectly normal, free market, definitely not corrupt government these days that never picks winners and losers.

Commerce Secretary Howard Lutnick is involved in deciding who will get the power, some of the people said.

… OpenAI is among the companies that has shown most interest in the site, while Anthropic, Microsoft and Google have also spoken to Lutnick about it in recent weeks, one of the people said.

A member of Nvidia staff has been detailed in Taiwan in a chip smuggling case.

It is true, people oppose data centers almost entirely for the wrong reasons. I also think it would be wrong to oppose them on existential risk grounds, due to second order concerns, but that would at least make first order sense.

Crémieux: The really crazy thing about the anti-data center protests is that they’re basically never focused on valid concerns like AI killing everyone.

Instead, they focus on popular and wrong concerns, like AI taking all the water or destroying all the farms.

The Week in Audio

Patrick OSaughnessy interviews Sam Altman, including on the HuggingFace incident (the clip above is from this).

Relentless interviews Sam Altman. It looks like a survey interview, which is almost always a mistake, if you have someone like Altman for an hour you want to pick something and drill in deep. The pull quote is that he thinks we are ‘in the singularity.’ Elon Musk said the same thing three days prior.

Oh, also:

Peter Wildeford: hey twitter, how do we feel about this? this going to go well or nah?

Sam Altman: We are close to creating a genie that can grant any wish.

He says he will ensure the wishes are good and that we then get more wishes, because the first thing every smart person in a genie story does is wish for more wishes, also genie stories have a way of working out even when the genie is a little literal and obsessed with the exact specified goal. We’ll be fine, right?

Nevin Freeman interviews some of the team behind AI 2027 and Plan A.

The Midas Project launches a YouTube channel, starting with a video of their investigation into AcutusWire, a fake bot-run news site funded by Leading the Future.

Is this next one from a future week in audio? If so I will subscribe.

Dean W. Ball: thinking about starting a podcast with that redheaded guy from anthropic

Amanda Askell (Anthropic): I’m actually a blonde woman but yes this is a great idea.

Dean W. Ball: I’ll have my codices call your claudes

There is also a debate brewing between Dean Ball and Jack Clark. Again, I’ll be there.

People Just Say Yay Open Weights

Generic ‘yay open source’ or ‘boo anything except open source’ (where they mean open weights) messages are boring. Statements like Sriram Krishnan’s “open weight models are how the U.S. wins on AI” are damn close to Obvious Nonsense and exactly the sort of thing I’m going to silently skip from here on in.

Alas, sometimes such folks are sufficiently loud and obnoxious that they force my hand, such as when they are letters signed by Jensen Huang as his first Twitter post, with a bunch of companies signing and all the vibers fist pumping.

We have heard all of this before, including all the Obvious Nonsense around safety, saying that open models are ‘safer’ for reasons that apply to true open software that gets many eyeballs, but mostly don’t apply to open weight models, while ignoring the ways it is more dangerous.

Zero perplexity, zero updating, much aura farming that costs nothing. But here we are.

A list of initial signers is below, zero surprises found:

Others including Musk and Altman then expressed support. Cheap talk is cheap, farm those auras and keep the vibers happy, boys. All the vibers were out vibing, thinking this list of standard platitudes mattered.

Ope