AI #170：大統領令の欠如

Last week ended on a cliffhanger of sorts. What’s in the Executive Order coming later today? What will be in the Magnifica Humanitas?

The Executive Order was postponed indefinitely, likely cancelled entirely except for work on securing critical infrastructure. David Sacks and others intervened to kill it, and American AI policy will continue to be maximally ad hoc.

Instead, we got Illinois SB 315, which is to be signed into law. It is a variation on California’s SB 53 and New York’s RAISE Act, while adding a third party auditing requirement.

The Magnifica Humanitas was revealed early in the week. I did an extensive readthrough, and have some follow-up commentary here to clear up some things I interpreted incorrectly and add richer context. It too ignores elephants in the room, not discussing AGI or existential risk and calling on people to ignore their incentives and instead do the right thing by prioritizing common good, especially access to good jobs and an end to war.

What I now believe I centrally misunderstood is that I interpreted this as a call to action, to use law and regulation to make this happen, because obviously that is the way you would actually engineer such outcomes. The Pope, I am told, instead really does think you can just ask the people to individually choose to not follow incentives. There is a section below with more on this.

A note on post frequency: For many months, I posted five times a week, and used any lulls as an opportunity to post non-AI things or split off subtopics. I have another project right now, and AI is in a relative news lull (I know, but yes, this is a lull), so instead I am using this as an opportunity to rest a bit. I will evaluate in July whether to return to a full all-weekdays post schedule.

Table of Contents

Language Models Offer Mundane Utility. Unleash auto mode.

Language Models Don’t Offer Mundane Utility. It is forbidden, thus not allowed.

Do The Math. Google proves nine more Erods problems, people react.

Huh, Upgrades. ChatGPT for PowerPoint.

On Your Marks. BenchBench, DeepSWE, we are too eval aware.

Get My Agent On The Line. Ask the AI to tell itself to do things.

Deepfaketown and Botpocalypse Soon. The rise of Pangram.

Copyright Confrontation. Individual researchers are now being sued.

Cyber Lack of Security. Progress in Project Glasswing.

Overcoming Bias. Models continue to be rather blatantly biased in some ways.

A Young Lady’s Illustrated Primer. AI in the classroom, and beyond it.

Unprompted Attention. Automate the automation of my work, please.

They Took Our Jobs. If they did it would be fine, right? We’d be rich.

Get Involved. How to contact me, how to comment, 80k hours has a new book.

Introducing. Qwen-3.7-Max.

In Other AI News. FAI launches a physical intelligence team.

Show Me the Money. Microsoft abandons Claude Code for Copilot. Yikes.

Show Me The Compute. Anthropic’s compute commitments over time.

Bubble, Bubble, Toil and Trouble. Bubble warners warn of bubbles.

People Just Say Things.

OpenAI PACs Just Say Things.

Quiet Speculations. Summer is when people convince themselves AI is stalling.

State AI Regulation Levels Up. Illinois passes SB 315, requiring model auditing.

The Quest for Sane Regulations. AI safety collaborations need legal safe harbor.

White House Attempts To Cripple American AI Industry. Driving away talent.

Our Offer Is Nothing. We tried it, and we’re all out of ideas.

Chip City. Huawei claims new chip scaling law. Okie dokie.

Greetings From The Department of War. Anthropic makes deal with the admin.

Marc Andreessen Just Says Things.

So Sayeth The Pope. A better understanding of where Leo is coming from.

Rhetorical Innovation. It is time to go there.

Aligning a Smarter Than Human Intelligence is Difficult. Escalating quickly.

People Are Worried About AI Killing Everyone. Thought on METR risk report.

Other People Are Not As Worried About AI Killing Everyone. The compound.

Everyone Is Confused About Consciousness. Caring helps you notice things.

The Lighter Side. Misaligned!

Language Models Offer Mundane Utility

Claude Code’s auto mode is highly recommended all around.

A lawyer who previously found models unusable for work due to hallucinations comes around for GPT-5.5 Pro.

Google claims its AI agents built an operating system with a single prompt and $916. Kapoor and Narayanan investigate, and find this basically press release puffery but with a grain of truth, even if the prompt was thousands of lines long and they haven’t released a bunch of other key data.

Robin Hanson on his use of current LLMs:

Robin Hanson: LLMs are now for me the sit-next-to-you-all-the-time-and-do-what-you-tell-them research assistants to professors that many imagine are typical, but few profs ever have. They can't be trusted on their own, but they offer decent first efforts & feedback.

AI can give you massive amounts of mediocre intelligence to do various tasks where you are bad or lack the time, to give you the chance to use your comparative advantage. Being mediocre at basically everything is a massive edge.

Judges are using an AI program called Learned Hand to distill legal motions, reach conclusions and draft tentative rulings.

Language Models Don’t Offer Mundane Utility

Anthropic continues to shut out users under 18 and says there needs to ‘be an adult in the room, a human in the loop’ and cites Haidt. I strongly disagree with this policy and would want my children freely using Claude at 13 at the latest, and this decision should at most by up to the parent by then. The good news is that it is not hard to get around such restrictions.

Berkeley law prohibits any use of AI whatsoever for any work submitted for credit, with the exception of identifying sources. Good luck with that.

Reminder: Don’t run your writing through an LLM unless you actively want AI slop. If you want help, have the LLM suggest individual changes. This includes writing issues for open source projects, the same as writing for humans.

Drew Breunig: We need a name for this, because Armin is putting his finger on a problem that’s everywhere: people running their writing through an LLM because they think it makes it clearer, when in actuality it sands off all the detail.

Armin Ronacher (talking about building Pi with Pi): That means the shape of the issue matters in a new way. A bad issue was always annoying, but at least a lot of issues were vague. Now we are also dealing with a class of issues that are 5% human and 95% clanker-generated and largely inaccurate shit. A bad issue that contains a plausible but wrong diagnosis

creates extra work.

The most frustrating failure mode right now is that people submit issues that

are not in their own voice. They contain an observed problem somewhere, but it

has been thrown into a clanker and the clanker reworded it and made a huge mess

of it. Typically, it was prompted so badly that the conclusions produced are

more often than not inaccurate but always full of confidence. The result is

complete guesswork on root causes, fake-minimal repros, suggested implementation strategies, analogies to adjacent but often the wrong code, and long lists of error classes that might or might not matter.

That is worse than no diagnosis.

Armin tries to tell their own AI to ignore such analysis, but it doesn’t work.

Do The Math

Google DeepMind’s AlphaProof Nexus solves 9 open Erdos problems, 44 OEIS problems, and more.

Various mathematicians respond to OpenAI’s disproof of the Unit Distance Conjecture.

Anthropic’s Levent is curious, so now we know that yes, Mythos can also solve the Unit Distance problem. Sometimes it lands on the same argument as OpenAI did, sometimes it finds a different one.

Gary Marcus does his best to be skeptical of OpenAI’s proof, without success.

What will mathematicians be for once AI is better in all areas of math? Daniel Litt suggests ‘locus of understanding’ which of course is another thing AI will be better at relatively soon. But we might insist a human do it anyway, as Daniel suggests, to try and help stave off human disempowerment.

Demis Hassabis cautions that no, solving Erdos problems is not ‘true invention’ and we are nowhere near AGI as he sees it. Contra Noah Smith, AI doing things humans cannot does not mean it can do the things that humans can, or that it is AGI. But yes, they are solving Erdos problems.

MTS: SITUATION DETECTED: Google DeepMind’s AI agent autonomously solved 9 of 353 open Erdos problems in mathematics, at a cost of a few hundred dollars per problem.

Robin Hanson: So what is the average social value (in $) per Erdos problem solved?

This is of course a Wrong Question, as the Erdos problems are proof of ability to solve problems and indication of future ability, not primarily mundane utility. But I asked Claude and it said median of $10k-$100k but with a long tail of $1m-$50m, with average of something like $1m-$10m.

Basically we got the first math problem that is exciting in itself rather than simply as an indicator, so of course the response is ‘well, sure, it’s fun and exciting but how much money did it just make?’

Niq Aily: the top 10 Erdős-inspired tools (probabilistic method, regularity, etc.) have unlocked on the order of $20–50B of cumulative economic value over 70 years through algorithms, networks, crypto, ML

The world is indeed this asleep.

AI Notkilleveryoneism Memes: I'm old enough to remember when everyone thought AI solving ONE novel math problem would be a front page story around the world

Today, AI solved not one, but NINE open problems - some 50 years old. AND proved *44* out of 492 open OEIS conjectures. Zero media coverage.

Nate Soares (MIRI): In sci-fi books and movies, AI solving a bunch of math problems that stood open for decades would've been a big deal. Why isn't the mainstream media turning this into a bunch of sensational stories?

(It's because humanity is sleepwalking into the creation of machine superintelligence. Once you realize that, it's easier to realize that we're also sleepwalking into a whole lot of danger.)

Huh, Upgrades

ChatGPT for Powerpoint. Google Slides instead, please.

Codex updates:

Codex can use apps on your Mac via your phone, even when it is locked and the screen is off, if you enable this in advance.

Use Appshots from your screen.

Goal mode now available in Codex app, IDE extension and CLI.

‘Advanced annotation mode’ lets you edit web pages directly via feedback + GUI.

Antigravity makes it easier to connect to the IDE.

Claude Code gets a new plugin to identify and fix vulnerabilities as you run code.

MiMo-v2.5 API pricing now permanently reduced ‘by up to 99%,’ more typically 5-8x. Lower tier models are often remarkably cheap, if your use case can handle it.

On Your Marks

Rohit gives us BenchBench, a test asking AIs to construct new benchmarks. Only one out of six models (GPT-5.2) created a benchmark that was neither too hard, too easy or too puzzle-like or checkbox-like or brittle. There’s a lot of ways to fail at benchmarks. This has to be done manually, so they only got a few shots each.

An obvious follow-up would be to offer feedback. Opus repeatedly created benchmarks that were good but too easy. That seems fixable.

Somehow I just figured out that SecureBio, which does evals on biological capability, has its own blog. Here is the assessment of GPT-5.5.

DeepSWE is a new agentic coding benchmark. GPT-5.5 leads at 70%, GPT-5.4 at 56% and Opus 4.7 at 54% are the only other models above 32%.

Apollo Research warns that models will soon be too eval aware for black box evaluations of alignment, so we will need white box evaluations instead. They want to do things like have steerable evaluation-awareness endpoints. This is an unsurprising reaction, but it is hard to express how much despair this plan and attitude invokes.

Get My Agent On The Line

The art of going meta by proxy, no Taelin is not the first to suggest this.

Taelin: I discovered a new joy in life. Don't ask Codex to do stuff. Ask Codex to ask Codex to do stuff. Rejoice as you watch it handling and correcting all the dumb shit that it does and that you'd be dealing with otherwise

"agent 3 reported a huge breakthrough, but upon closer inspection its code was just hardcoding the solution"

SURE IT WAS. AND IT IS YOUR PROBLEM NOW

Anthropic talks about how they build containment for their agentic AI systems, comparing the probability times blast radius to the reward. This is going to go great.

Deepfaketown and Botpocalypse Soon

Pangram Labs: Btw if you @ the Pangram bot and it likes your Tweet but doesn't reply, it means we've been blocked. So the answer is probably AI

Pangram Labs shows that last year’s Commonwealth Short Story Prize was also won by an AI-written story, and that 3 of the 5 entries this year were AI. Notice that this also shows a robustness, there are no false positives at even 1% level before 2025.

Tenobrus: people have lamented ai's lack of progress in non-verifiable fields, literature, etc. but it seems like there in fact must have been some significant progress, since blatantly ai-written stories are winning major awards!

I do think this is a strong benchmark. The prize is a worthless joke in terms of identifying literature you or I would want to read, or that the world should remember, but it is clearly measuring something, and winning is a hard and anti-inductive problem. If the AIs are willing, that’s impressive.

In other ‘you deserve it’ news, the author of “Future of Truth” included more than half a dozen fake or misattributed quotes because he trusted ChatGPT, oh well:

Will Oremus: I talked to the author of the "Future of Truth" book that turned out to have AI hallucinations. He told me he feels "seduced and betrayed" by ChatGPT, at one point suggesting it might have undermined him on purpose.

Nate Silver: How do you not double-check information provided by AI in a book about AI called The Future of Truth?!? There's no deeper story here except that Rosenbaum is lazy af and/or hired a ghostwriter.

The ghostwriter thing is way too common, largely tolerated in the industry when it shouldn't be, needs much better disclosure at a bare minimum. I'm old fashioned and think if your name is on the cover, the thesis and the writing should be truly yours.

Could ChatGPT have undermined him on purpose? Unlikely, but I wouldn’t rule it out

Ed Newton-Rex: Strong signs that >50% of Clive Lewis' Guardian article today was written by AI. That's according to Pangram, which has a very low false positive rate (1 in 10,000).

This would contravene the @guardian 's policies if so. Would be good to get clarity on this.

I mean, clicking back leads to this Tweet, and come on that is so obviously not a human voice.

Jack: tragic to see all of these politicians and celebrities using ChatGPT instead of real artisanal ghostwriters

one perk of the new age is the chance to remember just how many of these guys were never anything more than skinsuits over others' words in the first place

Emile Kroeger - arc: Wouldn't be surprised if they were hiring ghostwriters like they used to, except that now the ghostwriters are using ChatGPT.

The thing about AI-generated content is that we are now often at a point where it would have been in many ways fine content if a human had written it in 2022, but now it causes eye glazing among those who are used to seeing it and recognize it instantly.

Paul Graham: A lot of the emails I get from founders are now written in a hard-hitting journalistic style. I know they're written by AI, because no founder ever wrote this way before. And once you realize something is written by AI, it's hard not to ignore it.

I have never knowingly finished reading an email signed by a human but written by AI. It feels like being lied to, and who would stand for that?

It makes me think less of the author. It means they can't write well unaided (or feel they can't), and that they're trying to trick me. It's not impressive to use AI to write stuff for you; any teenager can do that.

I don’t see why it has to be a trick, or why anyone needs to be trying to impress. Perhaps the human simply wants to aid in their communication. That doesn’t make this a good idea, and I am in a similar place to Paul Graham. If you send me a cold email, I may read it if it is relevant to my interests, but if you had an AI write it then I will delete the moment I notice.

It’s worth noting that this is actually not so difficult in terms of direct writing:

Sriram Krishnan: the real challenge on the Internet these days is writing a long public document and proving that you didn’t use AI for it.

There is a broad ‘middle range’ where you cannot be sure, but good human writing is very obviously human. There is an aliveness, a perplexity, a willingness to ‘go there’ that AI simply will not produce. Often I am very confident something is not written by AI, the same way one is often confident on reflection that one is not sleeping.

What you cannot possibly do is prove you did not use AI in other ways, such as to ask questions, check facts, search the web or finding errors. There’s flat out nothing you can do, because the writing looks exactly the same, and it is exactly the same.

It’s not AI, but it is kind of still AI slop:

RahScene: Im half way through Daniel Yergin’s prize and its crazy how many of the current ai tics are in his work. If i didnt know when it was written i would have been like hold up.

Joe Weisenthal: This is the key thing. Up until very recently, nobody had any issue with these particular writing tics, and good writers employed them successfully.

Byrne Hobart: I feel this way about waitstaff at restaurants drawing a smiley face on the receipt. It was great the first few times, and then I learned that this predictably leads to higher tips and they get coached to do it.

Whereas if it is AI but not slop, it will ‘fool’ Pangram.

alice: NO WAY talkie fools pangram lol

j⧉nus: Pretty much any interesting ai output fools pangram

toni: and so does opus 4.7’s rendering of 'recueillement'

Sauers: it's sort of unintentionally based in this regard. a generic slop detector, not detecting humans or AIs, would be good.

I am totally fine with Pangram being a detector for ‘standard AI writing’ and for nonstandard AI writing to get through.

This is also the right mistake to be making. What is great about Pangram is the very low false positive rate. If the test has a substantial false positive rate, you basically can’t use it, as people can say it got this one wrong, and you don’t want to risk making a mistake.

Whereas if you have a substantial false negative rate, especially if it is concentrated in ‘better’ uses of AI, then that is annoying but essentially fine.

We will be putting that to the test.

Jack: before AI writing slop there was business school slop, and most companies were perpetually in that mode. probably not a coincidence that the CEO of the most reliable anti-slop tool acts and talks like a human

Daniel: a CEO showed up to challenge a critic to a duel and the critic accepted and now we await the results

Ultimately, what are you looking for? Why do you care if an AI wrote something?

One good reason is to call out people for wasting our time and polluting the commons.

Jack: I run Pangram when I find a piece of writing that wastes the time and attention of the audience and is ob