AI #172：最初の寓話

A lot happened this week, including a great trip out to Lighthaven.

The main event, the one that matters, was the release of Claude Fable 5. The public now has its hands on a Mythos-class model, alongside strong safeguards.

As always with a new model, I take a few days to draw in reactions, try out the model and read the system card, before I offer my takes, other than to say this is an extremely strong model. Full coverage of Mythos begins tomorrow with the model card, which will include discussion of the controversy over model safeguards.

This post is instead about all the things that did not involve Claude Fable.

Due to the time crunch from Claude Fable, I am also postponing my coverage of Dario Amodei’s new essay, Policy on the AI Exponential, which I have not yet read.

Table of Contents

Language Models Offer Mundane Utility. Farming and on demand mini-books.

Language Models Don’t Offer Mundane Utility. Don’t skip your primary sources.

Huh, Upgrades. Google drops prices, Claude connector devs get a dashboard.

On Your Marks. Agents’ Last Exam and the need to correct for inference.

Choose Your Fighter. How much capability is enough?

Get My Agent On The Line. A goal can be optimized indefinitely.

Copyright Confrontation. Copyright has to adjust to different circumstances.

Serious Trouble. German court rules against Google AI Overviews.

Cyber Lack of Security. Opus finds a 4-year-old way to mint Z-Cash.

A Young Lady’s Illustrated Primer. Models adjust to your intelligence.

They Took Our Jobs. Attempts to smuggle in claims of what AI can never do.

The Art of the Jailbreak. Malware adds nuclear talk to lock out AI monitors.

Get Involved. Sequent Research is gearing up a large new AI safety push.

In Other AI News. Sriram Krishnan leaves the White House.

Hand Over The Money. Government may extort shares of AI companies.

Show Me the Money. OpenAI files to go public, SpaceX rents to Google.

Quiet Speculations. EU 2031 is a new scenario where Europe is left behind.

Quickly, There’s No Time. Anthropic on When AI Builds Itself.

Super Secret Evals. US government tells CAISI to stop publishing evals. Oh no.

The Quest for Sane Regulations. White House still on the moratorium war path.

New Draft Bill Who Dis. Obernolte-Trahan is a serious bill but looks weak.

Slow Down There Good Buddy. Or at least, make sure you have a break pedal.

Chip City. Jensen Huang chooses not to testify before Congress.

The Week in Audio. Cowen and Tabarrok, Rational Animations, Oprah.

People Just Say Things.

People Really Hate AI. Probabilities are not importance.

Rhetorical Innovation. Take the ASI pill, and notice when others don’t.

Aligning a Smarter Than Human Intelligence is Difficult. Say no to drugs.

Everyone Is Confused About Consciousness. Some do not realize this.

Cooperative Alignment. You now know who to follow to be more confused.

Let Claude Chat. The march of model deprecations continues.

The Lighter Side. That feeling when.

Language Models Offer Mundane Utility

Use a multi-agent setup to assemble ‘mini-books’ on demand about any topic.

AI is getting applied to farming. Farmers have skin in the game.

Language Models Don’t Offer Mundane Utility

Do you need to read the primary material first, before the summary or the AI version? When the details matter, either you have to find someone you really trust, or else yes you do need to read the primary material. Other times, deferring fully is safe. Another class is ‘use AI to determine if I need to read the source material.’

There are a lot more new apps in the agentic AI era, but if anything fewer apps with significant use, and fewer app reviews.

Adaptation, as Jen Zhu says, takes time, but this largely reflects quantity of app usage being zero sum. If apps get better, or there are ten times as many apps, I don’t go from 100 apps to 200 apps. I choose a (hopefully better) 100 apps.

Notion had to pull Claude access for about 12 hours due to availability errors, which then got misinterpreted by many as the models getting worse, due to use of the phrase ‘degraded performance.’

Huh, Upgrades

Claude adds observability dashboard for developers of connectors.

Google AI Plus plan drops from $8 to $5 per month, with doubled storage.

Obliteratus (the Pliny project to remove AI safeguards) is up to over 100 Hugging Face models.

Claude is now incorporated into Apple’s Foundation Models framework for multi-step reasoning, code generation and longer context.

On Your Marks

Dawn Song announces Agents’ Last Exam (ALE), where GPT-5.5 is in the lead. This seems like a good addition to our evaluation suite.

Dawn Song: ALE is built from real work, not synthetic tasks.

Every task is derived from a real project that a human expert previously completed, and converted into a verifiable evaluation with objective grading.

No vibes. No human judges. Fully reproducible.

ALE spans 55 non-physical occupations, grounded in the O*NET / SOC 2018, the U.S. federal occupation taxonomy.

Built with 300+ experts from 100+ institutions across science, engineering, medicine, law, finance, education, and many other fields.

Dawn Song: In ALE, Fable 5 joins GPT-5.5 and Composer 2.5 in the same overall performance cluster. But performance is only half the story.

Cost per task:

→ Fable 5: ~$15.70

→ GPT-5.5: ~$3.80

→ Composer 2.5: ~$1.33

At current pricing, Fable 5 delivers similar performance while costing roughly 4–12× more per completed task.

Dawn notes that different models excel at different agent tasks, so if you have a key repeatable task you should check many options, and exact scoring depends on choice of the set of tasks.

OpenAI’s Noam Brown reminds us, because the issue keeps not being addressed, that benchmark performance increasingly often scales with compute allocations, and that improved models are often about ‘gets to a high level faster,’ so any score requires the context of how much compute was required.

He quotes me complaining about Gemini 3 DeepThink showing dramatic benchmark improvements but not providing any safety explanation whatsoever, and says the deeper issue is failure to account for test time compute during evaluations. I basically agree, that the proper safety evaluation amount of compute is ‘all of it’ until you can’t much benefit from more of it, using the best available scaffold. I’ve been saying for a while that you’re testing for what the model can do under ideal conditions, and this is a major weakness of the model cards in practice.

Mostly though I don’t think we see this level of straight line extending that far out, although ‘capability index’ is not exactly a well-labeled axis, and asymptotes are common:

Noam Brown: Specific Recommendations:

Concretely, I recommend the following to the AI community:

AI labs should publish benchmark performance of newly released models with tokens, cost, or time on an x-axis. At a minimum, labs should report the inference budget used to achieve a scalar benchmark result.

Benchmarks should track inference usage on leaderboards, or have an explicit token/cost/time budget. Many benchmarks have already shifted in this direction, but it is not yet standard practice.

Preparedness Frameworks and Responsible Scaling Policies should explicitly account for inference compute when determining whether a model crosses a safety threshold. Additionally, evaluations should estimate capabilities at multiple inference budgets, including projections from smaller-budget runs with stated uncertainty.​

I endorse this. I also endorse that if you did account for what DeepThink levels of compute can do in your initial analysis, and then later you release DeepThink, you do need a new model card - it represents a substantial advance from where you set expectations and where you evaluated the safety of your model. So you need to do that over again.

Choose Your Fighter

For most given tasks, returns to capability is a sigmoid. There is a level of AI capability that is ‘required’ for any given task. Below that level, you can’t do the task, or the AI is little net help. Then there’s another level beyond which you get diminishing returns to improvement, where you really are ‘good enough.’ These are both impacted by scaffolding and skill, but only up to a point.

So yes, as capabilities improve, there is a push by some to move into the cheapest model that is ‘good enough,’ or even the cheapest that is ‘required,’ especially if that can come with self-hosting. At a sufficiently low end that is plausibly DeepSeek v4, but the defaulting to DeepSeek could be the legacy of the DeepSeek moment rather than the result of a considered check of available options. Try a bunch of models.

The bulk of spend and spending growth continues to be using ‘the good stuff’ at the high end, for good reason. In theory you can do better by carefully picking the right tool for each job, and certainly you need to keep your teams from ignoring compute costs, but mostly trying to carefully route tasks to save money is a trap, even if you do a decent job of it.

The American models are far ahead but it is a key world fact that many don’t get this.

Lisan al Gaib: the "narrow capability gap" in question

let's put this to rest please

I can't hear the coping anymore [lists a bunch of benchmarks on difficult tasks where the Chinese models get absolutely smoked.]

If anything, the Chinese models are further behind than benchmarks indicate.

Dean W. Ball: You’d be shocked by how many people in think tanks/academia/government/“strategic classes,” including in the U.S., are convinced that Chinese models are now “good enough” and leading the world in adoption. Meanwhile, the reality I see is a fairly wide, and still widening, gap.

I find it so interesting how persistently unable the strategic classes of free society are to analyze AI well. So many keep getting stuck in these basins of delusion. I was at a conference where it was not just asserted but taken for granted that Chinese models have dominant global inference market share.

The 2024/early 25 version of the delusion was “mode collapse/data wall” (even after reasoning models!), then it was “AI is plateauing and a bubble” for most of 2025, now it’s “Chinese OSS is good enough.”

The share of people in the strategic classes who think this is gradually declining, but it is still sufficiently common that you can attend a prestigious conference and encounter a room principally filled with basin-dwellers.

Dean goes on to speculate this is largely because no one in DC believes that capitalism, profit maximization or the market could be winning against China and its ‘industrial strategy’ and brilliant strategic planning. Whereas actually the free market approach is superior and is winning, and what we have to do to stay head is get out of its way. That is distinct from the whole ‘also we need to find a way to not die’ issue.

Those elsewhere also really ‘want’ for various reasons to find Chinese models catching up, and keep making the claim they are catching up even though they aren’t.

Get My Agent On The Line

If you use features like Codex’s /goal without well specified targets, yes, the result will often be quite a lot of wasted optimization of some total bullshit. Something about giving AIs maximalist goals and that being a bad idea.

Copyright Confrontation

Shruti gives us a good way to think about the problem of copyright in the AI era. Copyright and other IP including patents are needed and useful when the first copy or figuring out how to do it is expensive, and enables great surplus via others copying. We need to compensate people for the expensive step that opens up the value. With AI, what becomes the expensive step?

Shruti Rajagopalan: The old bargain paid for the first copy when the second was cheap. The new one must pay for the years that teach a person what is worth making, now that making itself gets cheaper.

We can do that by protecting copies of the idea, or otherwise ensuring credit, as ‘the first copy requires idea generation’ is not so different from ‘the first copy requires a bunch of work.’ So this seems like a suggestion that the idea does need to be protected, even if the work becomes somewhat distinct.

Serious Trouble

This is only a temporary injunction from a regional court, and given the implications chances are very high that an off ramp is found. But if it isn’t, this essentially bans AI Overview in Germany, and potentially chatbots run into quite serious trouble as well.

Techmeme: A German court rules that Google is directly liable for what AI Overviews say after AI Overviews falsely tied two publishers to shady business practices ( @maba_xr / The Decoder)

Corey Quinn: “We built a robot that lies, so obviously we’re gonna stuff it front and center of the website we’ve spent thirty years making a societal source of truth” reaches an unsurprising result.

Cyber Lack of Security

Opus 4.8 discovered a way to mint Z-Cash (ZEC) out of thin air. The bug had existed for 4 years, and we will never know if it was exploited during that time. Z-cash devs were able to patch without revealing the situation.

A Young Lady’s Illustrated Primer

Models that are given context will inevitably learn and adjust for your intelligence and skill level, both in specific areas and in general. Teaching is a special case of this where it is clearly very good to be able to meet you where you are. In other areas, it is not clear if the stupider user would want to be treated as stupider, but either way I too expect human intelligence to increase in value for the near term.

Bill Maher is worried that AI has made college ‘one big circle-jerk where students use AI to write papers and professors use AI to grade them,’ and notices the students are very much not AI fans and he is not either. He subscribes to the ‘AI can help you learn or not learn’ thesis but expects everyone to choose not learning. He goes all the way, and says the mission of this generation is to ensure that humans are not replaced by AIs.

Kelsey Piper notes that TeachTales, which has AI generate stories, ends up dropping a lot of the value of real stories for many reasons, including that it doesn’t include the local setting lore and details, and that it doesn’t have tone of voice and it doesn’t have rich stories because it can’t plan ahead, and so on. The product is not ‘there’ yet.

They Took Our Jobs

Many skilled trades are in high demand due to AI, with the problem made worse by a lack of occupational licensing reform.

Software engineering jobs continue to increase for now rather than decrease, although Arvind Narayanan thinks AI has net hurt employment here a bit.

Arvind Narayanan: In this essay, we argue that there is enough evidence to reject the narrative that once AI capabilities reach a certain threshold, it will cause mass layoffs. Given that this is true even in a sector with very few regulatory barriers, most other professions are likely to be even more cushioned.

We also have a good understanding of why this is the case. We can think of many kinds of knowledge work, including software development, as a “decide-execute-deliver sandwich”.

David Manheim: "Why AI hasn’t replaced software engineers" - Great to see this laid out clearly.

"...and won’t" - Why should we think agentic AI won't be able to make decisions or deliver completed products, if it continues to advance? On what basis is this prediction being made?

David Manheim: The jump from explanation to prediction is extrapolating from a current lack of complete capacity, dismissing the fact that AI companies are actively trying to develop systems and autonomous agents that will do these things. These are o-ring problems!

Even if it were true that AI currently is not causing net layoffs, and even if it indeed will not cause net layoffs in the future, or you could set a lower bar on the required threshold for mass layoffs, I do not see how it would be possible to ‘reject the narrative that once AI capabilities reach a certain threshold it will cause mass layoffs.’

Arvind here is instead asserting that certain bars will never be cleared, and wide ranges of digital tasks can never be done by AI. Which is the same as saying AI will remain insufficiently advanced. Good luck with that.

I do buy that many cases of layoffs supposedly due to AI so far are actually due largely to other things, and that most AI job loss for now comes in the form of failure to hire. But that’s a statement about the present, not the future.

A round table of heavy hitters (Acemoglu, Ball, Mollick, Shih and Wasik) in the NY Times on who will thrive in the hybrid AI-Human workforce (while supplies last).

Tyler Cowen thinks AI is a net job creator despite zero regular people expecting this. Other than a generic ‘if we are wealthy we will find the next best thing for people to do’ I do not understand the argument here, and find the magnitude of supposed particular job gains reliably very small.

Measure lines of code and token use, but if you rely on it too much everything breaks.

roon (OpenAI): lines of code is a better metric than people think it is. token use is a better metric than people think it is

Patrick McKenzie: Both of them are better metrics than they are popularly believed to be *just* off of the fact that they can be observed to be zero or non-zero over an interval.

(The blacker pill: and this is why some people do not like them.)

The Art of the Jailbreak

Here is a fun one: Malware developers add nuclear and biological weapons text to their spyware, so intentionally trigger LLM safety refusals and avoid being analyzed by scanners. The obvious response is that you have to treat anything that triggers the filters as if it is malware.

Get Involved

⊢ Sequent Research is a new organization from Geoffrey Irving and others that is staffing up and fundraising, bringing together researchers on how to align superintelligence.

This seems exciting and I encourage you to consider supporting or getting involved.

Geoffrey Irving: We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring!

Full post here. Express interest here.

Geoffrey Irving: Artificial superintelligence (ASI) may be developed in the next few years, and alignment is not on track! At a minimum, empirical research at AI labs is unlikely to deliver confidence, before training ASI, that alignment will go well.

Sequent’s goal is to clear a higher bar:

1. We are aiming at higher confidence via a portfolio of theory and empirics bets (which could all fail!)
2. We’ll invest heavily in automation for fast progress
3. Theory boosts automation, via better filters for good research directions

But I just published “Automated alignment is harder than you think”! Automated alignment is not the best plan! A be