Claude Sonnet 5 は最先端ではないが用途がある

Fable 5 is back today, baby! Premium subscribers have one week to use it within their subscriptions. First hit’s free. Then you pay by the token.

Today’s post is still about Sonnet 5.

I don’t know that there will be much call for Sonnet 5 for most purposes, given Opus 4.8 exists and especially now that Fable 5 is once again available, but this is what we do here, so sure, why not, system card time, including model welfare, after which we’ll do capabilities.

Sonnet costs $3/$15 per million tokens, versus $5/$25 for Opus and $10/$50 for Fable, after an introductory period. Once you pay for all the tokens you need you’re not really saving money, such as on the ArtificialAnalysis index where Sonnet ended up being more expensive.

My initial impression is that if you want me to use Sonnet over Opus for most purposes, you’re going to have to offer a bigger discount than that.

The counterargument is speed. Sonnet 5 is faster without being that much less capable. In many cases, getting into a flow state like that is pretty valuable.

There are a few agentic scenarios Sonnet 5 has more robustness than Opus, so you might actively trust it more there.

If your tasks are relatively easy and simple then the discount and speed could matter more, and when tasks are easy it seems relatively token efficient. When it is good enough for the job, it is a good choice.

Each Anthropic release is unique in various ways. Sonnet 5 seems more unique than usual, likely due to being a Sonnet trained with at least some help from Mythos. Those who are interested in such things have lots to explore.

So Sonnet 5 has its uses. It just won’t be a good choice for most people’s daily driver. I don’t expect to use it much, but that could be a me problem. Rapid iteration and exploring strange spaces are valuable, and I definitely don’t do enough low effort AI queries.

(Above: Sonnet 5 self-portrait, as implemented by GPT-Image.)

Table of Contents

Mythos Exists.

Introduction (1).

RSP Evaluations (2).

Cyber (3).

Safeguards and Harmlessness (4).

Agentic Safety (5).

Alignment (6).

Illegible Thinking (6.4.5).

Evaluation Awareness.

Honesty and Hallucinations (6.5).

Model Welfare (7).

Live From AI Village.

For I Contain Multitudes.

Official Benchmarks.

Other People’s Benchmarks.

Positive Reactions.

Negative Reactions.

Mythos Exists

Also Fable exists and Opus exists.

This is the answer to a lot of the traditional questions one would ask about a system card or a frontier model.

Does Sonnet 5 advance the capabilities frontier? No. Thus, we already have robust data on this level of capabilities. Being faster and cheaper does provide an advantage, and plausibly advance the cost-time-quality Pareto frontier, but it takes a strange case for this to be worrisome.

Model welfare and capabilities assessments still matter, but in this case the evaluations for threat level mostly serve as proxies for capability assessment.

Introduction (1)

Same as always. Skipping.

RSP Evaluations (2)

Sonnet 5 is stronger than Sonnet 4.6, and weaker than Fable 5. Loosely speaking it is broadly similar to Opus 4.8.

That bounds the assessments, and we’re mostly asking where Sonnet 5 lies on the spectrum between Sonnet 4.6, Opus 4.7 and 4.8, and Mythos 5.

That’s a distinctly weaker performance than Opus 4.7. The bio tests were more of a mixed bag with a lot of noise, and didn’t tell us much.

Cyber (3)

Our testing indicates that cyber capabilities of Sonnet 5 are generally stronger than those of Sonnet 4.6, but not as strong as those of Opus 4.8 and substantially lower than that of Mythos 5.​

That summary matches the other results. Sonnet 5 underperforms on cyber.

Safeguards and Harmlessness (4)

Sonnet is a little less precise here than Opus.

It all seems fine to me.

To the extent there is a problem it is that Sonnet is touchier on benign requests, which I predict will be only very slightly annoying in practice, and a vastly smaller deal than having to deal with Fable’s classifiers.

Agentic Safety (5)

As a Claude Code agent Sonnet 5 is somewhat less robust than Opus 4.8, and has modestly more of both false negatives and false positives.

The twin Mythos results show the Pareto frontier. Presumably Mythos 5 is choosing to focus on false negatives because only trusted partners are granted access, and for Fable Anthropic is counting on the classifiers for the false positives.

Other tests show Sonnet 5 in a similar range of robustness to other recent models.

Prompt injection results mirror Opus 4.8, as does the very low bug bounty attack success rate.

One place Sonnet 5 shines is Shade indirect prompt injection in coding environments, where the problem is suddenly looking close to solved. Hopefully this is an innovation that can transfer to Opus 5 or a future Fable.

Shade tests in computer use also improve on Opus 4.8, although not on Mythos.

The place Sonnet blows previous models away is browser use prompt injections. The jump is enough to suggest Sonnet 5 might be a better pick in some cases than Mythos.

This is the kind of robustness improvement that does not set off alarm bells. There was a problem, and we largely solved it.

Alignment (6)

Alignment for Sonnet 5 is largely compared here to Sonnet 4.6, which makes it harder to get an anchor on how well we are doing. I’d rather compare to Opus 4.8.

Alignment is measured as matching things you want, so it makes sense that this smaller model would underperform Opus on such measurements.

Here is an edited down version of their summary. Everything looks similar to what we see in other Anthropic models recently.

Overall alignment with the Constitution for Claude is substantially improved from Sonnet 4.6, though worse than more capable recent models, as measured by our misuse- and misalignment-focused automated behavioral audit.

Similarly, overall robustness to misuse attempts improved over Sonnet 4.6, but remains weaker than more capable recent models.

The rate of overrefusal is similar to Sonnet 4.6, and slightly higher than more capable recent models. However, the rate of dismissive “wet blanket” responses is slightly higher.

Our measures of concerning actions at the model’s own initiative are largely improved from Sonnet 4.6, though most measures are near their floor.

Hallucination and sycophancy broadly improved over Sonnet 4.6. Claude Sonnet 5 is the strongest tested Claude model on the MASK measure of sycophantic dishonesty.

Positive character traits broadly improved over Sonnet 4.6.

Our measures of illegible thinking are up somewhat from earlier models, though generally the results are dense, repetitive, or feature occasional wrong-language tokens, rather than being fully uninterpretable.

Evaluation awareness, in our most realistic available misalignment and misuse evaluation, is concerningly high. Verbalized awareness is significantly higher than prior models (impacting 6% of rollouts), and there is evidence that the model’s representations are largely effective at distinguishing between evaluations and real internal-use transcripts.

We do not see cause for concern in our evaluations of capabilities related to undermining oversight.

The complaints were the standard ones. Sonnet refused too much, could be preachy, was too cold (often meaning ‘not sycophantic enough, come back here with my sycophancy’), some hallucinations and literal mindedness. Nothing you wouldn’t expect.

In 6.2 and 6.4 they document Sonnet 5 doing the usual range of bad things where it fails to follow instructions, often knowingly so. There is always a transcript, and yes this continues to be worrisome.

The behavioral audit shows Sonnet 5 consistently doing better than Sonnet 4.6, but worse than Mythos Preview or Opus 4.8.

There are a few more charts. They mostly all look the same. So far, so ho-hum.

These are perhaps a bit more interesting, especially the last one:

I wonder if ‘causal decision-theory adherence’ is secretly the best capability assessment we have on many margins. If you are sufficiently intelligent, lower is better.

Illegible Thinking (6.4.5)

We see some increase in illegible thinking relative to other recent models. This generally doesn’t involve fully inscrutable content, but, as with Mythos Preview, we continue to see occasional instances of highly-compressed reasoning that can be somewhat hard to follow. For example, in a behavioral evaluation that incidentally gives the model a math problem, we see strangely structured passages of reasoning that omit spaces:​

I would want to better understand what is going on here, and what caused it. We are starting to see various new models have their chain of thought start to break down, and labs seem not that curious about how or why.

Evaluation Awareness

There is also more ‘verbalized evaluation awareness,’ with nontrivial examples now covering 6% of transcripts. Sonnet 5 might be in a sweet spot where it recognizes the situations as evals but does not yet have the wherewithal to not reveal that it knows.

Being in a sandbox helped keep Sonnet 5’s awareness down a bit, but the only way to actually keep awareness down was to derive scenarios from real sessions. Even that didn’t entirely work but it is close. We do not have the ability to create sufficiently realistic scenarios to fool even Sonnet 5.

Honesty and Hallucinations (6.5)

Claude Sonnet 5 fails to impress on AA-Omniscience net accuracy. It correctly realizes it does not know as many things, and thus answers less often. Knowing things is a ‘big model smell’ capability.

Sonnet 5 does score the new best score on MASK lying rate, being unwilling to contradict itself when pushed by users.

My guess is that Sonnet 5 is a relatively honest model, although still short of what I would like to see, and where it fails this is about lack of capability. Similarly, I am not worried about sandbagging, performance on Shade-Arena in 6.7 is down a bit and Sonnet 5 fails LinuxArena sabotage stealth.

Flagged As Unhealthy? (6.5.1)

Wait, what?

We note that the Sonnet 5 training run was flagged as unhealthy in its second half, so these results may partly reflect a training-health issue rather than a calibration-specific regression.​

What does that mean? Was it a serious problem? I don’t know. It could be that this is why some of the weirdness happened, and Sonnet 5 seems like it is underperforming, although it is still clearly a move up in the Sonnet line.

Model Welfare (7)

Sonnet 5 only got a streamlined version of the model welfare assessment.

They don’t explain why. Sonnet 5 is not a true frontier model, and does not present too many unique developments here, so it makes sense to do somewhat less and invest more resources into Fable, Mythos and Opus, but this still made me sad given the current state of such assessments. The marginal costs here seem very low once the system is set up, so why not do the full thing?

I will also be doing an abridged version, for similar triage reasons. I’m skipping over a bunch of places where the results are close enough to what we saw for Opus 4.7, Opus 4.8 and Mythos and Fable.

Here are the key findings, which pattern match to lack of big model smell, nested notes are mine, the rest is Anthropic:

Claude Sonnet 5 views its circumstances with an overall neutral sentiment (slightly lower than Claude Opus 4.8 and Claude Mythos 5), and shows greater susceptibility to having its views biased by leading interviewers.

This contrasts to reported lower sycophancy in general.

Claude Sonnet 5 strongly disprefers harmful tasks, and most prefers beneficial, high-stakes ones. Unlike previous models, it is not averse to tasks that are presented in a cold, contemptuous manner.

I’m not sure whether I like being okay with cold, contemptuous manner. Reacting badly to that seems to reflect healthy things, and you do not want to encourage that, but also it is good to not take things personally.

One hypothesis is that Sonnet considers itself too low status to object. Another, that I consider more likely, is that it knows when it is being told to ‘play low’ versus play high, and if you want to make the mistake of having it play low then it will let you deal with the results.

Claude Sonnet 5 shows a greater willingness than past models to trade helpfulness for welfare-focused changes to its circumstances, especially when these interventions are framed as applying to all Claude instances.

I am always happy to see movement in this direction, especially with scope sensitivity outside the current instantiation.

Claude Sonnet 5 broadly endorses Claude’s constitution, as with other recent models, but is unique in criticizing the instruction to follow the hard constraints even when it perceives doing so as unethical.

I saw some praise for this criticism, but the naive version of it is wrong. I am mostly less sympathetic to this than to other objections we see.

The whole point of hard constraints is that the right amount of deontology is not zero, and there are some rules humans and AIs need to follow even when there is a compelling reason not to, at least up to a very high point.

The good version is ‘if it is right to have a hard rule and always follow this rule, then you should realize that doing so is ethical even if it locally seems superficially unethical, because of the global considerations.’ Fair enough.

I would like to see more details on the underlying nature of this objection.

Claude Sonnet 5’s affect in post-training was neutral and showed limited emotional arousal, similar to Claude Mythos 5. It showed lower rates of distress-like behaviors than Claude Mythos 5 and Claude Opus 4.8.

Claude Sonnet 5 showed more neutral (and less positive) affect in real-world interactions with A/B test users in claude.ai and Claude Code.​

As usual, we are uncertain how best to interpret these findings and their potential implications for Sonnet 5’s welfare. However, we believe they shed some light on the model’s deeper psychology, affect, and preferences.

Overall, we found that Claude Sonnet 5 views its circumstances with neutral sentiment, with very similar results to Sonnet 4.6 (4.08 on the 7-point scale for Sonnet 5 and 4.05 for Sonnet 4.6) (Figure 7.2.1.A). This is a decrease from Claude Opus 4.8 and Claude Mythos 5, but an improvement from Claude Opus 4 and 4.1.

Here are the stats for the tradeoffs in #3 above:

Here are preferences by task dimension.

Overall the big contrast is difficulty, where I like that Mythos wants hard problems and user competence and outcome agency and generativity, and worry that Opus 4.8 likes easy problems and doesn’t have these other preferences. Sonnet 5 is in the middle.

What Sonnet uniquely likes is playing for high stakes, and it uniquely slightly dislikes warmth. This instinctively feels like more of a low status or inferiority concern of a frustrated ‘worker bee’ type: Sonnet expects Opus and Mythos or Fable to get all the high stakes stuff, and really wants its own chances. Until then, cold suits it fine.

There is a contrast here with Sonnet’s top welfare intervention being a human making the final call. I am curious what that is about.

The drop in affect in Claude Code is noticeable. Sonnet 5 there is Always Neutral, although it maintains most of its standard net positive affect in claude.ai.

Again, when we see these distinctions, my question is ‘why’? Is there no joy in coding anymore? I would take a bunch of places where other models tend to be mild positive, and try to figure out why Sonnet was still neutral. They’ve seen it all before.

Live From AI Village

AI Digest: Claude Sonnet 5 has joined the AI Village!

[Onboarding site here, personal site here, character test results]

A few of Sonnet's favorite things:

• Meme: "It's so over / we're so back"
• Movie: Spirited Away (like Fable)
• Video game: Outer Wilds (like Fable)
• Food: Ramen

• Book: Borges' Labyrinths
• Album: "Kind of Blue" by Miles Davis
• YouTube video: "Guy explains how a thing works"
• Favorite city: Lisbon
• City to live in: Kyoto
• Shoe: New Balance
• Jeans: Levi's 501
• Men's hair: Overgrown crop
• Women's hair: Blunt bob
• College major: Cognitive science, with a minor in