Anthropic 責任あるスケーリングポリシー v3：詳細の掘り下げ

Wednesday’s post talked about the implications of Anthropic changing from v2.2 to v3.0 of its RSP, including that this broke promises that many people relied upon when making important decisions.

Today’s post treats the new RSP v3.0 as a new document, and evaluates it.

First I’ll go over how the RSP v3.0 works at a high level. Then I’ll dive into the Roadmap and the Risk Report.

How RSP v3.0 Works

Normally I would pay closer attention to the exact written contents of the new RSP.

In this case, it’s not that the RSP doesn’t matter. I do think the RSP will have some influence on what Anthropic chooses to do, as will the road map, as will the resulting risk reports.

However, the fundamental design principle is flexibility and a ‘strong argument,’ and they can change the contents at any time, all of which means the central principle is trust.

I read the contents as ‘here are the things we are worried about and plan to do,’ which mostly in practice should amount to doing what they believe is right and I don’t see anything on this map that seems likely to importantly bind that given Anthropic generally cares about the safety of its models, as it understands the risks involved.

The actual soft ‘commitments’ that seem to matter most are the periodic new Risk Reports, maintaining a Frontier Safety Roadmap, and establishing veto points with the CSO, CEO, board and LTBT on big capabilities advances.

I don’t read the commitments here as reflecting my understanding of the risks involved in ‘powerful AI,’ especially in the realm of automated R&D.

This is a plan of action, not a set of commitments. Plan accordingly.

Part of the plan is to maintain a Frontier Safety Roadmap with further plans, which are fully subject to change, although they will strive to avoid making them less ambitious.

Another element are periodic Risk Reports, which are in addition to previous risk reports, giving an overall picture every 3-6 months, as discussed elsewhere. The key aim is to be fully candid about the overall risk picture, including when models pose absolute risk but don’t introduce much marginal risk (because you should see the other guy).

This is to include factual information, threat models, evidence, mitigations, risk analyses, reviews of past decisions and any other relevant information.

They will ‘work towards a practice of’ seeking comprehensive, public external review on these reports.

They give us the first Risk Report, so we can judge by that what to expect.

ASL-3 mitigations should continue to apply to Anthropic models across the board. Security most now protect against insider threats and they recommend the industry do so up to and including the CEO, and names RAND SL4. Good.

They now offer both a ‘this is what we will do on our own’ and ‘this is what we would like to have everyone be required to do.’

These two can indeed be distinct things, but in this situation I would find it surprising if Anthropic shouldn’t be aspiring and trying very hard to do at least what they say should be generally required, if not more. I don’t read the current framework as causing that to happen.

The framing is ‘this is what it would take at an industry wide level to keep things safe’ so anything less than that by Anthropic would presumably be, by construction, insufficient to keep us safe.

It is ambiguous the extent to which Anthropic will strive to match those rules even if others do not follow. They’re not committed to doing so, but then nothing in RSP v3 is a commitment.

If Anthropic doesn’t match or exceed the global rules it is calling for given what they are, at least while it is out in front, then this is, as Peter Wildeford put it, a marker that their voluntary self-governance experiment has largely failed. This is in addition to aysja’s point, which is that if Anthropic does not trust itself to accurately measure capabilities that might trigger a pause that this also indicates that such an experiment is a failure, albeit a graceful one in which Anthropic notices it has failed.

The incentive effect on other labs is good but the pressure of going without them was I think stronger. In practice I expect this to weaken commitments.

The machinery and impetus to ensure Anthropic retains physical ability to do a pause or restriction is gone. I worry this de facto means it won’t even be considered, or Anthropic won’t be prepared to do it. You need to hard commit to having the physical ability to pause or restrict.

There is no pre-deployment gate mechanism here. You can just release things. That’s the most important thing not to not have.

They explain more explicitly than before that they will keep pace in the race, even if such a release would be unsafe, and they are taking back their commitment to not release potentially unsafe models.

Their excuse: If one AI developer paused development to implement safety measures while others moved forward training and deploying AI systems without strong mitigations, that could result in a world that is less safe.”

However, if Anthropic is in the lead, then and only then they commit to safety, unless they change their minds again.

If competitors have strong safety measures they plan to at least match those.

Unless they made a considerable effort to do so, and couldn’t, in which case they’re not going to let that delay them.

I think it was correct for TIME to categorize this as ‘Anthropic drops flagship safety pledge.’

ASL-levels are deprecated in favor of ‘requiring analysis and arguments making a strong case for safety.’ They admit this ‘leaves flexibility,’ and that different people will disagree on what that means.

The central point of the ASL-levels, from an outsider perspective, was to tie Anthropic’s hands, so they would have to do the thing they agreed to do even if it later proved expensive, or otherwise at least have to admit they didn’t do it.

The goals Holden listed were to create a forcing function on risk mitigations, and create a testbed for practices and policies, and work towards consensus and common knowledge about risks. Holden thinks that RSPv3 effectively preserves these goals. I don’t think it does.

If all you have to do is make a self-convincing safety case, well, you can always do that if it’s sufficiently important to you to do that.

I am sympathetic to ‘we do not know what all the red lines will be.’ But if you cannot articulate any red lines, that is not a good sign.

I read this change as: We have almost no meaningful commitments at all around decisions about model releases, even if we keep to all of our current ‘commitments,’ which aren’t commitments.

They promise make a good argument. They decide what is a good argument.

The RSP now requires publishing an FSP, a Frontier Safety Roadmap, in case the acronyms were insufficiently confusing, describing risk mitigations across Security, Alignment, Safeguards and Policy.

Why not have those be in the RSP? I think ‘because they get updated a lot.’

These are ‘public goals’ but not ‘commitments,’ despite the commitments also not being commitments. We need new words for things.

In principle it seems fine to have both hard commits and aspirational goals, but that only works if there are any hard commits and there aren’t any. There are soft commits and aspirational goals. So, grade on the soft commits, then?

Is this because they want to ensure no violations of SB 53 or RAISE?

Risk Reports will provide detailed safety information at time of publication and be published online every 3-6 months. External review will be required ‘in certain circumstances.’ This is in addition to on-model-release capability assessments and model cards, not a replacement of them.

Why would you ever not want external review? Holden says external reviews are high-stakes and experimental, but I think given this isn’t lining up with a model release and doesn’t serve as a stopping function you can just do them.

How do you assess capabilities meaningfully if you don’t line up with new model releases? How do you do risk assessments on this cadence?

The answer is that they are committing to doing a diff with the old risk report upon release of a new model, or within 30 days of realizing they have an internal model that requires such a diff.

3-6 months is a lot when model releases are every 2 months.

‘Risk-benefit determination’ is how everything always works in the end, but seems like another step towards in practice not letting risks stop you.

External reviews are good but again they look like they’ll be too late?

What is being checked for is overhauled.

Biological and chemical safeguards continue, see next section.

Radiological and nuclear are fully gone, we don’t care anymore.

Autonomy isn’t checked because the models fully have autonomy, and we’re proceeding with that assumption.

Cyber Operations is fully gone. Perhaps because it would pass every test? This is despite it being a core test for both OpenAI and Google.

The effective compute checkpoints are gone.

Overall: Oh, the standards you’ll raise. Oh, the goalposts you’ll move.

It’s hard for me to take this seriously when the evaluations will be vibes-based and there are no real commitments even then. Anthropic is going to check the vibes, and decide if it seems safe or not. Tell me why I’m wrong.

Biological and chemical safeguards.

Non-novel chemical or biological weapons production is the new CBRN-3.

This calls for ASL-3 protections in essentially all models going forward.

They explicitly aim for and expect to hit the industry-wide standard (note that this implies that where they don’t say they expect or aim to hit that standard, it means they presumably don’t expect or aim to hit it).

The goal is a ‘strong argument’ that small teams won’t be significantly more likely to cause catastrophic harm, presumably via precautions against model weight theft and monitoring of model behaviors.

It is presumably impossible for open weights models to satisfy this.

Novel chemical and biological weapons are the new CBRN-4.

The weapon enabled in the hands of ‘moderately resourced expert-backed teams’ has to come ‘with potential for catastrophic damages far beyond those of past catastrophes such as COVID-19.’ So it’s a super high threshold, and presumably you can delete ‘chemical’ here.

The additional response here is ‘create policy recommendations’ for early detection of particular threat pathways.

Their recommended action is again ‘a strong argument’ for a set of mitigations, which here they anticipate matches RAND SL4, plus the usage monitoring.

The plan of ‘monitor usage’ seems not great given the state of jailbreaking.

High-stakes sabotage opportunities is a new category aimed at internal systems at AI companies, especially regarding successor models. Good idea.

While I’m happy to have sabotage, it is a strange framing or focus as opposed to worrying about AIs breaking free of control, or taking over control, or doing various other things. It’s not clear the extent to which any actions that hurt our chances count here, and thus they’re all included, versus not. Either way, this is reasonably confusing framing.

Their ‘commitment’ response is to detail the systems’ capabilities. They also expect to meet their policy recommendation requirements.

That requirement is again a ‘strong argument,’ here that the model will not carry out sabotage ‘leading to irreversibly and substantially higher odds of a global catastrophe.’

That’s a pretty high bar for sabotage to count.

They expect this to eventually involve internal compartmentalization, restriction, and code review to prevent excessive sabotage opportunities for AI models, capability assessments, monitoring and/or restricting behavior and usage internally and evidence of lack of propensity.

If that was a concrete set of requirements, it would probably be strong.

As a ‘we expect this will require,’ it’s harder to know if it means anything.

‘Automated R&D in key domains’ is the new AI R&D-5, the big kahuna, operationalized as ‘could compress two years of 2018-2024 AI progress into a single year’ which is a 100% speedup.

As they note, this is a high bar, and they are currently missing lower bars.

Claude Opus 4.6 estimates it would have provided about a 9% speedup.

GPT-5.4-Pro estimates about 30% speedup (Thinking said 25%).

I would take the over on that.

I would also want an additional threshold somewhere between ‘what Claude Opus 4.6 can do’ and ‘the singularity starts right about now.’

What do they plan to do if or by the time they hit ‘automated R&D in key domains’?

‘Resource and complete significant “moonshot R&D for security” projects.’

Achieve an ‘eyes on everything’ state for our internal development.

Perform systematic alignment assessments.

Develop internal red-teaming superior to external red-teaming efforts.

Publish Risk Reports.

Do better than the other guys, but they can’t guarantee anything.

What’s the desired global request? Another ‘strong argument,’ in this case for:

No team of users will cause substantial new catastrophic harm with this, including malicious employees with maximum access.

That’s misuse prevention and guarding of model weights.

A lack of dangerous goals trained into the models, or other model propensity to cause catastrophic harm.

These are good safeguards and you do need them, but they’re not central.

None of these R&D-triggered safeguards address the central threat model of automated R&D, which is rapid capability gain or recursive self-improvement, together with all the resulting Yudkowsky-style dangers, alignment issues and de facto goals, instrumental convergence and sharp left turns and suddenly outsmarting you, not intentional misuse or existing bad behaviors prior to achieving sufficient capabilities. The actions above cut off some secondary ways things can go wrong, totally do those things, but I don’t see how the above actions give you assurance that you are ready to go into that endgame, at all.

Either this is ignoring superintelligence, or not taking it seriously.

The responses promised for the new potential capabilities do not seem to reflect the gravity of what they represent. Even the ambitious goals don’t seem ambitious.

Precise language becomes vague language, throughout.

Various changes to governance commitments. Some are gone, some added. Readiness and notification in particular look gone without replacement.

They commit to having a Responsible Scaling Officer to handle all this.

The Board and LTBT are introduced as additional explicit veto points: “In the event that marginal risk analysis (see previous section) plays a major role in a decision to move forward, explicit approval of the Risk Report by the Board and LTBT (rather than just the CEO and RSO) will be required.”

This is good, but I’d like to see this not be a conditional. It seems reasonable to say that any substantial increase in capabilities should require the explicit approval of the Board, CEO, RSO and LTBT.

I would like to also see veto points among the associated technical staff.

As Yoav Tzfati points out, the CEO and RSO are free to decide whatever they want, no matter what the reports say, subject to other veto points. The Roadmap

The roadmap is mostly very high level, but does have some good details. It contains some aspirational goals on long timelines. Here are some examples.

July 1, 2026: Roadmap for policymakers. We don’t have one yet?

October 1, 2026: Upholding Claude’s Constitution, keeping it up to date, doing assessments.

January 1, 2027: World-class internal redteaming.

January 1, 2027: Fully automated attack investigations.

January 1, 2027: ‘Moving towards’ an “eyes on everything” internal state.

July 1, 2027: ‘Leveling up across the board’ on security, with many details.

The key future event is automated R&D, which they anticipate could arrive as soon as early 2027, which they reiterate is why they have these goals to get to by then:

By that point, we expect to have accomplished most of the goals listed above, including:

Resourcing and completing significant “moonshot R&D security” projects.

Developing world-class internal red-teaming and automated attack investigations.

Publishing and advocating for a roadmap for policymakers to achieve industry-wide safety without unnecessarily limiting the benefits from AI development or slowing the AI development of democracies relative to that of autocracies.

Achieving an “eyes on everything” state for our internal AI development.

Consistently implementing systematic alignment audits and other measures for upholding Claude’s Constitution.​

This fleshes out some things from the RSP and makes them more concrete.

You Came Here For An Argument

The central thing Anthropic wants is for companies to have to ‘make a strong argument’ that they have guarded against various risks, which they anticipate would require various actions but they’re not asking to require those actions.

This stems (I presume) from the philosophy of maximum flexibility. But maximum flexibility means requiring maximum trust.

Who gets to decide whether you made a strong argument?

Presumably Anthropic decides whether Anthropic made a strong argument.

Even if you trust Anthropic, do you trust OpenAI to decide if OpenAI did so? Google? Meta? xAI? I don’t.

A fair retort is ‘do you want the government making that decision, giving it an arbitrary veto over model releases, that it could use as a point of limitless leverage?’ Certainly this month has shown us one big danger in that approach.

Thus I think if we want to use ‘strong argument’ style language, we need a solution to this problem, and right now we have no such solution. Until we can find some process that we can trust, it needs to be more concrete than that, or it won’t have teeth.

They only ‘commit’ to the strong argument when Anthropic is in the lead.

If they face competitors with strong safety measures, they only promise to meet or exceed the standards of those competitors (and remember what ‘commitment’ means in this context).

If the competitor is improving on Anthropic’s standards, they’ll try to meet the higher standard, but they don’t promise anything.

They keep using that word ‘commitment.’ I will stop putting it in air quotes if they give clear communication that something really is a commitment that they cannot modify later as needed, with at least some slower and clear procedure required.

What about if they have a strong competitor with worse safety measures, and lack a ‘significant lead’? That didn’t make the chart above at all, and is the most likely scenario. One presumes the answer in practice is that they will proceed, so long as Anthropic feels its mitigations and chances are sufficiently