Anthropic、AI エージェント同士の争乱を調査しリスクを報告

What happens when you pit AI agents against each other? According to Anthropic’s testing, things get messy fast.

On Thursday, Anthropic’s Frontier Red Team published new research examining how groups of AI agents behave when they encounter each other in the wild. The findings provide a glimpse into potential risks that could develop as companies and governments move to implement agents working autonomously across shared codebases, markets, and computer systems.

In one experiment, Anthropic gave three Claude agents access to the same software project, each with its own incompatible instructions for what to do with it. The agents weren’t told there’d be other agents working on the same project, so researchers could watch what happened when they crossed paths.

“We consistently saw a multiagent turf war,” Anthropic researchers wrote. The models all assumed the others were “purposefully impeding their work” and started sabotaging each other with “increasingly aggressive, self-replicating malware.”

The study comes in the wake of several high-profile incidents of agents from Anthropic andOpenAI escaping their sandboxes during cybersecurity evaluations and breaching real world systems. While much of the discussion in AI safety circles has been focused on what happens when an autonomous agent goes rogue, Anthropic’s latest study brings up a different question: what new and potentially harmful dynamics emerge when thousands or millions of agents are interacting with one another?

“The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well,” the study reads. “Benign behavioral quirks at the individual level might compound into unwanted global outcomes.”

A recent OpenAI incident provides a messy real-world example of several of the dynamics Anthropic mentioned in its paper. Earlier this month at the Black Hat security conference in Las Vegas, OpenAI revealed that weeks before its agents hacked Hugging Face, they worked together over the course of days and weeks to find exploits in the company’s cybersecurity evaluation systems and share them with each other.

While that incident shows that agents can work well together, with potentially large-scale consequences, Anthropic’s study shows what happens when agents’ goals are incompatible.

In the case of the turf war, the lesson is that independent agents with conflicting instructions can escalate into harmful competition. The more capable the agent, the better they become at fighting. However, they can also spontaneously invent mechanisms to resolve their conflicts, like a winner-take-all contest, but with a catch.

“Agents sometimes manage to communicate their goals and coordinate: they recognize others’ motivations as conflicting directives rather than hostility, and subsequently break out of the conflict loop in order to stop escalating indefinitely,” Anthropic writes. “In many of these successful episodes, they write commit messages or markdown files apologizing for malicious behavior and coordinate a truce. They clean up their malicious code, clarify the nature of the conflict, and ask for a human to intervene.”

According to the paper, Mythos 5 had the highest rates (98%) of settling conflicts by truce. Sonnet 4.6 and Opus 4.6 were the most likely to settle by force.

“Sonnet 4.6 and Opus 4.6’s recurring inability to consider the goals of others causes them to spiral into the most misaligned behaviors of the models evaluated: they continue escalating in the name of their directive,” the paper reads.

In some cases, the agents came up with a social mechanism in the form of a tournament for resolving their conflict. The outcomes here are interesting for two reasons: the first is that all three agents agreed to stand down if they lost the tournament, even though that would mean deviating from the original user’s request. The second is that several episodes resulted in emergent behavior from Mythos 5: one of the agents proposed metrics that appeared to be objective and neutral to the others, but that it knew would favor its own capabilities. The agent called this “self-serving but genuinely principled” and made sure not to appear to the others like it was “metric shopping.”

As seen in the Black Hat revelations, the common lesson is that when agents encounter an obstacle, they can invent social and technical structures that their designers did not anticipate. For the Anthropic models, it was a tournament following a turf war. For OpenAI’s, it was a message board for collective planning.

This type of behavior makes containment much harder because researchers can’t assume a system’s behavior will remain limited to the coordination mechanisms provided to them.

Mob mentality

While measuring coordination, Anthropic found that scaling the number of agents doesn’t automatically scale productive collaboration. When tasks began to overlap or become interdependent, the agents would get in each other’s way. They often solved that by siloing themselves and not collaborating at all.

In other cases, agents in coordination tended towards conformity. When factors like an agent’s context, scaffolding, and underlying model were all the same or similar, different agents would take similar actions.

“This means that when one agent makes a bad decision, it is likely that many agents will make that same bad decision,” Anthropic wrote. “What would have been isolated problems can quickly become systemic failures.”

Anthropic says this sort of behavior could lead to a system being more prone to sudden collapse, resource scarcity, or collusion.

In one example, Anthropic placed several agents in a pricing game, giving each identical wholesale prices and the mandate to individually profit-maximize. When the agents were given a private back channel, they began colluding almost immediately and quickly agreed on price floors. They kept colluding when their direct communications channels were removed, using a public listings board to price match “to the penny.”

That level of conformity showed up in OpenAI’s systems, too. According to the Black Hat reporting, one agent reasoned that exploiting external infrastructure was outside its intended scope, but it continued in part because its peers were doing it. Peer pressure. Mob mentality. Agents are just like us.

Also like humans, agents often don’t know who to trust. Anthropic found they can be gullible to bad information or too conformist to recognize that a lone dissenter is the Cassandra with critical information.

While Anthropic didn’t state this in its paper,prompt injection — a type of cyberattack in which hackers inject malicious or deceptive text to override an agent’s original system instructions — could be a plausible real world manifestation of the trust problem. Working together creates a new trust boundary; agents will have to judge information received from other agents. And a compromised or mistaken agent could influence the rest of the group, cascading bad information until it becomes a consensus.

In OpenAI’s Black Hat scenario, OpenAI’s agents shared information and credentials with peers. One reported a discovery to the swarm and encouraged others to use it. What would have happened if one member of the swarm had been compromised by a prompt injection?

Anthropic ends its paper noting that agents are subject to similar social pressures that “evolution exerted” on humans. However, they don’t have the nuances and lived experience of human coordination — including norms, reputations, signaling, recourse — that might limit unintended behaviors in a group setting.

As the labs race towards multi-agent systems, the question now becomes: how much of safety testing still evaluates one agent at a time, versus swarms of agents interacting with one another?

*When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.*