AI エージェント評価の未来：エージェントが審査役となる時代へ

In October 2024, a research paper proposed that the best way to evaluate an AI agent is another AI agent. In 2025 that paper got accepted to ICML, one of the most prestigious conferences for machine learning research, a strong signal that the top people in the field thought there was something there. By January 2026, enough people had written about the idea to support an academic survey. By July 2026, it started shipping in products. Ideas do not usually move from paper to production in 20 months, and when one does, it is worth asking why. I’m here to make the case that agents changed what failure looks like, and so the evaluation layer has no choice but to change with it. If you run evals today, I believe the way you run them is about to change.

Agent-as-a-Judge is an evaluation approach in which an AI agent investigates another agent’s behavior, gathers the evidence it needs, and makes a judgment against defined criteria. Unlike a single-pass LLM judge, an agent judge can follow a trajectory, inspect tool calls and intermediate steps, and decide what evidence to examine before reaching a verdict.

imageTimeline from the Agent-as-a-Judge research paper through ICML acceptance to agent-as-a-judge shipping in production evaluation products.

Why LLM-as-a-Judge breaks down for agents

Evals grade outputs, but agents fail in trajectories. The eval most teams run today is LLM-as-a-judge: an LLM scoring the output of another LLM. You write a prompt template containing a rubric, you do “column mapping” (wiring template variables like {input} and {output} to specific fields of your logged data), and every sampled row gets one LLM call that returns a label, a score, and an explanation. This design was a good fit for the applications of 2023. An app took a prompt and returned an answer, so the failure, if there was one, was sitting right there in the answer.

Agents broke that assumption. An agent produces a trajectory: a full sequence of model calls, tool calls, and intermediate decisions between the user’s request and the final response. The failures that matter now live inside that sequence. Anyone operating agents in production knows the catalog: stuck loops, context forgotten partway through a run, failed recovery from errors, plausible answers built on a tool call that silently returned garbage, and agents claiming a task was done when it wasn’t. Every one of those can coexist with a final answer that looks perfectly reasonable.

The mismatch is structural, not a prompt-quality problem. When you map columns into a judge template, you are deciding where the failure lives before you have looked at it. If the actual failure is a stale argument passed to a tool in step 14 that the agent then papered over, no judge pointed at the final answer will ever see it. You need something more capable, looking at the bigger picture: another agent. The judge has to be able to go looking through all the evidence. And this isn’t just my guess: we already have research that it works.

Agent-as-a-Judge already beat the average human evaluator

That research is “Agent-as-a-Judge: Evaluate Agents with Agents” (Zhuge et al., October 2024) which came out of Meta AI and KAUST. Its senior author is Jürgen Schmidhuber, co-inventor of the LSTM, the neural network architecture that ran most speech recognition and machine translation in the decade before transformers: a strong signal that this is an idea to take seriously.

The team that wrote the paper built DevAI, a benchmark of 55 AI development tasks, ran three open-source coding agents against it, and evaluated the results three ways: a panel of human experts, an LLM judge, and an agent judge equipped with tools for locating files, reading code, and tracing requirements through the generated project.

One interesting finding: the three expert evaluators disagreed with each other 10-30% of the time, and only converged after hours of real-world debate. Human evaluation is the gold standard for evals, but we’ve known for a long time that human evaluators are hardly unanimous. Against the consensus those debates produced, the agent judge scored around 90% alignment. By contrast, the plain LLM judge managed roughly 70%. But the most interesting finding: the agent judge reached the consensus answer more frequently than any individual human expert did. Agents are genuinely competitive with human expertise.

imageAgent-as-a-Judge benchmark results showing an agent judge reaching consensus more frequently than individual human experts.

It’s also worth mentioning that in addition to being as good if not better, an agent judge is orders of magnitude cheaper and faster. The agent judge evaluated the full benchmark for $31 in API calls, against nearly $1300 worth of paid time from the experts, and finished in about 2% of the time it took humans. The agent can also run continuously without getting tired, a major problem for human judgement.

Which isn’t to say the agent was perfect. The paper’s authors tried giving the judge a memory module, which let it build on past judgments, but that made it less accurate because one early mistake propagated through every later decision that referenced it. The judge that worked was the one that gathered fresh evidence for every requirement.

Why the field is moving toward agentic evaluation

After the ICML 2025 acceptance, follow-up work produced judges that verify math by invoking theorem provers, judges that run generated code before scoring it, courtroom-style frameworks where prosecutor and defender agents argue in front of a judge agent, and evaluation agents for web agents in benchmarks like Mind2Web 2 at NeurIPS 2025. Domain-specific judge agents appeared in medicine, law, finance, and education.

By January 2026 the field was large enough to need a comprehensive survey (You et al.), which formalizes the transition from LLM-as-a-judge to agent-as-a-judge and sorts judge systems into three developmental stages: procedural judges that follow fixed multi-agent workflows, reactive judges that route and invoke tools based on what they find, and self-evolving judges that write their own rules. The survey’s diagnosis of why the shift happened matches the practitioner experience exactly: single-pass judges carry biases they cannot audit, cannot verify anything against the real world, and get overloaded when their instructions have many constraints.

Agents as a judge are extremely capable, but that doesn’t mean everything should be an agent.

Where Agent-as-a-Judge fits in a tiered eval strategy

At Arize we often cite the Swiss cheese model of layered defense as the right way to build evals. Start with cheap, deterministic code evaluators, layer on LLM judges, and top off with human intervention when necessary. Each approach has holes, but layered together you catch nearly everything. Agent as a judge is a new layer you can add to the stack, catching trajectory failures, multi-step reasoning quality, and unexpected failures you never wrote a specific eval for.

Arize AX shipped a productized agent as a judge last month: a coding agent in a sandbox that reads trace data at runtime, takes scoring instructions in plain language with no column mapping, and writes its verdicts back as ordinary eval columns. Signal, which shipped alongside it, is the same idea run continuously against production, grouping recurring failures into ranked issues with evidence and proposed fixes. Other vendors will build equivalents, because nothing about the logic is proprietary: if failures live in trajectories, the judge needs runtime access to trajectories, and a judge with runtime access and discretion about what to examine is an agent by definition. Before adopting any of this, though, it is worth asking what the failure modes of the judges themselves are.

Who judges the judges?

The agent as a judge performance numbers I cited above came from a benchmark where a human consensus existed to measure against: production traffic has no such consensus, so spot-checking judge verdicts against humans remains necessary. What does running a full agent per evaluation cost at production scale, and how aggressively should you sample? An agent judge is itself nondeterministic, so how consistent is the same judge on the same trace across runs? And research is already cataloguing biases specific to trajectory judging, such as favoring longer trajectories regardless of whether the extra steps added value. None of these is a reason to stay on output-only evals, which have all the same problems plus blindness to the trajectory. They are the reasons the original paper validated its judge against humans, and the reason you should too.

Evaluation is becoming part of the agent, not a report about it

The 2024 paper closes with an outlook its authors call a flywheel: agent judges providing rich intermediate feedback that agents then learn from, so that evaluation stops being a report card and becomes a training signal. Twenty months later the judges are real, they run continuously, and they propose their own fixes. Every prior generation of software treated evaluation as something you do to a system from outside. Agentic evaluation is the first version where the eval layer participates: it investigates, it accumulates evidence, and it feeds what it learns back into the thing it is judging. That is where the field is going, and the teams that treat their evals as agents to be engineered, rather than tests to be written, will get there first.

The post Where agent evals are going: Agent-as-a-Judge appeared first on Arize AI.