エージェント評価：詳細ガイド（53 分読了）

Evaluation is one of the most important research areas for large language models (LLMs). Recently, patterns in LLM usage and evaluation have drastically changed. Whereas we previously evaluated LLMs using benchmarks composed of static questions or short conversations, we now have agent systems that operate over long time horizons and interact with the environment. Agents are difficult to properly evaluate due to their complexity and autonomy. To accurately measure the capabilities of an agent system, we must build harnesses that are realistic and capable of testing agents similarly to how they are used in practice. Building such evaluation capabilities is now more important than ever due to the growing adoption of agents in high-stakes applications like coding and medicine.

This overview will provide a detailed guide of how current agent systems are evaluated. We will begin by developing an understanding of agents in general, covering everything from basic concepts to multi-agent systems. We will then provide a clear framework for the agent evaluation process based upon common patterns observed in practice. Building upon this knowledge, we will end with several case studies of recent agent benchmarks and provide a roadmap that outlines how to build our own agent evaluation by applying similar concepts. Although evaluation is time-consuming and difficult, learning how to properly evaluate agents is incredibly valuable. By rigorously measuring performance and not relying on anecdotal checks, we can rapidly improve agent capabilities.

“LLMs are becoming increasingly capable of handling complex, multi-step tasks. Advances in reasoning, multimodality, and tool use have unlocked a new category of LLM-powered systems known as agents.” - from [3]

During the early days of the agentic era, the distinction between an agent and a normal LLM was unclear. After all, LLMs are at the core of all agent systems. Over time, the key characteristics of agent systems began to emerge:

• Reasoning
• Calling tools
• Solving complex, multi-step problems
• Recovering from errors
• Taking actions on behalf of a user

LLMs are capable of reasoning, tool calling, and problem solving, but agents are distinguished by their ability to combine these components within an agentic loop to solve difficult problems; see below. In fact, the simplest definition of an agent is an LLM that autonomously uses tools in a loop. Unlike conventional LLMs, agents must assess intermediate results and dynamically recover from their own errors to operate effectively within this agentic loop.

Given that agents reason over long time horizons and use tools, *they can also be characterized by their level of autonomy*. Conventional LLMs are often used for narrow, well-scoped tasks (e.g., classification or question answering) where the model produces an output but does not independently continue acting or have the ability to modify its environment. On the other hand, agents are usually allowed to control their own workflow over a period of time and can use tools to take actions autonomously (e.g., modify a calendar event or book a plane ticket).

As shown above, the input-output interface of an LLM is straightforward: *we provide a prompt as input and receive an output*. This simple interface played a large role in the adoption of AI by the general public: *LLMs are both useful and intuitive*. This text-to-text interface can also be extended to handle a variety of auxiliary behaviors like reasoning and tool use, making the gap between a conventional LLM and an agent relatively small. In particular, an agent system is typically composed of three components:

• The underlying LLM (or reasoning model)
• Tools for the agent to use
• Instructions for the agent

The LLM serves as the brain of the system, judging progress toward a final solution and deciding the steps that must be taken in order to get there. In many cases, however, arriving at a correct final solution requires more than just the LLM itself. We must clearly specify the agent’s expected behavior and enable the model to invoke tools or perform complex reasoning when necessary.

Tool use. In order to interact with the external environment, LLMs need to be able to use tools like APIs, CLIs, or MCP servers. For example, if we want our agent to reserve a table for us at a local restaurant, we can simply teach the model how to craft a call to the OpenTable API. Concretely, tool calls can be handled naturally within the LLM’s token stream by creating a set of special tokens related to tool calling and teaching the LLM how to use these tokens; see below.

For example, Qwen3 models handle tool calling via several XML-style tags that are stored as special tokens within the tokenizer:

• <tool> and </tool> are used to encapsulate tool definitions.
• <tool_call> and </tool_call> are used to encapsulate a specific tool call, including the tool to be invoked and associated parameters.
• <tool_response> and </tool_response> are used to encapsulate the result or response from the tool that is called.

The structure of the instruction template for Qwen3 is shown below. We define available tools and the expected format for calling them in the system message, which provides necessary context for the model to *i)* understand what tools are available and *ii)* construct valid calls to these tools. The model can then craft tool calls by outputting <tool_call> and </tool_call> tokens while generating output. At inference time, we interrupt generation whenever a </tool_call> token is generated. We can then parse the request, invoke the tool, concatenate the result to the current output, and restart generation. The model then has the result of the tool call in its context when generating the remaining output.

Creating useful tools that can be easily understood and correctly invoked by agents is an art. Tools should be well-documented, have a clear purpose, overlap minimally with other tools, and recover gracefully from errors. The easiest way to determine whether a tool is designed correctly is to simply ask yourself: *Would a human engineer be able to use this tool from the provided documentation?*

Tool calling serves multiple purposes within an agent system. The LLM can use tools to retrieve external information to help reduce hallucination, accomplish tasks within its environment, execute code, call other agents, and more. If there is no API available for a task that needs to be accomplished, we can even rely upon computer-use primitives to allow our agent to directly interact with arbitrary applications on a computer—*similarly to a human using a computer*.

*“Computer use agents interact with software through the same interface as humans—screenshots, mouse clicks, keyboard inputs, and scrolling—rather than through APIs or code execution… Each tool should have a standardized definition, enabling flexible, many-to-many relationships between tools and agents. Well-documented, thoroughly tested, and reusable tools improve discoverability, simplify version management, and prevent redundant definitions.”* - from [1]

We can teach an LLM to use tools in multiple ways—*such as through in-context learning or finetuning*—but tool calling is no different than any other skill that an LLM could learn. There are even entire benchmarks for measuring tool calling capabilities, where performance is measured using a variety of different metrics:

• Invocation accuracy measures whether the LLM correctly decided to call a tool when it should, or avoided calling one when it should not. For example, the model may call a tool when it should answer the question directly.
• Selection accuracy measures whether the LLM called the correct tools, usually by keeping track of a ground truth trajectory that includes a list of necessary tools for solving a particular problem.
• Structural accuracy and schema validity measure whether the structure of a tool call is correct. For example, our model should not include wrong arguments in a tool call or provide an incorrect call structure.
• Trajectory accuracy looks at the sequence of tool calls made by the model when solving a problem and compares them to ground truth in some way (e.g., correct call order, correct selection, using unnecessary tools, and more).

We can also use outcome-oriented evaluation by focusing on whether the LLM’s final answer is correct instead of the tools it uses to produce an answer.

Reasoning also plays a key role in our ability to build effective agent systems. Although we can use a standard LLM as the underlying model for an agent system, we usually benefit from using a model that has reasoning capabilities. To solve multi-step tasks, an agent must be able to decompose difficult problems into smaller, simpler parts and solve each of those parts—*possibly with the help of tools*—to arrive at a final solution. Additionally, the model must be able to self-reflect and recover from its own mistakes while solving problems. Handling long-horizon problems in this way requires a level of reasoning and reliability that has only recently become possible with the advent of reasoning models.

Unlike a standard LLM that produces output given a prompt, reasoning models do not immediately provide an answer. Instead, they write a long—*usually hidden*—textual reasoning trajectory1 prior to their final answer; see above. This thinking process naturally lends itself to inference-time scaling laws. The model can be taught to dynamically reason for different lengths—*often measured by the number of tokens in the reasoning trace*. Longer reasoning traces require more inference-time compute, as we are generating a larger number of tokens. However, this compute investment usually leads to improved performance, *allowing harder problems to be solved by allowing the model to spend more time reasoning*. For more details, please see the long-form writeup on reasoning models linked below.

Reasoning models and standard LLMs do not have to be mutually exclusive. We will soon learn about multi-agent systems that can be composed of multiple LLMs or agents. For example, we could use reasoning models for high-level planning and workflow management—*areas where reasoning capabilities are important—*while exposing a standard LLM as a tool for simpler subtasks.

Instructions. The final component of an agent system is the agent’s instructions. The core of these instructions can be taken from existing documentation for the domain (e.g., annotation guidelines or policy documents). Instructions should be as clear as possible, clarify edge cases via explicit rules or concrete examples, and specify the exact actions expected of the agent. Ideally, instructions should strike a balance between simplicity and specificity. We want instructions to be detailed enough to reliably guide agent behavior, but prompts that are overly complex become brittle and difficult to maintain. To help with writing better instructions for an agent, we can use a model-in-the-loop approach—*or an automatic prompt optimizer*—to iterate upon the instructions and improve their clarity.

“The agent tended to try to do too much at once—essentially to attempt to one-shot the app… this led to the model running out of context in the middle of its implementation… the agent would then have to guess at what had happened, and spend substantial time trying to get the basic app working again.” - from [4]

Instructions for an agent should explain not only the problem being solved, but also the best strategy for solving the problem. For example, we can instruct the agent to break a problem into smaller parts to avoid trying to solve a problem that is too large in a single shot. Additionally, we can further clarify the expected behaviors of the agent in this process, such as ensuring every step of a solution is implemented as a fully functional feature, writing persistent notes that can be used as a reference by future agents, or maintaining a global to-do list.

ReAct framework. A full agent system combines all of these components together in a while loop: *this is called an agentic loop*. In this loop, the agent will reason, call tools, and take actions based on the provided instructions until a terminal state has been reached according to the exit conditions for the agent2. At this point, the agent will relinquish control of the workflow—*either to a human user or another agent*. This entire process is considered a single “run” of the agentic loop.

The idea of an agentic loop is simple, but many variants of this idea exist. One of the earliest variants is the ReAct—*short for REasoning and ACTion—*framework [5], which proposes a fixed structure for the agentic loop; see above.

The ReAct agentic loop is organized into a set of sequential steps. At each step, our agent observes the current state of the environment, reasons about the best next steps, and takes some action based upon its reasoning process. A concrete example of this iterative problem-solving framework is provided below.

The key novelty of ReAct is its emphasis on both thinking and action. The agentic loop is structured such that the model explicitly reasons through each action that is taken and considers all context—*such as observations from tool calls or prior thoughts*—when deciding the correct action to take. Put simply, reasoning and action have a symbiotic relationship. For more details, see the full overview of ReAct and other common patterns in agentic frameworks linked below.

Single-agent systems are simple and incredibly powerful when equipped with the necessary tools for solving relevant tasks. Due to the complexity of orchestrating multiple agents, we should almost always start with a single-agent design and optimize this basic setup as much as possible—*single agents are easier to evaluate and maintain*. We can expand the agent’s capabilities by incrementally adding more tools and conditional logic—*in the form of prompt templates or modified instructions*—to the system. For the agent to perform well, we should provide clear names, descriptions, and arguments for all tools, as well as detailed instructions that outline the expected actions and outcomes for a task.

When are multiple agents needed? Multi-agent systems distribute task execution across multiple specialized agents in a coordinated fashion; e.g., see Claude’s multi-agent deep research architecture depicted above [13]. Multi-agent systems help logically separate components of a difficult task, but they are also more complex. For this reason, we should expand to multi-agent systems only when necessary. Common signs that using multiple agents may be helpful include:

• Instructions for the single-agent system are bloated, and the agent is struggling to follow instructions even with clear logic or templating.
• Tools are being selected incorrectly by the agent because there are too many available tools.

The reasoning capabilities of modern LLMs make it possible to properly select tools even from a large set. Issues arise when the set of tools that are available have similar or overlapping purposes, making tool selection more subjective. If this problem cannot be solved with cleaner tool specifications, splitting tools across distinct agents can lead to more accurate and reliable selection.

Types of multi-agent systems. There are many ways that one could design a multi-agent system. However, two patterns arise frequently in practice:

• Manager setup: a central “manager” agent orchestrates specialized agents via tool calls, where each agent handles a specific task or domain.
• Decentralized setup: multiple agents operate as peers by handing off tasks to one another based on their respective specializations.

In the manager setup, task execution is controlled by a single, centralized agent. The manager agent acts as the “glue” for solving a task by developing a problem-solving strategy, delegating parts of the work to specialized agents, and deriving a final solution by synthesizing these results. For example, a multi-agent translation system could use a manager agent to identify the requested language pairs and route each pair to a specialized translation agent; see below.

Such a manager setup is related to the concept of a sub-agent. Rather than keeping all state for a task within a single agent, we can trigger a sub-agent to handle task components. In this setup, our main agent coordinates the high-level plan for solving a task and interacts with the user, while sub-agents perform specific technical tasks (e.g., exploring code or retrieving information) and return relevant context to the main agent. Using sub-agents can help improve system reliability by preventing the main agent’s context from being overloaded.

Decentralized multi-agent systems have no centralized agent to orchestrate task execution. Instead, control is passed among multiple agents that operate as peers. Agents pass control to one another via tool or function calls. Such a setup works best when we do not need a central agent to synthesize results or interact with the user. In a decentralized system, multiple different agents can control execution and interact with the user as needed. For example, we can create an agentic support system that first uses an agent to triage the issue, then passes control to a specialized agent to solve that particular type of issue; see below.

<img src="https://substackcdn.com/image/fetch/$s_!WEjd!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2