Amazon Science、AI エージェントの業務手順評価ベンチマーク「SOP-Bench」を公開

A standard operating procedure, or SOP, is the written set of steps an organization follows to correctly complete an important piece of routine work the same way every time. Almost every industry runs on SOPs. A hospital uses one to register a new patient, a logistics team uses one to decide whether a shipment qualifies as hazardous, a bank uses one to verify a new business customer, and a trust and safety team uses one to decide whether to remove a piece of content. SOPs carry an organization's hard-won knowledge, its compliance rules, and its decision logic in a form that any trained person can adopt and follow. As a result, they keep operations consistent and safe across different employees, shifts, and sites. SOPs are hard for AI agents to execute because they look cleaner than they actually are. A real procedure asks the reader to interpret instructions that were never fully spelled out, to draw upon knowledge that everyone in the field already shares, and to make judgment calls as conditions change. Consider the following passage from a patient intake procedure: Steps four and six tell the operator to verify the patient's insurance, without saying how the verification should be done or why it needs to be done twice. Someone who has worked an intake desk, however, knows that the first step confirms the patient’s coverage with the insurer, and the second step confirms that the patient’s information is correctly entered into the medical provider’s management system. An agent has none of that background, so it must guess what verification means here, remember what it did earlier in the procedure, and choose between tools that look nearly identical. This is the kind of moment where polished demo behavior quietly falls apart, and it is the kind of thing that most agentic benchmarks never test. Rigorously measuring what agents can and cannot handle is essential for building assistive tools that genuinely help rather than silently fail. Today we are sharing SOP-Bench, an openly available benchmark that measures how well AI agents carry out real SOPs authored by domain experts. It is the first benchmark of its kind to pair genuine enterprise procedures with functioning tools and ground-truth answers, so that an agent earns its score by completing the procedure rather than by producing text that an automated grader happens to like. We presented the benchmark at the 2026 Conference on Knowledge Discovery and Data Mining (KDD), along with experimental results showing where even strong foundation models come up short and an evaluation framework the community can build upon. Why existing benchmarks fall short Most agent benchmarks do one thing well. Some check whether a model can pick the right API for a request; others check adherence to a written set of constraints; still others measure the ability to plan a sequence of steps toward a goal. All of these are valuable, but each one isolates a single capability and tests it with clean, machine-formatted prompts that leave out the ambiguity and variability of procedures written by actual people. Executing an SOP requires all these skills, along with using multiple tools in a coordinated way across steps that depend upon one another, keeping track of what has happened so far, and recovering when something does not go as expected. Past efforts to adhere more closely to real business procedures have encountered limits. Some translate written procedures into executable workflows, but only for short descriptions in narrow domains, and the datasets behind them are often not released publicly. Others publish collections of genuine business procedures but stop at the text, without the tools or the known answers that would let anyone run an agent through the procedure and check its work. That is the gap SOP-Bench is built to close. It brings together elements that have previously appeared only separately: realistic procedures with the ambiguity left in, coverage across many different industries, working tools an agent can call, and a way to grade the result against ground truth. What we built SOP-Bench turns real procedures into runnable tasks. It covers 12 business areas, including healthcare intake, dangerous-goods classification, customer service, content moderation, financial compliance, and warehouse inspection, with more than 2,000 tasks in total. Each task comes with the tool interfaces an agent requires and a correct outcome. An agent runs the procedure by calling tools, and we can validate its work against ground truth rather than against a model's opinion of it. SOP-Bench is a framework rather than a fixed set of tasks. It comes with two baseline agents, but a team can drop in an agent of its own, test it against the included procedures, and even add its own procedures. That's because each procedure is just four things: the SOP text, the tools an agent can call, the specifications for those tools, and a set of test cases with known answers. The framework runs every task, keeps a full record of the tool calls and reasoning behind each decision, and grades the outcome against the known answers. Scores are reproducible, and failures can be tracked back to the steps where they happened. In practice, this lets a team try its own agents on its own SOPs before trusting them in production. Constructing realistic SOPs that span industries is difficult, but it’s where Amazon has an advantage. It Amazon provides experts from all relevant fields working in parallel, a culture in which those experts already document their work as written procedures, and enough infrastructure to execute thousands of tasks simultaneously. To construct SOP-Bench, we paired experts with AI, while letting the experts determine whether an answer was correct. They authored the original procedures from real industrial workflows and set the context for each task. An Anthropic Claude 3.5 Sonnet v2 model then handled the slow, mechanical work of turning each procedure into something a machine can run and generating the data schemas, the mock APIs and tool specifications, the tool code, and datasets that deliberately mix ordinary cases with edge cases and outright failures. Every generated item went back to the experts, who confirmed that the logic held, corrected the procedures, checked the data, and ran the code to be sure it behaved. No proprietary or sensitive data was involved at any stage. What we found We ran two deliberately simple agent designs, a function-calling agent and a reasoning-style agent, across 11 frontier models. These agents are a baseline for others to improve upon rather than an assertion of the best possible system. Even so, a few patterns came through clearly. Newer is not automatically better The most surprising insight was that upgrading the model sometimes lowered performance. On the reasoning-style agent, the newer Claude 4.5 family scored lower than the older Claude 4 family. The same reversal held when we compared individual models on the same setup. For a team running agents in production, this is the finding that matters most, because a routine upgrade can lower the success rate with no obvious signal that anything changed, and the only reliable way to catch it is to test on the procedures the team actually runs. More tools can make an agent worse We took a single video-annotation procedure and gave the agent two versions of its toolkit. One held exactly the six tools the task required. The other kept those six but buried them among 20 extra tools that looked plausible but did nothing useful. Success nearly halved with the larger toolkit, even though every tool the agent needed was available. The lesson is that capability is not free, and trimming an agent's tools to fit the task may be a key component of getting it ready to deploy. No single setup wins everywhere No one pairing of model and agent came out ahead across the board, and the combination that performed best on one procedure was often a weak choice on another. The gap between procedures was wide. On the easiest ones, such as triaging incoming e-mails by intent, agents arrived at the correct answer approximately nine out of ten times, while on the hardest, such as annotating objects in a driving video, they were correct approximately one out of four times, a more-than-threefold gap across the suite. Trusting a single benchmark score would tell a team almost nothing about how the same setup would behave on the next use case. How an agent is built matters as much as which model runs inside it When we compared the two agents head-to-head on the same model, the reasoning-style agent came out slightly ahead on average, yet it won on only eight of the thirteen procedure runs in the comparison, and it took about a third longer per task. Some procedures clearly favored one agent and some the other, so the shape of the procedure, rather than a single overall average, should drive the agent choice. One open question remains and runs counter to what might be expected. A procedure that was mostly long stretches of reading, with only a couple of points where a decision had to be made, gave agents more trouble than one packed with many more decisions. The natural assumption is that complicated logic is the hard part, but here the longer, simpler-looking procedure scored far worse. We are not claiming that the length of the reading is the cause, since the two procedures differ in other ways as well, including how many tools they involve. But this is a question that the benchmark was built to help examine, one we hope other groups will examine with us. Taken together, these results are not a verdict on any single model. They are a map of where the field still needs to invest and a reminder that raw capability does not guarantee reliability on the kind of procedural work that businesses depend upon. More practically, they help teams identify the specific steps where human oversight remains essential. AI’s weakness on those steps surfaces only in sustained, tool-using runs against realistic procedures, which is why a static skills test is not enough for agents that are being deployed alongside human operators on procedural tasks. Get started with SOP-Bench We are releasing the full benchmark on GitHub and on HuggingFace. The release includes the 12 expert-authored procedures, the generated tools and datasets, the two baseline agents, and the evaluation code that scores an agent's runs against ground truth. Researchers and teams can evaluate their own agents against the existing procedures or extend the benchmark to new domains using the same human-and-AI method we used to build it. We are especially interested in procedures from industries we have not covered yet. We also plan to add harder variants of the same procedures, instructions that include images and tables, and procedures with nested structures that force an agent to switch context partway through. If you build agents, evaluate them, or want a clearer picture of where they stand on everyday operational work, we would welcome your contributions and feedback.