AWS、AIエージェント評価の生産用ブループリントを公開

*This post was co-written with Motorway and the AWS Prototyping and AI Customer Engineering (PACE) team.*

Motorway, a UK-based online car marketplace, runs a daily auction where up to 8,000 dealers bid on up to 2,500 vehicles. Motorway worked with AWS Prototyping and AI Customer Engineering (PACE) to build an AI-powered dealer stock search agent that transforms how dealers find vehicles, replacing hours of manual filtering with natural language queries.

The challenge

The agent gives a confident-sounding response, but how do you prove it works reliably with real money on the line?

• Tool selection errors cause wrong search results, eroding dealer trust.

• Semantic search misinterpretations return irrelevant results. A query like “Petrol, Hybrid and electric cars up to 5 years old” requires the agent to correctly parse multiple constraints.

• Context drift in multi-turn conversations loses dealer refinements.

• Non-deterministic outputs make single-trial testing unreliable.

The solution

Together, Motorway and AWS built an end-to-end evaluation pipeline that reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from few hours to few minutes.

The pipeline combines the Strands Agents SDK with Amazon Bedrock AgentCore, a fully managed service for deploying and operating AI agents at scale. In this post, you will learn how to build this pipeline for your own agents:

• A two-phase evaluation strategy spanning build-time testing with strands-agents-evals (the open source evaluation library for Strands Agents) and production monitoring with Amazon Bedrock AgentCore Evaluations.

• A three-layer framework for assessing tool usage, reasoning, and output quality.

• A five-stage deployment pipeline with quality gates that block releases when metrics fall below thresholds.

A companion repository provides a deployable blueprint that you can adapt for your own agents. Although the blueprint uses AWS services, the core principles are essential and system-agnostic requirements for any production-ready AI agent. These principles include the three-layer evaluation framework and the use of the pass^k metric for consistency.

Prerequisites

You must have the following prerequisites to follow along with this post.

• An AWS account with permissions for Amazon Bedrock, AWS Lambda, Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon EventBridge, Amazon CloudWatch, and Amazon Simple Notification Service (Amazon SNS).

• AWS Cloud Development Kit (AWS CDK) v2 installed and bootstrapped.

• Python 3.14+.

• Access to Anthropic Claude models and Amazon Titan models through Amazon Bedrock model access.

• Familiarity with Python, AWS CDK, and agent concepts (tool calling, multi-turn conversations).

Time to complete: 30–45 minutes for initial deployment and 2–3 hours to customize for your domain.

Estimated costs: Running the sample evaluation suite costs approximately $5–10 in Amazon Bedrock inference charges. Production monitoring costs vary based on sampling rate.

Security note: The companion repository implements least-privilege AWS Identity and Access Management (IAM) roles, stores API keys in AWS Systems Manager Parameter Store (not environment variables), and uses typed parameters to help prevent injection attacks. See the repository README for details.

The worked example: A dealer stock search agent

Motorway built the dealer stock search agent on the Strands Agents SDK and Amazon Bedrock AgentCore. The agent exposes eight tools that combine structured filtering across over 89 vehicle attributes with vector similarity search powered by LanceDB and Amazon Titan Text Embeddings V2.

Dealers typically spent hours browsing through listings using CSVs and rigid filters. By introducing a conversational AI agent, dealers can now talk to the agent: “Find me diesel SUVs under 25k near my dealership” or “something sporty and automatic for a family.”

With around 1,500 concurrent users during peak hours, getting agent behavior right isn’t optional. A tool selection error or semantic search misinterpretation directly impacts user trust. Figure 1 shows the end-to-end request flow: dealers submit natural language queries through a web interface, which routes to Amazon Bedrock AgentCore Runtime. The runtime orchestrates calls to eight different tools while using Amazon Bedrock models (Claude for reasoning, Amazon Titan for embeddings). Tool responses flow back through the runtime to generate the final dealer-facing results.

Why agent evaluation is different

Large language model (LLM) evaluation focuses on text generation quality: coherence, factual accuracy, response relevance. Agent evaluation assesses something fundamentally different. Think of it this way: LLM evaluation examines engine performance. Agent evaluation assesses how the whole car drives in traffic, in rain, or with a backseat full of passengers.

Traditional LLM metrics don’t tell you whether the Motorway agent called the right search tool for “Grade 1 Suzuki models.” They don’t reveal if the agent passed correct filter parameters to LanceDB. And they miss whether a dealer refining results from a previous turn gets the right response.

Evaluation dimension

Why it matters for agents

Task completion

Agents run multi-step workflows where partial completion is common

Tool use correctness

Wrong tools or bad parameters can derail entire workflows

Reasoning coherence

Flawed reasoning leads to unpredictable failures as conditions change

Reliability and consistency

Non-determinism means the same input can produce different outcomes

Safety and compliance

Autonomous agents can take actions with real-world consequences

Cost and efficiency

An agent requiring 50 API calls per task may not be economically viable

A precise query like “Volkswagen Golf 7-12 years old” might work perfectly, but a colloquial variant like “I’m looking for an older VW” could fail if the semantic search layer isn’t properly evaluated.

Catch issues before deployment with strands-agents-evals

The blueprint implements evaluation across two phases that map to the GenAIOps lifecycle. Build-time evaluation catches issues before deployment, and production evaluation catches what synthetic tests miss. The following diagram (Figure 2) shows how the tool usage, reasoning, and output quality layers must pass before deployment. The framework evaluates agents across three layers:

• Layer 1 (Tool Usage) validates correct tool selection and parameter passing with a greater than 95 percent threshold.

• Layer 2 (Reasoning) assesses logical decision-making with a greater than 85 percent threshold.

• Layer 3 (Output Quality) measures response helpfulness and accuracy with a greater than 90 percent threshold. All three layers must pass before deployment proceeds.

During development and continuous integration and continuous deployment (CI/CD), the pipeline uses the strands-agents-evals framework to catch issues before deployment. It provides output validation, trajectory evaluation, multi-turn conversation simulation, and automated experiment generation. Each is designed to work natively with agents built on the Strands Agents SDK. The framework provides three primitives:

• Experiment: A collection of test cases run against the agent.

• Case: Input query, expected output, and expected tool trajectory.

• Evaluator: Scoring logic (deterministic or LLM-based).

Structure your tests into layers. Layer 1 runs deterministic code-based graders for tool selection accuracy. Layers 2 and 3 use LLM-as-judge evaluators (using an LLM to score agent outputs) for reasoning and output quality.

Custom Evaluator subclasses handle domain-specific concerns. For the Motorway agent, these cover data freshness, dealer scoping, and safety guardrails. Your agent will have its own domain constraints.

Three types of graders

The evaluation framework uses three grader types, each suited to different evaluation needs.

Grader type

Layer

What it measures

Trade-off

Code-based deterministic

Layer 1

Tool selection, parameter passing, trajectory ordering

Fast, cheap, reproducible

LLM-as-judge (Claude Sonnet 4.6)

Layers 2–3

Reasoning quality, output helpfulness, goal success

Flexible; non-deterministic (controlled via pass^k)

Human review

Calibration

Edge cases and safety

Expensive; used to calibrate LLM judge prompts

In practice, grading what the agent produced catches more issues than grading the path it took. You care that the user got relevant results, not which tool the agent called first.

Three-layer assessment framework

Build-time evaluation operates across three distinct layers, each with specific pass/fail thresholds.

Layer 1: Tool Usage (> 95 percent threshold). Did the agent call the right tools with correct parameters?

• “Diesel vehicles from £7,000 to £20,000” should use search_vehicles with typed filters (fuel_type=diesel, min_price=7000, max_price=20000).

• “Modern hatchback with low mileage” should trigger hybrid_search, combining semantic embeddings with structured filters.

You measure this deterministically: ToolSelectionGrader checks which tools were called, TrajectoryOrderGrader verifies the call sequence.

Layer 2: Reasoning (> 85 percent threshold). Was the decision-making process logical? The HelpfulnessEvaluator and TrajectoryEvaluator from strands-agents-evals use LLM-as-judge scoring to assess whether the agent’s reasoning holds together. An agent that arrives at the right response through illogical reasoning will fail unpredictably as conditions change.

Layer 3: Output Quality (> 90 percent threshold). Was the response helpful, accurate, and actionable? The OutputEvaluator and GoalSuccessRateEvaluator from strands-agents-evals use LLM-as-judge evaluation to assess whether the user got a useful, well-formatted response.

The three layers must pass before deployment. A failure in a layer blocks the pipeline.

Handling non-determinism

Because LLM outputs vary between runs, single-trial results can be misleading. The run_all_layers() function in the companion repository accepts a num_trials parameter to address this. Two metrics from the code generation research community help measure reliability:

• pass@k measures the likelihood of succeeding at least once in k attempts. This metric is useful when finding one correct solution is sufficient.

• pass^k (pass to the power of k) measures the probability of succeeding in k consecutive trials. This metric is useful when users expect reliable behavior every time.

For customer-facing agents, pass^k matters most. An agent with a 75 percent per-trial success rate has only a 42 percent chance of passing three consecutive trials (0.75³). Users expect consistent quality on every interaction.

In the companion code, run_all_layers(task_fn, registry, num_trials=5) runs the evaluation layers with multi-trial support and gates deployment on pass^k. See the full implementation.

Test case management

Test cases are organized by category:

• Happy path: Common queries that should succeed.

• Edge cases: Ambiguous queries, slang, multi-turn refinements.

• Safety/Guardrails: Queries the agent should refuse or redirect.

When production monitoring detects an issue, that interaction becomes a new test case. Motorway’s suite grew from an initial 50 cases to 150 in three months, each grounded in real user behavior. Start with 20 to 50 cases and let production data grow the suite.

Include negative cases where the agent should *not* call certain tools. For example, a profile query should call the profile tool, not the search tool. A structured query should use structured search, not a raw SQL fallback. One-sided evals create one-sided optimization.

Multi-turn conversation testing

Single-turn evaluation misses a critical dimension: conversational coherence. Dealers naturally refine searches across multiple turns:

• Turn 1: “Find me diesel SUVs.”

• Turn 2: “Now show me only the automatics.”

• Turn 3: “What about estates instead?”

The strands-agents-evals framework provides ActorSimulator to generate realistic multi-turn interactions and InteractionsEvaluator to score context retention across turns. Multi-turn tests catch context drift, filter accumulation errors, and pronoun resolution failures that single-turn tests miss.

Monitor production behavior with AgentCore Evaluations

After you deploy your Strands Agent to Amazon Bedrock AgentCore Runtime, AgentCore Evaluations provides continuous monitoring. It integrates with Strands Agents through OpenTelemetry instrumentation (the industry-standard observability framework). Figure 3 illustrates the architecture in production, with observability traces sampled at 1–5 percent and with metrics aggregating to Amazon CloudWatch.

Two monitoring approaches

AgentCore Evaluations offers two complementary modes:

On-demand evaluation analyzes specific agent interactions by selecting spans from Amazon CloudWatch logs. This is useful for debugging issues or validating fixes.

Online evaluation automatically samples live traffic and applies evaluators in the background. Configure a sampling rate (1–5 percent is recommended), select up to 10 evaluators, and let it run.

Built-in and custom evaluators

AgentCore provides pre-configured evaluators for common scenarios. The following table shows the built-in evaluators and what they measure.

Evaluator

Level

What it measures

Builtin.Helpfulness

TRACE

How helpful the agent’s response is (0–1 score across 7 levels)

Builtin.GoalSuccessRate

SESSION

Whether the user’s overall goal was achieved

Builtin.ToolSelection