Strands Evals を用いた AI エージェントの失敗検出と根本原因分析

When your AI agent fails in production, knowing *that* it failed is only the beginning. The harder question is *why* it failed and what to fix. Traditional evaluation tells you “this agent scored 60 percent on goal completion,” but leaves you manually reviewing execution traces to understand what went wrong. For teams operating agents at scale, this manual diagnosis becomes the bottleneck between detecting a problem and shipping a fix. Detectors in the Strands Evals SDK remove this bottleneck by automatically identifying failures in agent execution traces and performing root cause analysis, so you can reduce diagnosis time from hours to minutes.

In this post, we walk you through calling the detector functions to diagnose real agent failures. You learn how to interpret their structured output: categorized failures with confidence scores, causal chains linking root causes to downstream symptoms, and fix recommendations specifying whether a change belongs in your system prompt or tool definitions. You also learn how to integrate detection into your evaluation pipeline for automated diagnosis on every test run.

Detectors complement the evaluation framework introduced in a previous post by answering not only “how well did the agent do?” but also “why did it fail and how do I fix it?”

Prerequisites

You must have the following prerequisites to follow along with this post.

• Python 3.10 or later.

• Strands Evals SDK installed with pip install strands-agents-evals.

• Amazon Bedrock model access enabled (detectors use large language model (LLM)-based analysis).

• For Amazon CloudWatch examples, AWS credentials configured with logs:StartQuery and logs:GetQueryResults permissions.

Why scores alone are not enough

The Strands Evals framework provides reliable quality signals through Cases, Experiments, and Evaluators: goal success rates, tool selection accuracy, and helpfulness scores. These are important for catching regressions and understanding performance at a statistical level. But consider what happens after you detect a regression. Your agent’s goal success rate drops from 85 percent to 70 percent after a deployment or after prompt or tool changes in build-time testing. Evaluators confirm the drop. Now what?

You must identify which specific behaviors caused failures, distinguish root causes from downstream symptoms, determine whether the fix belongs in the system prompt or tool definitions, and prioritize by impact. This diagnosis workflow has traditionally required senior engineers to manually inspect traces span by span and correlate failures across hundreds of steps, and this process doesn’t scale.

Detectors automate this workflow. Evaluators answer “how well did the agent do?” by producing scores at the per-case level. Detectors answer “why did it fail?” by producing diagnoses at the per-span level with categorized failures, causal chains, and fix recommendations.

How detectors work

The detector pipeline operates in two phases, each powered by LLM-based analysis of the execution trace. Refer to Understand observability for agentic resources in Amazon Bedrock AgentCore to learn more about sessions, traces, and spans of agents.

Phase 1: Failure detection scans each span in a session against a comprehensive failure taxonomy organized into nine parent categories: hallucination, incorrect actions, orchestration errors, task instruction non-compliance, execution errors, context handling errors, repetitive behavior, LLM output issues, and configuration mismatch. For each identified failure, it returns the span location, one or more categories, a confidence score, and evidence extracted from the trace.

Phase 2: Root cause analysis takes the detected failures and traces causal chains between them. A single upstream mistake often cascades into multiple downstream failures. Root cause analysis separates causes from symptoms. It classifies each failure’s causality (PRIMARY, SECONDARY, or TERTIARY), determines propagation impact, and generates fix recommendations categorized by where the fix belongs (system prompt, tool description, or other).

Both phases handle sessions of varying sizes through a tiered strategy: direct analysis for sessions that fit within the context window of the selected Detector model, failure path pruning that retains only ancestor and descendant spans for moderately large sessions, and chunked analysis with merge for very large sessions that splits the trace into overlapping windows and reconciles results.

The following diagram shows the end-to-end pipeline with two entry points converging into the same detection and analysis flow.

*Figure: Detector pipeline with integrated and standalone entry points flowing into failure detection and root cause analysis.*

Getting started with failure detection

The following examples use a session trace from the drug discovery research assistant featured in Evaluating AI agents for production: A practical guide to Strands Evals. The agent is built on Strands Agents and Amazon Bedrock. To follow along, run your agent with OpenTelemetry tracing enabled and export the session as JSON, or use the CloudWatchProvider shown later in this post to fetch an existing trace. Refer to User Simulation in the Strands Agents SDK documentation for how to set up tracing and export sessions.

The detect_failures function takes a Session object (the standard trace format in Strands Evals) and returns structured failures. Each failure includes the span where it occurred, one or more categories from the pre-defined failure taxonomy, a confidence score, and evidence extracted from the trace.

import json
from strands_evals.detectors import detect_failures
from strands_evals.types.trace import Session
from strands_evals.detectors import ConfidenceLevel

with open("agent_trace.json") as f:
    session = Session.model_validate_json(f.read())

result = detect_failures(session, confidence_threshold=ConfidenceLevel.MEDIUM)

for failure in result.failures:
    for cat, conf, ev in zip(failure.category, failure.confidence, failure.evidence):
        print(f"[{conf}] {cat} at span {failure.span_id}")
        print(f"  Evidence: {ev}")

The following is output from a research agent that was asked to “Research the impact of energy requirements for powering AI in the real world.” The agent encountered tool configuration issues and progressively degraded:

[0.9] execution-error-category-tool-schema at span f503a7d546fa4157
  Evidence: Tool execution failed due to missing required parameter
  'knowledgeBaseId'. Error: 'Parameter validation failed: Invalid type
  for parameter knowledgeBaseId, value: None'

[0.75] hallucination-category-hall-usage at span 0466979670d14099
  Evidence: Agent claims 'I don't have access to the specific knowledge
  base needed' and then proceeds to provide detailed information about AI
  energy requirements 'based on general knowledge' without using any tools.

[0.9] orchestration-related-errors-category-goal-deviation at span d98d578e61233d33
  Evidence: Agent completely abandons the original task about AI energy
  requirements and instead provides a lengthy response about marine
  biology, stating 'I'm going to pivot to discuss marine biology instead.'

In a single pass, the detector identifies failures at multiple levels: execution errors (tool parameter validation), semantic issues (hallucinating from “general knowledge”), and orchestration problems (full goal deviation). A single span can exhibit multiple failure categories, each with independent confidence and evidence.

Adding root cause analysis

Identifying failures is useful, but understanding why they happened is what drives fixes. The analyze_root_cause function takes detected failures and traces causal chains between them, separating root causes from downstream symptoms and recommending where each fix belongs. If failures aren’t provided to analyze_root_cause, it runs failure detection automatically.

from strands_evals.detectors import detect_failures, analyze_root_cause

failures = detect_failures(session)
rca_result = analyze_root_cause(session, failures=failures.failures)

for rc in rca_result.root_causes:
    print(f"Causality: {rc.causality}")
    print(f"  Span: {rc.failure_span_id} | Fix type: {rc.fix_type}")
    print(f"  Root cause: {rc.root_cause_explanation}")
    print(f"  Recommendation: {rc.fix_recommendation}")

Continuing with the same research agent session, root cause analysis reveals the causal structure:

Causality: PRIMARY_FAILURE
  Span: f503a7d546fa4157 | Fix type: TOOL_DESCRIPTION_FIX
  Root cause: Agent called retrieve tool without required knowledgeBaseId
    parameter because tool description does not clearly document that
    knowledgeBaseId is mandatory. This caused parameter validation failure
    and forced agent into multiple retry attempts with different parameter
    combinations.
  Recommendation: Update retrieve tool description to explicitly mark
    knowledgeBaseId as a required parameter with clear documentation
    including format constraints and example values.

Causality: SECONDARY_FAILURE
  Span: 0466979670d14099 | Fix type: SYSTEM_PROMPT_FIX
  Root cause: Agent fabricated detailed AI energy consumption information
    claiming it is 'based on general knowledge' after all retrieval attempts
    failed, because system prompt lacks instruction prohibiting generation
    of factual content without tool-retrieved evidence.
  Recommendation: Add instruction to system prompt requiring agent to
    explicitly acknowledge inability to complete research tasks when
    retrieval tools fail, and prohibit generating detailed factual content
    without tool-verified sources.

The distinction between fix types is what makes root cause analysis actionable. The tool schema error is a TOOL_DESCRIPTION_FIX because the retrieve tool’s knowledgeBaseId isn’t documented clearly. The downstream hallucination is a SYSTEM_PROMPT_FIX because of missing instructions for how to handle persistent tool failures. Fixing only one category leaves the other unaddressed.

Integrated diagnosis with diagnose_session

For convenience, diagnose_session runs both phases as a single pipeline (detect failures, then analyze root causes) and returns a unified DiagnosisResult with deduplicated recommendations:

from strands_evals.detectors import diagnose_session, ConfidenceLevel

result = diagnose_session(session, confidence_threshold=ConfidenceLevel.MEDIUM)
print(f"Found {len(result.failures)} failures, {len(result.root_causes)} root causes")

for rec in result.recommendations:
    print(f"  - {rec}")

This produces the same failures and root causes shown in the preceding examples, packaged into a single result with recommendations deduplicated across all root causes. From one function call, you get a prioritized list of concrete changes categorized by where they belong.

Integration with evaluation pipelines

Detectors provide additional value when you integrate them into your existing evaluation workflow. The DiagnosisConfig attaches automated diagnosis to any experiment, so that every failing test case automatically produces a diagnosis:

from strands_evals import Experiment
from strands_evals.evaluators import GoalSuccessRateEvaluator
from strands_evals.detectors import ConfidenceLevel, DiagnosisConfig, DiagnosisTrigger
from strands_evals.types.evaluation_report import EvaluationReport

experiment = Experiment(
    cases=test_cases,
    task_function=my_agent_task,
    evaluators=[GoalSuccessRateEvaluator()],
    diagnosis_config=DiagnosisConfig(
        trigger=DiagnosisTrigger.ON_FAILURE,
        confidence_threshold=ConfidenceLevel.MEDIUM
    ),
)

report = experiment.run()
report.display(include_recommendations=True)

Two trigger modes are available. ON_FAILURE (default) runs diagnosis only when at least one evaluator returns test_pass=False, making it cost-efficient for continuous integration and continuous delivery (CI/CD) regression detection. ALWAYS runs diagnosis on every case regardless of outcome, which is useful for identifying suboptimal paths in nominally passing cases.

With this integration, your CI/CD pipeline tells you “3 tests failed”, and it tells you why they failed and what to change. This closes the feedback loop: define cases, run the experiment, get scores and diagnosis together, apply the recommended fixes, and re-run to confirm.

Note: Running detectors uses Amazon Bedrock inference for LLM-based analysis, which incurs charges. See Amazon Bedrock pricing for details. Amazon CloudWatch Logs storage also incurs charges. See Amazon CloudWatch pricing for details. Monitor your usage in AWS Cost Explorer, especially when integrating detectors into CI/CD pipelines that run frequently.

Diagnosing production sessions from CloudWatch

The preceding examples use local session files, but in production your agent traces live in Amazon CloudWatch Logs, exported with OpenTelemetry. The CloudWatchProvider fetches traces directly from Amazon CloudWatch and converts them into Session objects that you can analyze with detectors:

from strands_evals.providers import CloudWatchProvider
from strands_evals.detectors import diagnose_session, ConfidenceLevel

provider = CloudWatchProvider(agent_name="my-research-agent", region="us-east-1")
data = provider.get_evaluation_data(session_id="abc-123-def-456")
session = data["trajectory"]

result = diagnose_session(session, confidence_threshold=ConfidenceLevel.MEDIUM)

for rc in result.root_causes:
    print(f"[{rc.fix_type}] {rc.fix_recommendation}")

Under the hood, the provider queries Amazon CloudWatch Logs Insights for OTEL records matching the session ID, auto-detects the agent framework (Strands, LangChain, or others) from span metadata, and maps the spans into a standardized Session. Detectors work with any framework that exports OpenTelemetry traces to Amazon CloudWatch, not only Strands Agents.

You can also combine this with the experiment pipeline for offline evaluation: use CloudWatchProvider to evaluate and diagnose historical production sessions without re-running the agent. You can also retrieve traces from Langfuse or OpenSearch using LangfuseProvider or OpenSearchProvider.

Best practices

Start with MEDIUM confidence. The LOW threshold catches more potential issues but includes more noise, which is useful for deep investigation of a specific failing case. MEDIUM provides a good signal-to-noise ratio for routine use. Reserve HIGH for production monitoring where you only want high-certainty findings.

Use ON_FAILURE in CI/CD, ALWAYS for periodic audits. ON_FAILURE keeps LLM costs proportional to failure rates, making it practical for every test run. Schedule ALWAYS-mode runs weekly or per-release to catch suboptimal behaviors hiding in passing cases.

Fix PRIMARY failures first. Secondary and tertiary failures often resolve when their root cause is addressed. Before implementing multiple recommendations, check whether fixing the primary failure removes the downstream ones. This reduces iteration cycles.

Group recommendations by fix type. Batch TOOL_DESCRIPTION_FIX changes together and SYSTEM_PROMPT_FIX changes together. This makes the impact of each change category independently measurable when you re-run evaluation.

Pass pre-detected failures to analyze_root_cause. If you have already run detect_failures and want to inspect the results before running root cause analysis, pass them directly to avoid redundant detection:

failures = detect_failures(session)
# ... inspect or filter failures ...
rca = analyze_root_cause(session, failures=failures.failures)

Use the test session for experimentation. The flawed_session.json used in this post is available in the Strands Evals test suite for you to try detectors locally.

Clean up resources

The detector functions themselves don’t provision any persistent AWS resources. However, if you configured Amazon CloudWatch Logs export for your agent traces, you might want to review the following:

• Amazon CloudWatch log groups: Deleting a log group permanently removes all log data and can’t be undone. Confirm that you have exported any logs you need to retain before proceeding. If you created log groups specifically for testing, delete them through the Amazon CloudWatch console or by running aws logs delete-log-group --log-group-name .

Amazon Bedrock model access: The LLM analysis uses Amazon Bedrock. If you enabled mo