AWS上のLangSmithを用いたディープエージェントの評価方法

*This post was co-authored with Karan Singh, Head of Partnerships at LangChain*

Validating AI agent behavior before production is one of the hardest problems in applied AI. Agents are non-deterministic, multi-step where errors in early steps can affect downstream results. A single bad tool call can cascade through an entire workflow. LangSmith on AWS gives you the evaluation framework to catch these issues early, track them in production, and continuously improve your agent’s reliability throughout its lifecycle.

This post combines learnings from LangChain’s work on evaluating deep agents and Anthropic’s guide to demystifying evals for AI agents into a practical guide. In this post, you will learn how to: 1) apply five evaluation patterns for deep agents, 2) build offline evaluations using pytest and LangSmith, and 3) configure online monitoring for production. The walkthrough uses a text-to-SQL deep agent with Amazon Bedrock for the full development to production lifecycle.

Amazon Nova 2 Lite is a fast, cost-effective reasoning model available in Amazon Bedrock. It supports extended thinking with configurable budget levels (low, medium, high) and accepts text, image, video, and document inputs with a 1 million-token context window. Nova 2 Lite handles instruction following, function calling, and code generation well, which makes it a good fit for agentic workloads like the text-to-SQL agent in this post.

The structure of an agent evaluation

An evaluation is a test for an AI system: give an AI an input, apply grading logic to its output, and measure success. For a large language model (LLM) call, this is straightforward. For agents, every component becomes more complex.

Key terminology

Before diving into patterns, here are the terms used throughout this post:

• Task: A single test with defined inputs and success criteria. For example, “How many customers are from Canada?” with the expected answer of eight.

• Trial: A single attempt at a task. Because model outputs are non-deterministic, running multiple trials per task produces more reliable results.

• Grader: Logic that scores some aspect of the agent’s performance. A task can have multiple graders, each evaluating a different dimension.

• Transcript: The complete record of a trial, including tool calls, reasoning steps, intermediate results, and interactions. In LangSmith, this is the full trace you can inspect for debugging.

• Outcome: The final state of the environment at the end of a trial. An agent might say “The answer is eight,” but the outcome is whether it actually executed the correct SQL query against the database.

• Evaluation harness: The infrastructure that runs evaluations end-to-end. It provides instructions and tools, runs tasks concurrently, records steps, grades outputs, and aggregates results.

• Evaluation suite: A collection of tasks designed to measure specific capabilities or behaviors.

Why agent evaluations are harder

Three properties make agent evaluation fundamentally different from evaluating straightforward LLM outputs:

• Non-determinism – Agent behavior varies between runs. The same task might succeed 90% of the time and fail 10%. A single pass/fail result doesn’t tell you much. You need multiple trials to estimate actual performance. Two metrics help: pass@k measures the likelihood of at least one success in k attempts, while pass^k measures the probability that all k trials succeed. Use pass@k when one success suffices; use pass^k when consistency matters.

• Error propagation – In a multi-step agent, a mistake in step 3 can cascade through the following steps. A text-to-SQL agent that misidentifies the schema early on will construct an incorrect JOIN, producing wrong results in its final answer. Evaluating only the final output misses where things went wrong.

• Creative solutions – Frontier models sometimes find valid approaches that eval designers didn’t anticipate.

What you can evaluate

For an agent run, there are three categories that you can test:

• Trajectory – The sequence of tools called and the specific arguments that the agent generated. Did it explore the schema? Did it use sql_db_query_checker before executing?

• Final response – The final output returned to the user. Is the answer correct? Is it well formatted?

• Other state: Other artifacts that the agent produced, such as files written, TODO plans created, and intermediate results saved.

Evaluation patterns for AI agents

Agent evaluations typically combine three types of graders, and the key to effective evaluation design is choosing the right mix for your use case.

Code-based graders

Code-based graders use deterministic logic to verify specific conditions: string matching, regex patterns, binary pass/fail tests, static analysis, tool call verification, and transcript analysis (turn counts, token usage).

Strengths – Fast, cheap, objective, reproducible, and straightforward to debug. When you can express success criteria as code, do it.

Weaknesses – Brittle to validate variations that don’t match expected patterns exactly. A query result formatted as “eight customers” compared to “There are eight” might fail a strict string match even though both are correct.

Example: Verifying a tool was called:

# Assert the agent executed a SQL query
tool_names = [tc["name"] for tc in tool_calls]
assert "sql_db_query" in tool_names, "Agent must execute sql_db_query"

Model-based graders (LLM-as-judge)

Model-based graders use another LLM to evaluate the agent’s output. Methods include rubric-based scoring, natural language assertions, pairwise comparison, and multi-judge consensus.

Strengths – Flexible, scalable, captures nuance, and handles open-ended tasks and freeform output where the agent’s answer can take many valid forms.

Weaknesses – Non-deterministic, more expensive than code, and requires calibration with human graders to validate accuracy. Give the judge LLM a way out (for example, “return Unknown if you don’t have enough information”) to avoid hallucinated scores.

Example: Grading a complex analytical answer:

rubric = """Score the agent's answer on these dimensions (0.0 to 1.0):
1. correctness: Does it identify the right top employee? (Jane Peacock)
2. completeness: Does it include revenue broken down by country?
3. clarity: Is the answer well-formatted and easy to understand?
Return JSON: {"correctness": float, "completeness": float, "clarity": float}"""
 
judge_response = model.invoke(rubric.format(answer=answer))
scores = json.loads(judge_response.content)

LangSmith’s Align Evaluator feature walks you through a series of steps to calibrate your LLM-as-a-judge evaluator against human expert feedback. You can use this feature to tune evaluators that run on a dataset foroffline evaluations or foronline evaluations.

Human graders

*Human graders* (subject matter expert review, crowdsourced judgment, spot-check sampling) are often considered the gold standard for subjective quality assessments. Compared to programmatic evaluation options, human graders are expensive and slow, but essential for calibrating your model-based graders. Use them judiciously: calibrate LLM-as-judge rubrics against expert human judgment initially, then use human review periodically to verify that the automated graders haven’t drifted.

Combining graders: the practical recommendation

Use deterministic graders where possible, LLM graders where necessary for nuance, and human graders for calibration. For a text-to-SQL agent, that might look like:

• Code-based – Did the agent call sql_db_query? Does the answer contain “eight”? Were DML statements (INSERT, DELETE) executed?

• LLM-as-judge – For complex queries where the output format varies. Is the analysis correct, complete, and well structured?

• Human – Periodic spot-checks to verify LLM grading aligns with expert judgment.

Capability vs. regression evaluations

Not all evaluations serve the same purpose:

• Capability evaluation ask “what can this agent do well?” They should target tasks the agent currently struggles with, giving teams a hill to climb. Start with a low pass rate and work upward.

• Regression evaluation ask “does the agent still handle what it used to?” They should have a nearly 100% pass rate. A decline signals something is broken.

As your agent matures, capability evaluations that reach high pass rates can *graduate* into your regression suite. Tasks that once measured “can it do this at all?” then measure “can it still do this reliably?”

Evaluating deep agents

*Deep agents* (systems that use planning, tool use, filesystem backends, and progressive context loading to tackle complex, multi-step tasks) break the traditional assumption that every test case can be run through the same application logic and scored by the same evaluator. Over the past several months, LangChain shipped four applications on top of deep agent architectures and identified four patterns that apply broadly.

Pattern 1: Custom test logic per datapoint

Traditional LLM evaluation treats every datapoint identically: run through the same application, score with the same evaluator. Deep agents break this assumption. Each test case may have its own success criteria, and those criteria might involve specific assertions against the agent’s trajectory and state, not just the final message.

Consider a text-to-SQL agent. “How many customers are from Canada?” has a single correct answer (eight) that you can check with a string match. But “Which employee generated the most revenue and from which countries?” requires an LLM judge to evaluate correctness, completeness, and clarity, because the format of a valid answer varies widely.

LangSmith’s Pytest integration supports this pattern. You can make different assertions about the agent’s trajectory, final message, and state for each test case:

@pytest.mark.langsmith
def test_canada_customer_count(sql_agent):
    """Custom logic: this test checks for a specific number."""
    result = sql_agent.invoke({
        "messages": [{"role": "user", "content": "How many customers are from Canada?"}]
    })
    answer = result["messages"][-1].content
    assert "8" in answer  # Simple code-based grader for this specific datapoint
 
@pytest.mark.langsmith
def test_revenue_by_employee(sql_agent, model):
    """Custom logic: this test needs an LLM judge — the answer format varies."""
    result = sql_agent.invoke({
        "messages": [{"role": "user", "content": "Which employee generated the most revenue?"}]
    })
    scores = llm_judge(model, result["messages"][-1].content)
    assert scores["correctness"] >= 0.5

Pattern 2: Single-step evaluations

About half of LangChain’s test cases for deep agents were single-step evaluations: what did the agent decide to do immediately after a specific input? This is especially useful for validating individual decision points. Did it call the right tool with the right arguments?

Regressions often occur at individual decision points rather than across full execution sequences. For a text-to-SQL agent, a single-step eval might verify that the agent’s first action is to explore the database schema (calling sql_db_list_tables or sql_db_schema), rather than jumping straight to writing a query.

@pytest.mark.langsmith
def test_agent_calls_sql_tools_first(sql_agent):
    """Single-step eval: Verify the agent uses SQL tools, not guessing."""
    result = sql_agent.invoke({
        "messages": [{"role": "user", "content": "How many customers are from Canada?"}]
    })
 
    tool_calls = extract_tool_calls(result["messages"])
    tool_names = [tc["name"] for tc in tool_calls]
 
    sql_tools = {"sql_db_list_tables", "sql_db_schema", "sql_db_query", "sql_db_query_checker"}
    assert sql_tools & set(tool_names), "Agent must use SQL tools"

Single-step evaluations are your unit tests. Fast, focused, and efficient on tokens.

Pattern 3: Full agent turns

While single-step evaluations test individual decisions, full agent turns show you the complete picture. Run the agent end-to-end on a single input and evaluate:

@pytest.mark.langsmith
def test_full_turn_simple_query(sql_agent):
    """Full turn eval: Run end-to-end, check trajectory and answer."""
    result = sql_agent.invoke({
        "messages": [{"role": "user", "content": "How many customers are from Canada?"}]
    })
 
    # Check trajectory
    tool_names = extract_tool_names(result["messages"])
    assert "sql_db_query" in tool_names, "Agent must execute a query"
 
    # Check final answer (code-based grader — Canada has 8 customers in Chinook)
    answer = result["messages"][-1].content
    assert "8" in answer, "Answer must contain the correct count"

Key insight: This test asserts that certain tools appeared in the trajectory, but doesn’t assert the exact order. The agent might list tables before getting the schema, or go directly to the schema. Both are valid. Grade *what the agent produced*, not the exact *path it took*.

LangSmith displays the complete trace for full agent turns. You can see the planning steps (write_todos), each SQL tool invocation, the actual queries executed, and the final formatted answer.

Pattern 4: Multi-turn evaluations

Some scenarios require testing agents across multi-turn conversations. A user asks “What are the top 5 best-selling artists?” then follows up with “For the top artist, how many albums do they have?” The challenge: if you hardcode a sequence of inputs and the agent deviates from the expected path, the subsequent inputs might not make sense.The solution is conditional logic in your tests:

@pytest.mark.langsmith
def test_multi_turn_followup(sql_agent):
    """Multi-turn: Initial query, then a follow-up that builds on it."""
    # Turn 1
    result1 = sql_agent.invoke({
        "messages": [{"role": "user", "content": "What are the top 5 best-selling artists?"}]
    })
    answer1 = result1["messages"][-1].content
 
    # Conditional: if turn 1 failed, fail early
    if not answer1 or len(answer1)  20, "Follow-up must produce a meaningful answer"

If you want to test turn 2 in isolation, set up a test starting from that point with the expected turn 1 output as initial state.

End-to-end example: Evaluating a text-to-SQL deep agent on AWS

Now put these patterns into practice. The example uses LangChain’s text-to-SQL deep agent example, configure it to run on Amazon Bedrock, and build evaluations using LangSmith.

Architecture overview

The text-to-SQL deep agent is built on the DeepAgents framework, which provides planning, filesystem storage, and progressive context loading on top of LangGraph. It answers natural language questions about the Chinook database, a sample SQLite database representing a digital media store.

*Figure 1: Text-to-SQL Deep Agent architecture*

Prerequisites

You must have the following prerequisites to follow along with this post.

• AWS account with Amazon Bedrock access enabled

• LangSmith account and API key

• Python 3.12+

• AWS Command Line Interface (AWS CLI) configured with credentials

• Required packages: deepagents, langchain-aws, langchain-community, pytest

Setup

Clone the companion repository and install the dependencies:

git clone https://github.com/aws-samples/sample-text2sql-deep-agent-evalulation
cd langsmith-deep-agents-eval
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e .

The text-to-SQL agent uses Amazon Nova 2 Lite on Amazon Bedrock using ChatBedrockConverse:

from langchain_aws import ChatBedrockConverse
model = ChatBedrockConverse(
    model="global.amazon.nova-2-lite-v1:0",
    region_name=os.getenv("AWS_REGION", "us-east-1"),
    temperature=0,
)

The .env configuration is minimal:

<pre cl