Arize AX、プロダクションAIエージェントのデバッグにSignal機能を追加

Signal is a managed agent built into Arize AX that reviews your production traces on a recurring schedule, identifies recurring failure patterns, and turns them into prioritized issues with supporting evidence, a likely cause, and recommended next steps.

Self-improving agents do not rewrite and deploy themselves unchecked. They operate within a controlled loop: production behavior becomes evidence, evidence becomes an investigation, and proposed changes are tested before an engineer decides what ships.

Signal is the investigation layer in that loop. Built into Arize AX, it continuously reviews production traces, groups recurring failures into prioritized issues, and surfaces the supporting evidence, likely cause, and recommended next step. With repository access, Managed Agents can carry the investigation into the codebase and propose a pull request. Datasets, evaluations, and experiments then determine whether the change actually improved the agent.

Availability: Signal issue detection is available across Arize AX plans. Repository-backed investigation, pull-request creation, Agent Studio, and broader managed-agent workflows are Enterprise capabilities. For more information, check out the launch blog post.

TL;DR: What Signal does

Signal reviews production traces on a recurring schedule and groups related failures into ranked issues.

Each issue includes an investigation, supporting traces, and a suggested prompt, code, configuration, or evaluation change.

Teams can connect a GitHub repository so that, when you choose to act on an issue, Signal can investigate the implementation and open a pull request for review

Matching traces can become a regression dataset, while evaluators and experiments determine whether the proposed fix actually improved the agent.

Developers remain responsible for reviewing and merging changes. Managed agents do not deploy directly to production.

Why production AI agents are difficult to debug

Coding agents have compressed the time required to implement a change. The difficult part increasingly happens before the code is written: finding the relevant production behavior, recognizing the recurring failure, understanding its likely cause, and deciding what should change. Agent failures also rarely look like conventional software failures. Many of these patterns map to common AI agent failures that only become visible in traces. A request can return successfully while the agent:

Routes to the wrong worker

Calls the wrong tool

Passes malformed arguments

Retrieves an empty or irrelevant result

Repeats a planning step without making progress

Falls back to model memory without telling the user

Produces a plausible answer through an unreliable trajectory

The codebase defines the paths an agent can take. The trace records the path it actually took. That makes traces the source of truth for agent behavior. But traces alone do not solve the scale problem. A developer can inspect one failed run manually. Reviewing thousands or millions of runs for related semantic failures is a different task. Evaluations help identify which runs violated an application’s quality criteria. They might show, for example, that hundreds of runs selected the wrong agent or answered without grounded evidence. But a failed evaluation does not necessarily explain whether those runs share one routing defect, several unrelated tool failures, or a problem with session state. Signal performs that investigation continuously.

Tutorial: How to use Signal to debug a production AI agent

Step 1: Instrument the full agent execution path

Signal depends on production evidence, so the first requirement is adding instrumentation to your agent so that the agent telemetry can flow into Arize AX. A useful trace should capture more than the final model response. It should make the agent’s trajectory legible, including:

User input and final output

LLM calls

Latency and token counts

Tool names, inputs, and outputs

Retrieval operations

Agent or node transitions

Exceptions and error states

Retries and repeated steps

Session or user metadata required for segmentation

Arize supports instrumentation through a coding agent, as well as manual instrumentation. For the coding-agent workflow, install the Arize instrumentation skill:

Copy Code

Copied

Use a different Browser

npx skills add Arize-ai/arize-skills –skill “arize-instrumentation” –yes

The coding agent can inspect the application, add the relevant OpenInference instrumentation, and help verify that traces are reaching the project. Before enabling Signal, inspect at least one representative trace manually. Confirm that a developer could answer these questions from the trace alone:

Which route did the agent select?

Which tools did it call?

What did each tool return?

Where did the trajectory diverge from the expected behavior?

What did the user ultimately receive?

Missing tool outputs or flattened agent steps will limit the quality of any downstream investigation.

Step 2: Add evaluations for important behavioral contracts

Signal can begin working once traces are present. Evaluations make the evidence more precise by encoding the behaviors your application promises to users. For related guidance on building trustworthy agent scorecards, see AI agent evaluation and how to improve agent skills with tracing and evals. Consider a financial research agent that answers questions involving current market data. One behavioral contract might be:

PASS

FAIL

The agent calls an approved live-data or web-research tool for time-sensitive questions and grounds the answer in the returned content.

The agent answers from model memory, selects the wrong route, or hides a retrieval failure.

That contract could become an agent_selection_correctness, tool_call_required, or groundedness evaluator. Other useful production evaluations include:

Correct agent or tool selection

Successful task completion

Answer groundedness

Citation validity

Maximum retry count

Session coherence

Policy compliance

Format adherence

Choose evaluations that correspond to user-visible or operational contracts. A generic “response quality” score may reveal that something went wrong, while a routing or tool-use evaluator gives the investigation a much sharper starting point.

Step 3: Enable Signal on the tracing project

Once traces are flowing:

Open the relevant tracing project in Arize AX.

Select the Signal tab from the project navigation.

Click Enable Signal.

https://arize.com/wp-content/uploads/2026/08/enable-signal-1.mp4

Enabling Signal creates a recurring automation for that project. From Configure Signal, teams can adjust the cadence and provide additional context for the investigation. Signal reads traces from the selected project and presents its findings in the same project-level view. No repository connection is required at this stage. A useful adoption path is to begin with trace-only investigations, review the quality of the clusters, and add repository access after the team trusts the evidence.

Step 4: Read each Signal issue as an incident report

The Signal view shows issue volume over time and a ranked list of detected patterns. Opening an issue reveals the investigation and its supporting traces. A useful issue should help you understand:

The recurring behavior: What is the agent doing incorrectly?

The affected population: Which traces or sessions show the same problem?

The likely impact: Does the failure affect correctness, reliability, cost, latency, or user experience?

The evidence: Which spans support the diagnosis?

The proposed intervention: Should the team change a prompt, tool definition, state transition, evaluator, or code path?

Treat severity as a triage signal rather than an automatic merge decision. Frequency matters, although volume alone cannot capture impact. During the webinar, the Arize team showed an internal agent that entered a loop involving repeated completion and to-do updates. The behavior appeared across a relatively small number of traces, but one trace repeated the loop 105 times. That pattern created meaningful cost and reliability risk despite its limited trace count. By comparison, a formatting problem might occur hundreds of times while causing little user harm. Review the issue through three lenses:

How frequently does it happen?

How damaging is each occurrence?

How widely can the failure spread?

Step 5: Connect the repository and generate a pull request

When an issue is well supported by the traces, the next step is to give Signal enough context to connect the runtime behavior to the implementation. For supported Enterprise workflows:

Open the menu on the Signal tab.

Select Configure Signal.

Enable Attach Repo.

Choose or create a GitHub integration.

Select the repository from your connected GitHub App installation (shown as owner/repo).

Save the configuration.

Signal can then use the repository to investigate the relevant code and open pull requests for proposed fixes. The pull request still requires your review using the same standards you would apply to a human-authored change:

Does the diff address the root cause shown in the traces?

Is the change narrow enough to evaluate?

Does it introduce unrelated refactoring?

Are important invariants enforced in code?

Are prompt instructions being used where deterministic controls would be safer?

Does the change include or suggest an appropriate regression test?

Could it affect latency, token use, or tool-call volume?

Is there a clear rollback path?

Managed workers create artifacts for review. They do not merge or deploy the change on their own.

Step 6: Turn the production failure into a regression dataset

A pull request addresses the current defect. A dataset prevents the team from forgetting the behavior that exposed it. Save representative traces from the issue as dataset examples. Preserve enough metadata to explain both the input and the expected trajectory:

Dataset field

Example

input

The original user request

expected_tool

read_web_page

expected_behavior

Retry another source or report retrieval failure

forbidden_behavior

Answer from model memory

failure_mode

ungrounded_fallback

source_trace_id

Original production trace

category

time_sensitive_research

Include variation within the failure mode. For a retrieval problem, the dataset might contain blocked pages, empty pages, irrelevant results, timeouts, and successful pages. This helps determine whether the fix generalizes beyond the exact trace that inspired it. A useful division of labor is:

Artifact

Purpose

Pull request

Changes the behavior

Dataset

Preserves the evidence

Evaluator

Defines what success means

Experiment

Measures whether the change helped

When a failure is important enough to change production code, it is usually important enough to preserve as a regression case.

Step 7: Compare the baseline and candidate before merging

Run the same dataset against both the current production configuration and the proposed configuration. A basic experiment should compare:

Evaluator pass rate

Task completion

Tool selection

Groundedness

Latency

Token usage

Number of tool calls

New failure modes introduced by the candidate

In Arize AX, agent experiments can run a dataset against a registered agent endpoint and compare multiple runs. The comparison view includes row-level outputs, evaluator deltas, latency, token counts, and differences in tool-call paths when tracing is enabled. The safest sequence is:

Run the dataset against the current production agent.

Run the same dataset against the proposed change.

Compare the evaluator results and operational metrics.

Inspect every regression, even when the aggregate score improves.

Merge only after the candidate passes the team’s release criteria.

A correct result on one previously failing example is weak evidence. Consistent improvement across a representative dataset is much stronger.

Worked example: When a research agent answers from memory

Consider an agent responsible for answering questions about current companies and executives. The intended path is straightforward:

The user asks a time-sensitive question.

The agent calls a web research tool.

The tool retrieves a relevant page.

The agent grounds its answer in the retrieved content.

In one production failure from the webinar, the research tool returned a JavaScript-gated page with little usable text. The agent then skipped another retrieval attempt and answered from model memory. The final answer happened to be factually correct. The trajectory was still unreliable. For a different company, a recent leadership change, or a more consequential question, the same fallback could produce a stale or fabricated answer. An agent-selection evaluator marked the traces as incorrect. Signal then grouped several instances into one recurring issue: the research agent was answering from memory after unsuccessful retrieval. Instead of leaving the team with a long list of failed labels, the issue included matching traces and a proposed change. A candidate policy might look like this:

For time-sensitive questions:

1. Call an approved live-data or web-research tool.
2. When a source is blocked or empty, try an alternative source.
3. Categorize retrieval failures explicitly.
4. Report unsuccessful retrieval rather than guessing.
5. Ground factual claims in content returned by the tools.

Depending on the application, some of those controls may belong in the system prompt. Others deserve code-level enforcement. For example:

A maximum retry count should usually be deterministic.

Required citations can be validated after generation.

A “no answer from memory” policy can be enforced by checking whether an approved tool succeeded.

Retrieval failures can be represented as typed states rather than free-form model decisions.

Signal’s proposed fix should be treated as an engineering hypothesis. The trace evidence, dataset, and experiment determine whether that hypothesis survives contact with the system.

Common Signal use cases for production agents

Signal is especially useful for failures that recur across traces but remain difficult to identify through dashboards alone.

Failure mode

Evidence in the traces

Likely intervention

Wrong agent or tool selection

Similar requests repeatedly route to an inappropriate worker or tool

Improve routing logic, tool descriptions, selection prompts, or deterministic rules

Silent fallback to model memory

A retrieval or tool call fails, followed by an unsupported answer

Add fallback policies, alternate sources, grounding checks, or explicit failure states

Retry and planning loops

Repeated spans, state updates, or tool calls without progress

Add stopping conditions, idempotency controls, state-transition fixes, or budget limits

Cost regressions

A recurring path consumes unusually high tokens or tool calls

Reduce context, cache stable data, change planning behavior, or enforce execution budgets

Latency regressions

Particular tools or trajectories dominate end-to-end time

Parallelize safe operations, add timeouts, shorten retry paths, or change provider strategy

Retrieval degradation

Empty, irrelevant, blocked, or stale retrieval results recur

Improve source selection, reranking, chunking, failure handling, or freshness rules

Session-level incoherence

The agent contradicts prior turns or loses task state

Adjust memory boundaries, state persistence, summarization, or session-level evals

Unknown quality failures

Similar complaints or evaluator failures lack a predefined category

Create a new evaluator and regression dataset from the discovered pattern

Signal can also reveal where the AI trace is only part of the story. An agent may complete its LLM and tool steps correctly while a UI component, downstream service, or application state update fails. For investigations that need additional operational context, managed workers can use configured integrations such as GitHub, Datadog, or custom skills alongside the Arize traces.

When to use Agent Studio instead of Signal

Signal is purpose-built for recurring investigation of production traces. Agent Studio is useful when the team needs a custom worker with a narrower task, a different trigger, or additional systems. Examples include:

A daily cost analysis

A weekly agent health report

Investigation of traces that cross a custom metric threshold

Root-cause analysis triggered by a monitor

Dataset generation from a specific class of failures

A targeted repository fix

Correlation between AI traces and APM data

Teams can start Agent Studio from a template or a blank task, bind the worker to a tracing project, select a preset, attach the required skills, and run it as either a one-off session or an automation. Automations can run on a recurring schedule or a metric threshold. A practical investigation prompt might be: Review production traces from the last 24 hours. Find recurring failure modes affecting at least three sessions. For each cluster:

Cite the supporting trace IDs and provide a reusable filter.

Describe the user or operational impact.

Identify the most likely root cause.

Recommend the smallest safe prompt, code, or configuration change.

Specify the regression examples and evaluators needed to test the change.

Estimate possible effects on quality, latency, token use, and tool-call volume.

Then, rank the clusters by impact, frequency, and cost. Do not open a pull request. Return an investigation report for review. Beginning with a report-only workflow makes it easier to evaluate the worker’s reasoning. Once the clusters and recommendations are consistently useful, the team can connect a repository and allow the workflow to produce candidate branches or pull requests.

How Signal fits into a self-improving agent loop

In practice, a self-improving agent is a closed engineering loop with explicit review gates. That loop is the same shift described in from human-operated agent development to systematic agent improvement, and in the broader AI factory for self-improving agents in Arize AX. Production behavior becomes evidence. Evidence produces a candidate change. Evaluations determine whether that change deserves to ship. Signal provides the investigation layer in this loop: It connects what the agent did in production with what the engineering team should examine next. When managed workers need additional context, they can operate inside an isolated sandbox with access to a bound Arize project, an optional repository, and configured external tools. Their outputs can include investigations, evaluator labels, branches, and pull requests. Production deployment remains outside the worker’s control.

Best practices for getting useful Signal investigations

Trace decisions, not only model calls. A collection of isolated prompts and completions will rarely explain an agent failure. Capture routing, tools, retrieval, state changes, and parent-child relationships.

Write evaluations around behavioral contracts