Amazon Bedrock AgentCore を用いた大規模なエージェント AI の運用管理「AgentOps」の紹介

When you build agentic AI solutions, you face unique operational challenges. Agents make unpredictable decisions, costs spiral unexpectedly, and debugging non-deterministic failures seems impossible. Agentic AI applications don’t just execute predetermined workflows. They reason, adapt, and make autonomous decisions, and DevOps practices need to be adapted. That’s where AgentOps comes in, the operational discipline for deploying, managing, and continuously improving AI agents in production.

The first part of our blog series introduced how to operationalize generative AI workloads. In this post, we show how to accelerate the path to production for agentic AI workloads, check the quality of your agents and tools, and drive agentic AI adoption in your organization by implementing AgentOps with Amazon Bedrock AgentCore. We discuss best practices from real world implementations across four pillars: governance and security, build and operations, evaluation, and observability. We also show how AWS services, people, and processes come together into a reference architecture that you can adapt for your organization.

Note that this post focuses on operations and not agent design. The implementation examples use Amazon Bedrock AgentCore and supporting AWS services, but the principles discussed apply broadly. The reference architecture is a starting point: your organization’s requirements will determine how you adapt it.

AgentOps: The four pillars

This post covers best practices and real-world learnings for each of the AgentOps pillars:

• Governance & Security: use multi-account strategy, deterministic controls, reasoning controls and human-in-the-loop, to verify agents operate within authorised boundaries and every action is traceable.

• Build & Operations: treat every agent, tool, and memory configuration as a versioned, deployable artifact with its own CI/CD pipeline.

• Evaluation: evaluate at four levels, tool, conversation turn, session outcome, and system in development and production.

• Observability and monitoring: instrument across four telemetry layers so you can trace every agent decision, monitor quality drops, and track cost per interaction.

Amazon Bedrock AgentCore offers components that you can use independently or together to implement these pillars. It is AWS’s Agentic AI platform for building, deploying, and operating effective agents securely at scale. AgentCore works with any open source framework and any large language model (LLM) and you can transition from local development to production without managing infrastructure.

The AgentOps Lifecycle on AWS

Like other software solutions, agents follow a development lifecycle from idea to production, and that progression never truly ends. Agents require continuous operational attention and improvements across every stage. Below, we’ve mapped out how agentic AI impacts each stage of your DevOps pipeline: Plan, Develop, Build, Test, Deploy & Release, Maintain and Monitor.

DevOps Stage

AgentOps Considerations

Plan

Assess AI fit, risks, ethics. Secure legal/compliance approvals, establish performance metrics, prepare data. Define human oversight point, tool permissions, agent trust model, cross-agent authentication, initial agent design

Develop

Experimentation and model selection, evaluations, Retrieval Augmented Generation (RAG)/prompts, chunking strategies, guardrails. Orchestration, memory, state, tool registry/discovery, Model Context Protocol (MCP) tools, Agent-to-Agent (A2A), agent identity, agent evaluations, auth patterns

Build

Unit/integration/security/agent tests, deploy to pre-production. Workflow tests, tool chain validation. Role-Based Access Control (RBAC) validation

Test & release

Run quality, performance, end-to-end, security tests. Update release notes with AI considerations. Execution path evaluation end-to-end goals, loop limits, human-in-the-loop (HITL) tests, unauthorized agent actions.

Deploy

Deploy solution to production.Deploy MCP servers, tools. Concurrency, least privilege, networking for agent endpoints. Configure rollback strategies, canary deployments, or traffic management

Maintain and monitor

Track quality, guardrails, latency, throughput, responsible AI, errors, track usage and cost. User feedback. Traces/spans monitoring, drift, alerts, action audit trails, anomaly detection, guardrails for agent end-to-end calls

The pillars apply irrespective of where you are in the lifecycle. From a responsible AI perspective, you need systematic risk management throughout. “The Agentic AI Security Scoping Matrix: A framework for securing autonomous AI systems” can help identify and manage risks.

Solution Overview

The following reference architecture shows how the pillars, lifecycle, people, processes, and AWS services connect. Let’s go through it step-by-step.

Planning and setup

• The product owner registers the use case in a centralized catalog. Legal and compliance teams assess risks and provide guidance.

• Once the use case is approved, the product owner works with domain experts and technical teams to establish scope, success metrics, and source-of-truth test prompts for evaluation.

• Platform engineers deploy environments using IaC with access controls agreed with security teams and tagging for governance and cost tracking.

Development

• Developers and data scientists create agent, application, and tool repositories with seed code and begin building. They may use approved tools behind the shared AgentCore Gateway and agents behind the AWS Registry. New tool or MCP server requests go through the product owner, platform team, and legal for approval.

• Data engineers create datasets and evaluation sets for development and testing.

• Developers run manual and automated evaluations including tool selection accuracy, multi-step reasoning validation, conversation coherence, and memory persistence. Domain experts review results and provide feedback.

• Experiment results are tracked locally during development, then synchronized to the shared account for centralized tracking and cross-team comparison.

• Developers merge to main, triggering the deployment pipeline.

Build and deployment pipeline

• The CI/CD pipeline creates a release branch, deploys resources to pre-production including agent deployment to AgentCore Runtime via ECR, and triggers the evaluation pipeline. For RAG implementations, the ingestion pipeline deploys to the data governance account.

• In pre-production, integration, performance, UAT, regression, and generative AI evaluation tests run, including authentication flows, user context propagation, and authorization validation for tool access.

• QA engineers and domain experts validate against established metrics and approve promotion to production.

Production deployment and operations

• The solution is deployed to production. Production telemetry, user feedback, and performance metrics flow back to planning for continuous improvement.

• Agents are registered in the Agent Discovery API, making them discoverable for reuse and agent-to-agent collaboration.

• End users interact with the application and provide feedback. AgentCore Observability dashboards track decision traces, tool invocation patterns, latency, errors, memory usage, and cost per interaction.

Now let’s go through each pillar in more detail.

Pillar 1: Governance & Security

In agentic systems, a single user request can spread across hierarchical chains or trigger collaborative swarms where multiple agents act on the user’s behalf. Each interaction between user and agent needs to be tightly controlled. When Agent A calls Agent B, there can be ambiguity of what agent is authorized to perform which actions. If a user with limited permissions triggers an agent, the agent must inherit those restrictions. This ambiguity only compounds in deeper chain of calls. You need strict governance around who can access the agents, what data and tools and APIs the agents can access, who can authorize these permissions, and what occurs when issues arise.

The following diagram shows the security decisions to be made at each step when an agent handles a request. A user’s input flows through an environment, into the agent, which uses tools and memory to generate outputs. The application verifies the user’s identity, whether they are allowed to invoke the agent, and whether the agent can access the requested context, memory, and tools with the specific parameters. It also validates that inputs are safe and that the agent is authorised to return the specific outputs.

To achieve a layered security approach that helps agents operate within well-defined boundaries while maintaining auditability you should consider the following dimensions.

Multi-account architecture

AgentOps is an extension of GenAIOps, the same way MLOps is an extension of DevOps. If you followed Part 1: GenAIOps, the same design principles apply to AgentOps. You should follow a multi-account strategy for organizational isolation and Service Control Policies (SCPs) to set security guardrails across accounts.

The following reference diagram shows the multi-account AWS architecture:

• A shared services account with Amazon Elastic Container Registry (ECR) container images, pipeline artifacts, AWS Secrets Manager, and centralised monitoring and authentication services.

• Data accounts to separate producer accounts from data governance accounts, supporting isolation and secure access to knowledge bases aligned with compliance requirements.

• Application accounts. a. Dedicated development (dev), b. pre-production (pre-prod), and c. production (prod) accounts per line of business or application team and tagged for governance and cost tracking.

Accounts and resources are deployed and managed using Infrastructure as Code (IaC).

Controlled Model access

When using Amazon Bedrock, you control which models the applications have access to using SCPs and IAM identity-based policies. Your agents can use these models directly or via a generative AI gateway such as LiteLLM. With a gateway, you centralize access control and simplify governance implementation across multiple model providers while providing a unified API interface for rate limiting per user or agent, token budgeting, cost tracking and budget enforcement, model routing based on security policies, and centralized audit trails for compliance. AWS has published guidance on how to deploy a generative AI gateway. We initially placed the gateway in shared services for simplicity, but found it harder to attribute costs to individual agents and moved it to application accounts.

Identity and Access Control

You can use AWS Identity and Access Management (IAM) for fine-grained access control. Additionally, with AgentCore Identity you manage authentication and authorization across your agents, with fine-grained access controls and cross-agent authentication protocols that maintain security boundaries as requests propagate through your system. For more information refer to Amazon Bedrock AgentCore Identity: Securing agentic AI at scale. AWS CloudTrail can be used for comprehensive audit logging and forensic analysis.

Data governance

Data flows through multiple touchpoints: user inputs (text, attachments), agent instructions, outputs, accessed data sources, and memory operations, each presenting potential security risks. Configure Amazon Bedrock Guardrails to evaluate user prompts and model responses against your safety policies and to protect against threats like inadvertent PII disclosure. For detailed set-up instructions to implement guardrails and integrate them with a generative AI gateway refer to Safeguard generative AI applications with Amazon Bedrock Guardrails.

In addition to the above, use version control of evaluation datasets (with a few hundred examples) and systematically track changes to documents and generated embeddings within RAG knowledge bases to support evaluation and auditing requirements.

Memory

In agentic applications, data represents underlying facts, documents, and structured information agents query (knowledge bases, databases, APIs) accessed through retrieval mechanisms like RAG, governed through traditional access controls. On the other hand, memory is the agent’s working context (what it retains about conversations, user preferences, and interaction patterns). It is dynamic and conversational, evolving with each interaction.

With AgentCore Memory you get short-term memory and long-term memory with built-in and custom strategies for memory extraction. You can also override extraction logic or implement self-managed strategies for specialised requirements. Namespaces, which are defined at creation time as part of the strategy configuration in long-term memory, organise memory by actor, session, or strategy. They provide the structure that helps personalisation and shared learning across users. AgentCore Memory scopes data to individual aggregates at actor level. When agents need to learn cross-user patterns, memory can aggregate at higher application-wide levels. In a multi-account deployment pattern, each account (dev, pre-prod, prod) has its own AgentCore Memory resources that teams deploy and manage alongside their applications. This deployment pattern helps with security isolation, independent scaling, alignment with data residency requirements, and cost allocation per application.

Applications can access multiple memory resources. The following diagram illustrates this approach, showing how two applications, a fraud and a claims application, access risk signals and policy details from their dedicated resources, and user details from a shared memory resource. You can control which memory resources and information they have access to with IAM policies.

<img loading="lazy" class="alignnone wp-image-131957 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/05/19/ml-19465-image-3.png" alt="AWS architecture showing how Fraud and Claims agents use separate and shared long-term memory stores wit