AWS で現代的なデータメッシュ戦略を用いたエージェント型 AI アプリケーションの構築

When a customer service agent autonomously queries order databases, retrieves return policies, and synthesizes answers, it needs governed access to multiple data sources across your organization. Building agentic AI applications on a modern data mesh requires fine-grained access control enforced at every layer of the data interaction chain. AI agents that autonomously discover database schemas, construct SQL queries, and synthesize data from multiple sources expose governance gaps that the single-checkpoint model built for Retrieval Augmented Generation (RAG) can’t address. Organizations need controls from tool discovery through query execution to response synthesis.

In an earlier post, Build secure RAG applications with AWS serverless data lakes, we showed how to enforce fine-grained access control (FGAC) over RAG by filtering vector search results using metadata such as business domain and security classification. That approach worked because RAG’s data interaction was simple: retrieve chunks from a pre-built vector index, filter by metadata, and present results.

This post shows how to build a governed, serverless data mesh on AWS that provides the secure, scalable data foundation production agentic AI requires. The architecture extends the original with three key changes:

• Replacing Amazon OpenSearch Serverless with Amazon S3 Vectors for cost-optimized knowledge bases, which can reduce vector storage and query costs by up to 90% compared to specialized vector database solutions in moderate query-frequency workloads.

• Replacing general-purpose Amazon Simple Storage Service (Amazon S3) with Amazon S3 Tables (with built-in Apache Iceberg support) governed by AWS Lake Formation, delivering up to 10 times higher transactions per second compared to self-managed Iceberg tables, with fine-grained row, column, and cell-level security.

• Exposing the data mesh as Model Context Protocol (MCP) tools through AgentCore Gateway with AWS Lambda-backed interceptors for deterministic access control at every agent-to-tool invocation.

Prerequisites

To implement this architecture, you need the following:

• An AWS account with administrator access.

• AWS Identity and Access Management (IAM) permissions to create roles, policies, Lambda functions, S3 Tables table buckets, Amazon Athena workgroups, and Lake Formation configurations.

• Familiarity with AWS Lake Formation concepts (data lake administrator, LF-Tags, data filters).

• An Amazon Bedrock enabled account with model access configured.

• Amazon Bedrock AgentCore access configured in your account.

• The AWS Command Line Interface (AWS CLI) v2 installed and configured.

Architecture overview

The following diagram illustrates the end-to-end flow from customer request through governed data access and back. Each layer enforces its own authorization controls, so no single point of failure can expose unauthorized data. The architecture diagram shows four layers: Agent Layer with AgentCore Runtime and LangGraph agent, Gateway Layer with request and response interceptors, Tools Layer with four Lambda-backed MCP tools (get_user_tables, get_schema, run_query, kb_search), and Governed Data Mesh with S3 Tables, Athena, Lake Formation, and S3 Vectors. The arrows show data flow from customer through agent to governed data sources.

• Agent Layer – The customer interacts with AgentCore Runtime, a secure, serverless hosting environment that deploys agents in isolated microVM environments with session isolation. The agent runs within the LangGraph framework, which integrates with MCP tools through the MCPClient class.

• Gateway Layer – The Gateway includes a request interceptor that performs JSON Web Token (JWT) validation and scope enforcement, a response interceptor that handles tool filtering, data redaction, and audit logging, and AgentCore Policy with Bedrock Guardrails that evaluates inputs and outputs of every tool invocation for prompt injection, harmful content, and sensitive information exposure in real time.

• Tools Layer – Four Lambda-backed MCP tools (get_user_tables, get_schema, run_query, and kb_search) provide governed data access.

• Governed Data Mesh – S3 Tables (Iceberg) registered in the AWS Glue Data Catalog (s3tablescatalog), Amazon Athena with workgroup cost controls, Lake Formation enforcing row/column/cell-level security, and S3 Vectors powering the Amazon Bedrock Knowledge Bases.

Why agentic AI requires a new governance model

The RAG architecture enforced governance at a single checkpoint: metadata-filtered vector retrieval. That approach served RAG workloads well. Agentic patterns introduce additional steps, creating a multi-step chain where each step requires its own authorization decision. In RAG, the system queries one pre-built vector index with metadata filters at retrieval time. In agentic AI, the system discovers which tables exist, understands schemas, constructs SQL, retrieves from vector stores, and synthesizes results.

A metadata filter at a single retrieval boundary cannot govern this five-step chain. Vector databases synchronize permissions periodically, meaning revocations aren’t immediately reflected. This is an unacceptable gap when an agent is autonomously acting on data. Complex identity permissions such as role hierarchies, attribute-based access, and row-level filters can’t be expressed as straightforward metadata key-value pairs on vector chunks.

These limitations motivate the shift to a governed data mesh architecture where authorization is enforced natively at each data access layer.

Building a governed serverless data mesh

A data mesh decentralizes data ownership to domain teams while centralizing governance and discoverability. On AWS, domain teams own their data products end-to-end, the AWS Glue Data Catalog provides centralized metadata discovery, and Lake Formation enforces permissions with grant/revoke semantics across databases, tables, columns, rows, and cells.

Each producer domain resides in its own AWS account. Producers register data products in a central governance account, a dedicated AWS account that hosts the authoritative AWS Glue Data Catalog and Lake Formation permission policies for the entire organization. Data is shared through Lake Formation cross-account sharing. No data is copied. Only metadata is linked through resource links in consumer catalogs. At query time, Lake Formation verifies permissions and issues temporary credentials to the query engine. Tag-based access control (LF-TBAC) scales this dynamically. Administrators assign LF-Tags like classification=PII or department=customer_service to resources and grant permissions based on those tags.

The following subsections describe how we implement the two data layers of this mesh. First, we cover transactional Iceberg tables for structured data (order records, customer profiles) governed by Lake Formation row and column security. Then, we describe the vector store for unstructured knowledge (policies, FAQs) that powers semantic search.

S3 Tables with Apache Iceberg for transactional data

For our customer service agent, the Order Management domain team publishes order and customer data using Amazon S3 Tables. S3 Tables is the first cloud object store with built-in Apache Iceberg support, delivering up to 10 times higher transactions per second compared to self-managed Iceberg tables on general-purpose S3 buckets. It automatically handles compaction, snapshot management, and unreferenced file removal.

S3 Tables integrates with Amazon SageMaker Lakehouse, which populates the AWS Glue Data Catalog and federates access through Lake Formation. The three data products (customer_orders, customer_profiles, and interaction_history) are queryable from Amazon Athena, governed by Lake Formation permissions, and automatically compacted by S3 Tables.

Lake Formation data filters enforce row-level security so the agent can only access records belonging to the authenticated customer. A data filter on customer_orders with the row filter expression customer_id = :customer_id restricts every query to the current customer’s records, regardless of how the agent constructs its SQL. The run_query Lambda function injects the authenticated customer’s identity as a session parameter before submitting queries to Athena. Column-level security hides sensitive fields like payment_method and billing_address from query results entirely.

Building a knowledge base with Amazon S3 Vectors

Structured data alone is not enough. Customers need answers drawn from unstructured knowledge (product manuals, return policies, frequently asked questions (FAQs), and troubleshooting guides) that require semantic search capabilities.

Amazon S3 Vectors provides native vector storage and querying support as a fully serverless service. It supports up to 2 billion vectors per index and provides strong write consistency, meaning newly added vectors are immediately queryable.

Cost advantages of S3 Vectors

Customers who use the knowledge base feature in Amazon Bedrock can select S3 Vectors as a vector store, which can reduce costs by up to 90 percent compared to specialized vector database solutions in moderate query-frequency workloads. For high queries-per-second (QPS) workloads requiring single-digit millisecond latency, Amazon OpenSearch Serverless remains the better fit. AWS provides single-step export from S3 Vectors to Amazon OpenSearch Serverless collections for workloads that outgrow the S3 Vectors performance profile.

S3 Vectors supports filterable metadata (string, number, boolean, list types with operators like $eq, $ne, $gt, $in, $and, $or) and non-filterable metadata for larger contextual data returned with results. In our use case, documents are stored with filterable metadata keys like product_category and document_type, which supports targeted semantic search. The following example shows a metadata filter that retrieves only electronics return policies:

{"$and": [{"product_category": {"$eq": "electronics"}}, {"document_type": {"$eq": "return_policy"}}]}

Exposing the data mesh with AgentCore Gateway

With the governed data mesh and knowledge base in place, the next challenge is exposing these capabilities to the AI agent in a secure, discoverable, and standardized way. This section covers the tools, interceptors, and identity propagation patterns that make this possible.

AgentCore Gateway provides a centralized layer for managing how AI agents connect to tools. It consolidates authentication, observability, and policy enforcement into a single endpoint. The Gateway converts Lambda functions, APIs, and existing MCP servers into MCP-compatible tools with protocol translation, inbound OAuth authorization, and outbound credential management. Agents connect through streamable HTTP transport with an OAuth Bearer token.

Four core data tools

Four Lambda-backed MCP tools provide governed data access through the Gateway. get_user_tables queries the AWS Glue Data Catalog filtered by Lake Formation permissions to return authorized tables. get_schema retrieves column names, types, and descriptions for a specified table. run_query validates SQL against a read-only allowlist, injects customer identity for row-level filtering, and executes through Athena with byte-scan cost limits. kb_search performs metadata-filtered semantic search against the Knowledge Bases in Amazon Bedrock.

With the launch of Amazon Bedrock Managed Knowledge Base, knowledge bases are now available as a native pre-built target type in AgentCore Gateway. This means you can expose a knowledge base through the Gateway without a custom Lambda function thee Gateway automatically generates IAM roles, provides built-in observability and evaluation metrics, and enforces policies via AgentCore Policy. In this architecture, we use a custom Lambda-backed kb_search tool to demonstrate how Gateway interceptors enforce fine-grained authorization and metadata filtering at the tool invocation boundary. For production workloads where custom interceptor logic is not required, the native Managed KB target type reduces operational overhead by eliminating the Lambda function entirely while retaining MCP compatibility and AgentCore Policy enforcement.

The following JSON shows the tool schema registration for run_query.

{
  "name": "run_query",
  "description": "Executes a read-only SQL query against governed Iceberg tables via Amazon Athena with byte-scan limits and Lake Formation row-level security.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "sql": {"type": "string", "description": "A read-only SQL SELECT statement."},
      "database": {"type": "string", "description": "The Glue Data Catalog database name."}
    },
    "required": ["sql", "database"]
  }
}

Deploying the MCP tools and interceptors:

• Clone the AgentCore Gateway interceptor samples repository.

• For each Lambda function (get_user_tables, get_schema, run_query, kb_search, request interceptor, response interceptor), create the function using the AWS CLI:

aws lambda create-function --function-name get_user_tables \
    --runtime python3.12 --handler lambda_function.lambda_handler \
    --role arn:aws:iam::ACCOUNT_ID:role/mcp-tool-role \
    --zip-file fileb://function.zip

• Attach the IAM policies defined in the repository’s policies/ directory to each function’s execution role.

• Register the Lambda functions as MCP tool targets in AgentCore Gateway. For instructions, see Registering tool targets.

• Attach the request and response interceptors to the gateway. For instructions, see the AgentCore Gateway interceptor samples.

Note: For complete Lambda function source code, IAM policies, and deployment instructions for all four MCP tools and both interceptors, see the AgentCore Gateway interceptor samples.

Interceptors for deterministic access control

AgentCore Gateway interceptors are custom Lambda functions that enforce authorization at two stages in the request-response lifecycle. Interceptors act as middleware that inspects, transforms, or blocks requests and responses flowing through the Gateway. A request interceptor executes before the Gateway calls the target Lambda, and a response interceptor executes after the target responds but before results reach the caller.

Interceptor patterns solve distinct security challenges at each stage:

Pattern

Challenge solved

How

JWT scope-based tool invocation control

Unauthorized tool access

Request interceptor decodes JWT scope claim and blocks unauthorized tools/call invocations

Dynamic tool filtering

Tool discovery leakage