Amazon Bedrock、自動生成フィルタで契約検索精度を向上

Enterprises rely on large volumes of complex legal agreements to make critical business decisions — determining rights, renewal options, geographic restrictions, and compliance obligations. In industries like entertainment and media, where organizations manage thousands of contracts across multiple jurisdictions, this work remains largely manual: time-consuming, costly, and difficult to scale.

Our AI-Driven Annotation (AIDA) solution, powered by AWS, helps address this challenge by transforming unstructured contracts into searchable, actionable intelligence. AIDA enables users to ask natural-language questions across large contract repositories — but delivering precise answers requires more than semantic search alone. Legal documents are highly contextual, and retrieval-augmented generation (RAG)-based systems can surface more content than a language model can effectively process. Without careful control over which excerpts are retrieved, and without sufficient document-level context, important clauses risk being overlooked or misinterpreted.

In this post, we describe how AIDA works at a high level and how it helps address these challenges — grounding users in the right contracts, under the right legal context, and within the right access boundaries. Specifically, we explore how AIDA uses implicit and explicit filtering, along with metadata-enriched chunking in Amazon Bedrock Knowledge Bases, to dramatically improve contract search accuracy.

Solution overview

The reference architecture diagram represents how AIDA uses a RAG architecture built on Amazon Bedrock Knowledge Bases to create an intelligent document query system. Data transmission between components uses encryption in transit via HTTPS/TLS 1.2+, including document uploads to Amazon Bedrock Knowledge Bases, API calls to embedding models, vector database queries, and response delivery to client applications. For data stored in Amazon Bedrock, the AWS shared responsibility model applies. The implementation follows a systematic workflow:

Data ingestion flow

Step 1: Document ingestion and metadata configuration

The architecture begins with document ingestion, where contracts are synced into Amazon Bedrock Knowledge Bases alongside structured metadata files. The metadata should contain key contract attributes such as parties, effective date, termination date, jurisdiction, and other key attributes that can enable powerful filtering capabilities downstream.

Step 2: Chunking mechanism implementation

The knowledge base configuration phase implements a sophisticated chunking mechanism that breaks contracts into semantically meaningful segments optimized for retrieval. This chunking strategy helps provide each segment with sufficient context while remaining concise enough for efficient processing. The metadata supplied during ingestion plays a critical role in the system’s ability to apply intelligent filtering, particularly through the implicit filtering mechanism that distinguishes this architecture from standard RAG implementations.

Step 3: Vector database storage

Once documents are ingested and processed, the system converts document chunks into vector embeddings and stores them in one of several vector database options available with Amazon Bedrock Knowledge Bases, such as Amazon OpenSearch Service or Amazon S3 Vectors, a capability of Amazon Simple Storage Service (Amazon S3), both of which should be configured with encryption at rest enabled. This vector database serves as the foundation for semantic search, enabling the system to find conceptually similar content rather than relying solely on keyword matching.

Access to the knowledge base and model invocations are governed through AWS Identity and Access Management (IAM) policies, with Amazon CloudWatch logging enabled to maintain audit trails for compliance purposes.

In terms of security, AIDA uses role-based access that is enforced using AWS IAM policies at the application layer in AIDA via project scoped roles, which restricts what each user can do.

User interaction flow

Step 4: Query embedding generation

When a user submits a query, AIDA uses Amazon Bedrock Guardrails to help safeguard against prompt injections and data leaks, the system then follows an enhanced RAG workflow that begins by converting the query into embeddings using Amazon Bedrock’s embedding models. This transformation enables semantic comparison between the user’s question and the stored document chunks.

Step 5: Implicit and explicit filtering

Before performing semantic search, the system applies both implicit and explicit filtering mechanisms—a key innovation that significantly improves retrieval accuracy. The implicit filtering automatically applies metadata-based conditions, such as filtering by effective date ranges or specific contracting parties, before executing the vector search. This two-stage approach first narrows the search space through metadata constraints, then applies semantic similarity matching within that filtered subset, so that retrieved documents are both contextually relevant and meet specific business criteria.

Step 6: Semantic search and document retrieval

The vector database retrieves the most relevant document chunks based on semantic similarity scores, typically using cosine similarity metrics to identify the closest matches to the user’s query. This semantic search operates on the filtered subset of documents, providing high precision and relevance in the results.

Step 7: Prompt augmentation

The retrieved chunks are then used to augment the original user’s query in a carefully formatted prompt that combines the user’s question with relevant contract excerpts and metadata context. This augmented prompt provides the large language model (LLM) with the specific information required to generate a precise, grounded response.

Step 8: LLM response generation

The augmented prompt is passed to a LLM through Amazon Bedrock, which generates a contextually precise response grounded in the actual contract documents rather than relying on the model’s training data alone. This approach helps provide responses that reflect the specific content of the organization’s contracts.

In addition, AIDA uses Amazon Bedrock Guardrails to apply content filtering, sensitive information (PII) safeguards, and prompt safety controls, to keep responses secure and aligned with enterprise and legal standards.

Step 9: Response delivery with source attribution

The final response is returned to the user through the Retrieve API, so that answers are not only relevant but also traceable back to specific source documents. This traceability reduces hallucinations, a common challenge with LLMs, and provides verifiable contract intelligence that users can trust for critical business decisions.

Tuning RAG

When developing a RAG system for legal contract analysis, one of the core challenges is managing the volume of potentially relevant information. Consider the following question:

“*Please identify any expired licensing agreements governed under California law? How do these agreements renew?*”

Such a query might match hundreds of relevant text chunks across multiple documents. In RAG terminology, these excerpts are often referred to as candidate chunks.

When Amazon Bedrock performs a semantic search to find the relevant document excerpts, it is designed to return the top k (max 100) relevant candidate chunks. The remaining candidate chunks that are filtered out could potentially contain critical context. Consequently, the LLM responses might be incomplete, overlook key insights, or even be inaccurate.

Legal documents are highly structured and deeply dependent on context. In our example, a renewal clause in a California licensing agreement may operate differently from a renewal clause in a licensing agreement governed by another jurisdiction. If the retrieval step does not properly filter licensing agreements and California governing law, the system may return irrelevant contracts such as service agreements, NDAs, or agreements governed under other state laws. Even if those documents contain expiration or renewal language, they do not answer the original question. Effective metadata filtering becomes essential to narrow down the candidate pool to the most relevant subsets. This allows the subsequent LLM call to receive the context required to provide more precise and complete answers.

It is important to note that while these retrieval improvements significantly enhance accuracy, AI-generated contract interpretations should always be reviewed by qualified legal professionals before being used for business decisions. AIDA is designed as a decision-support tool that can augment legal expertise rather than replacing it.

Two ways to assist in fine tuning the RAG are implicit filtering and explicit filtering which we will discuss next.

Implicit filtering

Implicit filtering is a powerful capability in Amazon Bedrock Knowledge Bases that allows you to automatically filter search results based on metadata attributes without requiring explicit filter expressions in each query. This approach enables the system to pre-filter the vector store based on document metadata before performing semantic search, significantly narrowing the search space to the most relevant documents.

The implicit filtering mechanism operates through a two-stage process:

Stage 1: Metadata pre-filtering

Before executing the semantic similarity search, the system applies metadata-based conditions automatically. You can supply a custom metadata file (up to 10 KB per document) for each document in your knowledge base, containing attributes such as effective dates, document types, parties involved, or custom fields relevant to your use case.

Stage 2: Semantic search

After narrowing the document set through metadata filters, the system performs vector similarity search only within this filtered subset. This helps reduce noise and irrelevant information while improving retrieval accuracy.

Explicit filtering

Explicit filtering is applied consistently at the application layer, independent of user input. This approach helps provide search results that remain aligned with business policies, compliance requirements, and organizational constraints, rather than being left entirely to the interpretation of the retrieval engine. Explicit filters can therefore act as a safeguard, helping to verify that every query respects a critical boundary.

Common use cases include the following:

• Application-Level Constraints – Restricting retrieval results based on predefined system rules geographic restrictions – Designed to restrict users in certain regions only access documents aligned with local regulations (e.g., European users restricted to EU-only contracts).

• Temporal Constraints – Returning only documents within a specific time frame (e.g., contracts active in the last two years).

• Classification and Sensitivity – Restricting search results to only documents tagged with a required confidentiality or classification level.

Example of our explicit filtering:

{
  "andAll": [
    { "listContains": { "key": "region", "value": "Germany" } },
    { "stringEquals": { "key": "confidentiality_level", "value": "Public" } }
  ]
}

Beyond filtering: Enriching chunks with metadata values

While implicit and explicit filtering narrows the pool of candidate chunks, they do not automatically provide the LLM with structured document-level context as described by the document metadata.

Metadata refers to structured attributes that describe a document beyond its raw text. These attributes are generated after document ingestion and indexing. Without these metadata values and context, the model may generate incomplete or less precise answers, especially in scenarios involving high volumes of contracts that are similar and complex in nature.

For legal contract analysis, common metadata fields include:

• Contract Type (e.g., Licensing Agreement)

• Governing Law (e.g., California)

• Effective Date

• Expiration Date

• Parties

• Business Unit

• Confidentiality Level

This structured layer enables filtering, policy enforcement, and richer reasoning.

Let’s return to our guiding question:

“*Please identify any expired licensing agreements governed under California law? How do these agreements renew?*”

Suppose one retrieved chunk contains the following clause:

*“This Agreement shall automatically renew for successive one-year terms unless either party provides written notice of non-renewal at least 60 days prior to the expiration date.”*

If this clause is provided to the LLM without the document-level associated metadata values, the model cannot determine the following:

• Whether the agreement is a licensing agreement.

• Whether it is governed under California law.

• Whether it has already expired.

• The actual expiration date.

Without these metadata values, the model may generate incomplete or less accurate responses.

To address this limitation, we enrich candidate chunks with document-level metadata values. Rather than duplicating metadata for every chunk, which would waste input tokens, we first group chunks by document and attach the relevant metadata once. This gives each chunk a richer context and helps the LLM reason more effectively, leading to more accurate, consistent, and reliable outputs.

Finally, it’s important to recognize that the quality of improvements depends on the relevance and appropriateness of the metadata we choose. Filtering and enrichment work best when queries are aligned with the available metadata. Conversely, queries unrelated to those metadata values may not show the same benefits. Careful design and thoughtful selection of metadata fields are therefore essential to maximize the impact of this approach.

Evaluating the impact of filtering and enrichment

To evaluate the impact of filtering and metadata enrichment, we tested our guiding question:

“*Please identify any expired licensing agreements governed under California law? How do these agreements renew?*”

The experiments were conducted on the Contract Understanding Atticus Dataset (CUAD) (licensing and co-branding agreements). For each test, the retrieval depth was set to top-k = 15. We compared four progressively enhanced configurations.

• Baseline RAG (no filters, no metadata). In the baseline setup, retrieval relied purely on semantic similarity. The system retrieved 55 potential candidate clauses across documents. However, only a small portion of the relevant context made it into the top 15 results passed to the model. Observations:

The model correctly identified one expired agreement (Snap/United).

• Renewal terms were partially described.

• The response lacked confidence and full contextual grounding.

• Explicit filtering only. Next, we applied explicit filters (for example, restricting results to U.S. contracts). This reduced the candidate pool, but accuracy did not improve consistently. Observations:

One agreement was incorrectly classified as expired.

• Renewal clauses were misinterpreted.

Filtering alone narrowed the search space but did not provide enough structured context for precise reasoning.

• Implicit and explicit filtering. We then combined query-derived filters (California law) and application-level constraints (U.S. contracts). This reduced the number of competing documents and improved contextual focus. Observations:

The model identified the correct agreement.

• However, it hesitated to definitively classify it as expired.

• Renewal mechanisms were described but cautiously phrased.

Filtering improved retrieval precision, but uncertainty remained.

• Filtering and metadata enrichment. Finally, we enriched retrieved chunks with document-level metadata such as governing law, expiration date, and contract type. Observations:

The system correctly identified only Snap/United as expired.

• Renewal terms were clearly and accurately explained.

• The response was grounded in both clause text and structured attributes.

This configuration produced the most accurate and confident output.

Setup

Candidate pool (n)

Coverage (15/k)

Expired agreements identified

Renewal terms explained

Answer quality

Baseline RAG

55

27.3%

Correctly identified *Snap/United* (expired 12/31/2000)

Renewal details incomplete

Partial – expiration found, rationale weak

Explicit only

40

37.5%

Misclassified as expired

Renewal terms misinterpreted

Not precise

Implicit and explicit

20

75.0%

Identified *Snap/United* but only as “potentially expired”

Renewal mechanisms described

Mixed – hedged on expiration