Amazon Bedrock でクエリ認識圧縮により RAG コストを削減

Input tokens sent to the foundation model (FM) on every call are often a meaningful part of the cost of running Retrieval Augmented Generation (RAG) at scale. Query-aware compression offers one way to reduce how many of them reach the model. Amazon Bedrock provides the foundation models and features to build RAG applications. RAG retrieval usually tunes for high recall, returning a broad set of potentially relevant chunks so the primary model has thorough source material to work with. This design helps builders feel confident that the right information is available at inference time. As workloads scale, builders often look for ways to optimize the cost-performance tradeoff by reducing the number of input tokens the primary model processes while maintaining answer quality. The open, composable architecture of Amazon Bedrock supports custom post-retrieval processing steps that refine what reaches the primary model.

In this post, we describe a post-retrieval customization pattern that achieves significant input-token reduction, and therefore cost savings, while preserving answer quality. It’s compatible with RAG retrievers on Amazon Bedrock, including Amazon Bedrock Knowledge Bases. As a secondary benefit, removing irrelevant context reduces the surface area for hallucination. After retrieval but before the final answer call, a smaller, lower-cost model on Amazon Bedrock filters retrieved chunks against the user’s query. The primary model then receives the filtered context and generates the answer.

We cover the pattern’s architecture at a high level, show the core Amazon Bedrock implementation in a AWS Lambda function, walk through the cost model and the latency tradeoff, and describe how we evaluated answer quality. We also look at how this pattern can layer on top of existing Amazon Bedrock capabilities like prompt caching, Amazon Bedrock Intelligent Prompt Routing, and the Rerank API for compounding cost savings.

Prerequisites

To implement the solution, complete the following prerequisite steps:

• Have an active AWS account.

• Create an AWS Identity and Access Management (IAM) role for the Lambda function to access Amazon Bedrock. For instructions, refer to Create a role to delegate permissions to an AWS service.

• Add policy permissions to the IAM role.

• Amazon Bedrock model access in your AWS Region for the two models in the cascade: a smaller compression model (Anthropic Claude Haiku in this post) and a primary model (Anthropic Claude Sonnet). Request access on the Amazon Bedrock console under Model access.

How query-aware compression reduces RAG costs on Amazon Bedrock

A RAG flow using traditional RAG infrastructure or frameworks looks like:

• The application embeds the user query.

• The vector index returns the top-k chunks (commonly 5–20).

• Optionally, a reranker reorders those chunks by relevance.

• All retrieved chunks are concatenated into the prompt.

• The primary model generates the answer.

Retrieved context scales with top-k and chunk size: retrieving 5–20 chunks at typical chunk sizes puts many technical-documentation and legal RAG workloads in the range of several thousand input tokens per query. Reducing the per-query token count can yield meaningful cost savings.

Architecture overview

A smaller model reads the retrieved chunks alongside the query and outputs only the verbatim spans relevant to the question. We use Claude Haiku in this post, but the pattern works with other small/primary model pairs within a model family on Amazon Bedrock. Both the compression call and the primary model’s answer call run inside a single AWS Lambda function. Upstream, a retriever embeds the query and returns the top-k chunks. An Amazon Bedrock knowledge base, the fully managed RAG capability backed by Amazon OpenSearch Serverless, is one such retriever. The Lambda function receives those chunks as input and returns the final answer. The compression call is the only step added to a standard RAG flow.

Because the smaller model costs less per token than the primary model, trimming the context before the expensive answer call is where the savings come from. How large those savings are comes down to two things.

The following diagram shows the solution architecture.

**Figure 1: Query-aware context compression architecture on Amazon Bedrock

The flow proceeds through the following steps:**

• The user submits a query to the application.

• The application sends the query to the retriever, which searches the vectorized knowledge sources.

• The retriever returns the top-k chunks (full retrieved context) to the application.

• The application passes the query and all retrieved chunks to an AWS Lambda function, which sends them to a smaller model (Claude Haiku) through the Amazon Bedrock Converse API. This is the compression call.

• The smaller model outputs only the verbatim spans relevant to the query (the compressed context).

• The Lambda function sends the query and the compressed context to the primary model (Claude Sonnet) through the Converse API. This is the answer call.

• The primary model generates the final answer, which the application returns to the user.

Why this works economically

The economics depend on two factors: the price ratio between the small and primary models on Amazon Bedrock, and the compression ratio the smaller model achieves.

For a single RAG query with R retrieved input tokens, a compression ratio of c (where c > 1), a final answer output of A tokens, and per-token prices P_small_in / P_small_out (smaller model input and output price) and P_large_in / P_large_out (primary model input and output price):

The compression call adds the input and output cost of the smaller model. Savings come from sending R/c instead of R tokens to the primary model. The economics favor compression when:

• Retrieved context is large (the savings on the larger model dominate).

• The price ratio between large and small is high (Sonnet/Opus paired with Haiku).

• A meaningful portion of the retrieved content can be trimmed for a given query without affecting answer quality.

Implementation on Amazon Bedrock

The pattern fits between retrieval and the final answer call. We implement it as a single AWS Lambda function that orchestrates the two Amazon Bedrock model invocations using the Converse API. The function receives the user query and the retrieved chunks as its input event.

The compression prompt

The compression prompt is the most important part of the implementation. It must instruct the smaller model to extract spans rather than summarize, forbid paraphrasing and rewriting, and preserve enough surrounding context for citations to remain accurate.

COMPRESSION_SYSTEM_PROMPT = """You extract evidence from retrieved documents.
You will receive a user QUESTION and a list of CHUNKS.

Your job:
1. For each chunk, identify the spans (verbatim sentences or short paragraphs)
that contain evidence directly relevant to answering the QUESTION.
2. Output ONLY those spans, copied verbatim from the source. Do not paraphrase,
summarize, or rewrite.
3. Preserve chunk identifiers so the downstream system can cite sources.
4. If a chunk contains no relevant evidence, output the chunk identifier
followed by NO_RELEVANT_EVIDENCE.

Output format (strict):
[CHUNK_ID: ]

[CHUNK_ID: ]
NO_RELEVANT_EVIDENCE

Do not add commentary, headings, conclusions, or your own words. Only verbatim
spans from the source chunks, grouped by chunk identifier."""

The Lambda function

The function takes the user query and the retrieved chunks, reads the two model IDs from environment variables, and initializes an Amazon Bedrock Runtime client configured with adaptive retries. It then makes two calls through the Bedrock Converse API: the first to the smaller model to compress the chunks, and the second to the primary model to generate the answer from the compressed evidence. The compression call runs at temperature 0.0, which keeps the extraction deterministic so the smaller model copies spans as they appear in the source:

Step 1: Compress. The smaller model filters the retrieved chunks.

compressed = bedrock.converse(
    modelId=COMPRESSION_MODEL_ID,
    system=[{"text": COMPRESSION_SYSTEM_PROMPT}],
    messages=[{"role": "user",
               "content": [{"text": f"QUESTION:\n{query}\n\nCHUNKS:\n{chunks}"}]}],
    inferenceConfig={"temperature": 0.0},
)

Step 2: Answer. The primary model reasons over the filtered evidence.

answer = bedrock.converse(
    modelId=ANSWER_MODEL_ID,
    system=[{"text": "Answer using only the evidence provided. Cite sources."}],
    messages=[{"role": "user",
               "content": [{"text": f"QUESTION:\n{query}\n\nEVIDENCE:\n{compressed}"}]}],
)

Note: The prompts are an example and should be adapted to your documents and question types.

Methodology

Before recommending this pattern, we evaluated it empirically. The benchmark covered:

• A corpus of more than 500,000 documents spanning 9 enterprise source types, including chat messages, email, issue-tracker tickets, shared-drive documents, CRM records, meeting transcripts, code repositories, and wiki pages.

• A set of 500 questions across 10 categories, ranging from narrow factual lookups to broader multi-part questions, including both precisely worded and more informally phrased queries.

• Multiple compression conditions: no compression (baseline), compression, and rerank-plus-compression.

• Each query run through baseline and optimized pipelines, with answers scored by a large language model (LLM) judge.

• Quality scoring across four dimensions (correctness, completeness, citation accuracy, and conciseness) against a reference answer. Faithfulness was tracked separately, checking each answer’s claims against the evidence the primary model received.

These figures describe one corpus, one domain, and one query distribution. Results on your own documents and queries will differ.

結果

The following table summarizes the headline results across the queries, comparing the baseline, compression, and rerank + compression pipelines.

The following figures come from the benchmark. Results on your own corpus, queries, and model choices will differ. The following figure shows the average of query cost saving (left axis, percentage versus baseline) and the reduction in context sent to the primary model (right axis, times fewer tokens). Compression achieved a 33 percent cost saving or 8.6× fewer tokens. Rerank + compression reached 36 percent cost saving and 10.1× fewer tokens.

Figure 2: Cost savings and reduction in context tokens sent to the primary model

The following figure shows the LLM-judge scores (1–5) across the four answer-quality dimensions for each pipeline. Correctness stays within 0.07 of baseline across conditions. Completeness and citation accuracy are slightly lower under compression, while conciseness is slightly higher.

Figure 3: LLM-judge scores across the 4 answer-quality dimensions by pipeline

The following figure shows the hallucination rate for each pipeline, measured as the share of answers containing at least one claim not supported by the reference. The baseline is 51 percent, compression 44 percent, and rerank + compression 38 percent.

Figure 4: Hallucination rate by pipeline

The following figure shows the cost saving versus baseline for the typical-query set and the hard-query set. Compression moves from 37 percent to 26 percent, and rerank + compression from 40 percent to 30 percent between the two sets.

Figure 5: Cost savings for the typical-query and hard-query sets

Considerations for production use

Three things must be weighed before this pattern is adopted: the latency of the added compression call, the impact on answer quality, and whether the workload is a fit. Each is covered in the following sections.

Latency

Adding a smaller-model call introduces one extra step in the path. Claude Haiku is optimized for speed, and because the primary model then processes a smaller, focused context, part of that added time is recovered on the answer call. The total end-to-end latency is the compression call plus the answer call on the focused context. Net impact depends on how compute-bound the primary model is on the original context size.

For latency-critical surfaces (sub