AWS 生成 AI モデルアジリティソリューション：LLM の移行に関する包括ガイド

Maintaining model agility is crucial for organizations to adapt to technological advancements and optimize their artificial intelligence (AI) solutions. Whether transitioning between different large language model (LLM) families or upgrading to newer versions within the same family, a structured migration approach and a standardized process are essential for facilitating continuous performance improvement while minimizing operational disruptions. However, developing such a solution is challenging in both technical and non-technical aspects because the solution needs to:

• Be generic to cover a variety of use cases

• Be specific so that a new user can apply it to the target use case

• Provide comprehensive and fair comparison between LLMs

• Be automated and scalable

• Incorporate domain- and task-specific knowledge and inputs

• Have a well-defined, end-to-end process from data preparation guidance to final success criteria

In this post, we introduce a systematic framework for LLM migration or upgrade in generative AI production, encompassing essential tools, methodologies, and best practices. The framework facilitates transitions between different LLMs by providing robust protocols for prompt conversion and optimization. It includes evaluation mechanisms that assess multiple performance dimensions, enabling data-driven decision-making through detailed and comparative analysis of source and destination models. The proposed approach offers a comprehensive solution that includes the technical aspects of model migration and provides quantifiable metrics to validate successful migration and identify areas for further optimization, facilitating a seamless transition and continuous improvement. Here are a few highlights of the solution:

• Provides a variety of reporting options with various LLM evaluation frameworks and comprehensive guidance for metrics selection for target use cases.

• Provides automated prompt optimization and migration with Amazon Bedrock Prompt Optimization and the Anthropic Metaprompt tool, in addition to best practices for further prompt optimization.

• Provides comprehensive guidance for model selection and an end-to-end solution for model comparison regarding cost, latency, accuracy, and quality.

• Provides feature examples and use case examples for users to quickly apply the solution to the target use case.

• The total time required for an LLM migration or upgrade by following this framework is from two days up to two weeks depending on the complexity of the use case.

Solution overview

The core of the migration involves a three-step approach, shown in the preceding diagram.

• Evaluate the source model.

• Prompt migration to and optimization of the target model with Amazon Bedrock prompt optimization and the Anthropic Metaprompt tool.

• Evaluate the target model.

This solution provides a comprehensive approach to upgrade existing generative AI solutions (source model) to LLMs on Amazon Bedrock (target model). This solution addresses technical challenges through:

• Evaluation metrics selection with a framework that uses various LLMs

• Prompt improvement and migration with Amazon Bedrock Prompt Optimization and the Anthropic Metaprompt tool

• Model comparison across cost, latency, and performance

This structured approach provides a robust framework for evaluating, migrating, and optimizing LLMs. By following these steps, we can transition between models, potentially unlocking improved performance, cost-efficiency, and capabilities in your AI applications. The process emphasizes thorough preparation, systematic evaluation, and continuous improvement; setting the stage for long-term success in using advanced language models.

Solution implementation

Dataset preparation

An evaluation dataset with high-quality samples is critical to the migration process. For most use cases, samples with ground truth answers are required; while for other use cases, metrics that don’t require ground truth—such as answer relevancy, faithfulness, toxicity, and bias (see *Evaluation of frameworks and metrics selection* section)—can be used as the determination metrics. Use the following guidance and data format to prepare the sample data for the target use cases.

Suggested fields for sample data include:

• Prompt used for the source model

• Prompt input (if any), for example: Questions and context for Retrieval-Augmented Generation (RAG)-based answer generation

• Configurations used for source model invocation, for example, temperature, top_p, top_k, and so on.

• Ground truths

• Output from the source model

• Latency of the source model

• Input and output tokens from the source model, which can be used for cost calculation

It’s important to remember that high quality ground truths are essential to successful migration for most use cases. Ground truths should not only be validated regarding correctness, but also to verify that they fit the subject matter expert’s (SME’s) guidance and evaluation criteria. See *Error **Analysis section* for an example of a SME’s guidance and evaluation criteria.

In addition, if any existing evaluation metrics are available, such as a human evaluation score or thumbs up/thumbs down from a SME, include those metrics and corresponding reasoning or comments for each data sample. If any automated evaluations have been conducted, include the automated evaluation scores, methods, and configurations. The following section provides more detailed guidance on selecting evaluation frameworks and defining the metrics. However, it’s still valuable to collect the existing or preferred evaluation metrics from stakeholders for reference.

Include the following fields if applicable:

• Existing human evaluation metrics for the source model, for example, the SME score for source model.

• Existing automated evaluation metrics for the source model, for example, the LLM-as-a-judge score for the source model.

The following table is an example format of the data samples:

sample_id

…

question

content

prompt_source_llm

answer_ground_truth

answer_ source_llm

latency_ source_llm

input_token_source_llm

output_token_source_llm

llm_judge_score_source_llm

human_score_source_llm

human_score_reasoning_source_llm

Evaluation of frameworks and metrics selection

After collecting information and data samples, the next step is to choose the proper evaluation metrics for the generative AI use case. Besides human evaluation by a SME, automated evaluation metrics are recommended because they are more scalable and objective and support the long-term health and sustainability of the product. The following table shows the automated metrics that are available for each use case.

Model selection

The selection of an appropriate LLM requires careful consideration of multiple factors. Whether migrating to an LLM within the same LLM family or to a different LLM family, understanding the key characteristics of each model and the evaluation criteria is crucial for success. When planning to migrate between LLMs, carefully compare and evaluate various available options and check out the model card and respective prompting guides released by each model provider. When evaluating LLM options, consider several key criteria:

• Input and output modalities: Text, code, and multi-modal capabilities

• Context window size: Maximum input tokens the model can process

• Cost per inference or token

• Performance metrics: Latency and throughput

• Output quality and accuracy

• Domain specialization and specific use case compatibility

• Hosting options: Cloud, on-premises, and hybrid

• Data privacy and security requirements

After initial filtering based on these characteristics, benchmarking tests should be conducted by evaluating performance on specific tasks to compare shortlisted models. Amazon Bedrock offers a comprehensive solution with access to various LLMs through a unified API. This allows us to experiment with different models, compare their performance, and even use multiple models in parallel, all while maintaining a single integration point. This approach not only simplifies the technical implementation but also helps avoid vendor lock-in by enabling a diversified AI model strategy.

Prompt migration

Two automated prompt migration and optimization tools are introduced here: the Amazon Bedrock Prompt Optimization and the Anthropic Metaprompt tool.

Amazon Bedrock Prompt Optimization

Amazon Bedrock Prompt Optimization is a tool available in Amazon Bedrock to automatically optimize prompts written by users. This helps users build high quality generative AI applications on Amazon Bedrock and reduces friction when moving workloads from other providers to Amazon Bedrock. Amazon Bedrock Prompt Optimization can enable migration of existing workloads from a source model to LLMs on Amazon Bedrock with minimal prompt engineering. With this tool, we can choose the model to optimize the prompt for and then generate an optimized prompt for the target model. The main advantage of using Amazon Bedrock Prompt Optimization is the ability to use it from the AWS Management Console for Amazon Bedrock. Using the console, we can quickly generate a new prompt for the target model. We can also use the Bedrock API to generate a migrated prompt, please see the detailed implementation below.

Option A) Optimize a prompt from the Amazon Bedrock Console

• In the Amazon Bedrock console, go to Prompt management.

• Choose Create prompt, enter a name for the prompt template, and choose Create.

• Enter the source model prompt. Create variables by enclosing a name with double curly braces: {{variable}}. In the Test variables section, enter values to replace the variables with when testing.

• Select a Target Model for your optimized prompt. For example, Anthropic’s Claude Sonnet 4.

• Choose the Optimize button to generate an optimized prompt for the target model.

1. After the prompt is generated, the comparison window of the optimized prompt for the target model is shown with your original prompt from source model.

1. Save the new optimized prompt before exiting comparing mode.

Option B) Optimize a prompt using Amazon Bedrock API

We can also use the Bedrock API to generate a migrated prompt, by sending an OptimizePromptrequest with an Agents for Amazon Bedrock runtime endpoint. Provide the prompt to optimize in the input object and specify the model to optimize for in the targetModelId field.

The response stream returns the following events:

• analyzePromptEvent – Appears when the prompt is finished being analyzed. Contains a message describing the analysis of the prompt.

• optimizedPromptEvent – Appears when the prompt has finished being rewritten. Contains the optimized prompt.

Run the following code sample to optimize a prompt:

import boto3

# Set values here
TARGET_MODEL_ID = "anthropic.claude-3-sonnet-20240229-v1:0" # Model to optimize for. For model IDs, see https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html
PROMPT = "Please summarize this text: " # Prompt to optimize

def get_input(prompt):
    return {
        "textPrompt": {
            "text": prompt
        }
    }
 
def handle_response_stream(response):
    try:
        event_stream = response['optimizedPrompt']
        for event in event_stream:
            if 'optimizedPromptEvent' in event:
                print("========================== OPTIMIZED PROMPT ======================\n")
                optimized_prompt = event['optimizedPromptEvent']
                print(optimized_prompt)
            else:
                print("========================= ANALYZE PROMPT =======================\n")
                analyze_prompt = event['analyzePromptEvent']
                print(analyze_prompt)
    except Exception as e:
        raise e
 
 
if __name__ == '__main__':
    client = boto3.client('bedrock-agent-runtime')
    try:
        response = client.optimize_prompt(
            input=get_input(PROMPT),
            targetModelId=TARGET_MODEL_ID
        )
        print("Request ID:", response.get("ResponseMetadata").get("RequestId"))
        print("========================== INPUT PROMPT ======================\n")
        print(PROMPT)
        handle_response_stream(response)
    except Exception as e:
        raise e

Anthropic Metaprompt tool

The Metaprompt is a prompt optimization tool offered by Anthropic where Claude is prompted to write prompt templates on the user’s behalf based on a topic or task. We can use it to instruct Claude on how to best construct a prompt to achieve a given objective consistently and accurately.

The key steps are:

• Specify the raw prompt template, explain the task, and specify the input variables and the expected output.

• Run Metaprompt with a Claude LLM such as Claude-3-Sonnet by inputting the raw prompt from the source model.

The new prompt template is generated with an optimized set of instructions and format following Claude LLM’s bes