Amazon SageMaker Python SDK v3、LLM 最適化機能を統合

Optimizing generative AI inference deployments requires benchmarking endpoints, evaluating instance configurations, and iterating on deployment settings. The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook workflow. These recommendations are also accessible through the Amazon SageMaker AI UI and Boto3 APIs. With this release, you can benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration directly from a notebook using the Amazon SageMaker Python SDK v3.

In this post, we demonstrate how to use the new SDK interface for the end-to-end workflow to optimize generative AI inference deployments.

Benefits of generative AI inference recommendations in Amazon SageMaker AI

Generative AI inference recommendations in Amazon SageMaker AI automate inference optimization by:

• Benchmarking a live Amazon SageMaker endpoint against a synthetic or real-traffic workload, measuring throughput, time-to-first-token (TTFT), end-to-end latency, and more.

• Generating deployment recommendations ranked by cost-performance tradeoff using your actual usage patterns.

• Deploying the top-ranked configuration directly to an Amazon SageMaker real-time endpoint.

Previously, these capabilities required using Amazon SageMaker Studio or constructing AWS SDK for Python (Boto3) API calls. With this launch, they become Python SDK operations, fitting naturally into existing notebook and pipeline workflows.

New SDK interfaces

The new functionality is available under the sagemaker.serve.ai_inference_recommender package starting with version 3.17.0 and exposes the following primary operations:

Prerequisites

Verify you have the latest version of the Amazon SageMaker Python SDK installed:

pip install --upgrade sagemaker >= 3.17.0

You will also need:

• An AWS account with an AWS Identity and Access Management (IAM) role with Amazon SageMaker execution permissions.

• A deployed Amazon SageMaker real-time endpoint (or a JumpStart model to deploy; see the following section).

Solution overview

Consider a common scenario: you have a generative AI model ready for production and need to determine the optimal instance type, framework configuration, and serving parameters. Traditionally, this involves manual trial and error across multiple instance types, container versions, and concurrency settings. With the Amazon SageMaker Python SDK integration, you can automate this entire workflow in a single notebook. The following walkthrough guides you through the end-to-end journey using this notebook:

• Generate deployment recommendations: Let the service explore instance and framework configurations against your workload profile and return ranked options.

• Interpret and select: Review the ranked results, understand the tradeoffs, and pick the best fit.

• Deploy: Push the winning configuration to a live Amazon SageMaker endpoint.

• Benchmark: Validate the deployed endpoint under realistic load conditions.

• Compare frameworks: Optionally run LMI and vLLM head-to-head to find the best serving stack.

Generate recommendations from real traffic data

Your first step is to find the best deployment configuration for your model and workload. Rather than manually deploying across multiple instance types, call mb.generate_deployment_recommendations(…) to let the service explore instance types and framework configurations against your workload profile. The service deploys your model on each candidate, runs a load test matching your traffic pattern, and returns a ranked list of configurations optimized for your chosen performance target.

import time, uuid
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder
from sagemaker.train.configs import Compute
from sagemaker.serve import InferenceFramework, PerformanceTarget

uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
src_model_name = f"demo-rec-source-{uid}"
rec_ep_name = f"demo-rec-ep-{uid}"

mb = ModelBuilder.from_jumpstart_config(
    jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
    compute=Compute(instance_type=INSTANCE_TYPE),
    role_arn=ROLE,
)
source_model = mb.build(model_name=src_model_name)

rec_job = mb.generate_deployment_recommendations(
    tokenizer="google/gemma-4-e2b-it",
    concurrency=1,
    request_count=10,
    prompt_input_tokens_mean=32,
    output_tokens_mean=32,
    streaming=True,
    performance_target=PerformanceTarget.TTFT_MS
    instance_types=[INSTANCE_TYPE],
    advanced_optimization=False,
    framework=InferenceFramework.LMI,
    role_arn=ROLE,
    wait=True,
)

#Comparative table across all returned recommendations
print(mb.recommendations)

# .best is the top-ranked row
top = mb.recommendations.best
print(f"Throughput avg: {top.expected_performance.request_throughput.avg}")
print(f"TTFT p99: {top.expected_performance.time_to_first_token.p99}")

# auto_approve=True bypasses the ModelPackage approval-status check
rec_endpoint = mb.deploy(
    endpoint_name=rec_ep_name,
    role=ROLE,
    wait=True,
    auto_approve=True,
)
print(f"Deployed: {rec_endpoint.endpoint_name} ({rec_endpoint.endpoint_status})")

Recommendation results can also be represented as a Python data frame.

import pandas as pd

pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", 60)

rows = []
for i, rec in enumerate(mb.recommendations):
    raw = rec._raw
    spec = raw.model_details.inference_specification_name
    for m in raw.expected_performance:
        rows.append({
            "rank": i,
            "spec_name": spec,
            "instance": raw.deployment_configuration.instance_type,
            "metric": m.metric,  # ← attribute, not subscript
            "stat": m.stat,
            "value": float(m.value),
            "unit": m.unit,
        })
df_long = pd.DataFrame(rows)
print(df_long)

    rank  spec_name                          instance       metric                 stat  value      unit
0   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestThroughput      avg   112.7664   Requests/Second
1   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  OutputTokenThroughput  avg   3608.5300  Tokens/Second
2   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestLatency         p50   462.1300   Milliseconds
3   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestLatency         p90   999.5400   Milliseconds
4   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestLatency         p99   1069.8400  Milliseconds
5   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  TimeToFirstToken       p50   438.5300   Milliseconds
6   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  TimeToFirstToken       p90   983.3300   Milliseconds
7   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  InterTokenLatency      p50   0.7900     Milliseconds
8   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  InterTokenLatency      p90   2.8700     Milliseconds
9   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  ClientSideConcurrency        64.0000    Count
10  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestThroughput      avg   96.8522    Requests/Second
11  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  OutputTokenThroughput  avg   3099.2700  Tokens/Second
12  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestLatency         p50   541.1600   Milliseconds
13  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestLatency         p90   1122.2000  Milliseconds
14  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestLatency         p99   1162.5300  Milliseconds
15  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  TimeToFirstToken       p50   502.9000   Milliseconds
16  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  TimeToFirstToken       p90   1088.4800  Milliseconds
17  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  InterTokenLatency      p50   1.0000     Milliseconds
18  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  InterTokenLatency      p90   3.6100     Milliseconds
19  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  ClientSideConcurrency        64.0000    Count

How to interpret recommendation results

The recommendations table shows two candidate configurations (rank 0 and rank 1), both on ml.g6.2xlarge but with different LMI container versions. Here’s how to read the key metrics and choose between them:

Key metrics to compare:

• RequestThroughput (avg): Requests the endpoint can serve per second. Higher is better.

• OutputTokenThroughput (avg): Total tokens generated per second across all concurrent requests. Higher is better.

• RequestLatency (p50/p90/p99): End-to-end time from request to full response. Lower is better.

• TimeToFirstToken (p50/p90): How quickly the user sees the first streamed token. Lower is better.

• InterTokenLatency (p50/p90): Delay between successive tokens during streaming. Lower is better.

Choosing between the two configurations in this example:

Rank 0 (lmi-26-0-0) delivers 112.8 req/s throughput and 3,609 tokens/s, with p90 TTFT of 983 ms and p90 latency of 1,000 ms. Rank 1 (lmi-27-0-0) delivers 96.9 req/s throughput and 3,099 tokens/s, with p90 TTFT of 1,088 ms and p90 latency of 1,122 ms. Rank 0 wins on every dimension: approximately 16% higher throughput and approximately 10 percent lower latency. The service ranks it first because the job was configured with performance_target=PerformanceTarget.TTFT_MS, meaning the optimizer prioritized configurations that minimize time-to-first-token.

General decision framework

Latency-sensitive applications (chatbots, interactive UIs): Prioritize low TTFT (p90/p99) so users perceive fast responses.

Throughput-sensitive workloads (batch summarization, offline processing): Prioritize high RequestThroughput and OutputTokenThroughput to maximize tokens per dollar.

If two configurations are close on your primary metric, use the secondary metrics as tiebreakers, then factor in cost (a smaller instance at similar performance saves money).

In this example, the top-ranked configuration (lmi-26-0-0) is the clear choice because it dominates across all metrics at the same concurrency level (64).

Deploy from previously run recommendation job

In production workflows, you often generate recommendations in one session and deploy in another. For example, a data scientist might run the recommendation job during experimentation, while an MLOps pipeline deploys the result during a release cycle. Use ModelBuilder.from_recommendation_job(job_name) to hydrate a ModelBuilder from a completed job:

from sagemaker.serve import ModelBuilder

# Hydrate a fresh ModelBuilder from a completed recommendation job
mb = ModelBuilder.from_recommendation_job("my-rec-job-name")
print(f"Loaded {len(mb.recommendations)} recommendations")
print(mb.recommendations)

# Deploy the top-ranked recommendation
endpoint = mb.deploy(
    role=ROLE,
    wait=True,
    auto_approve=True,
)

Deploy a JumpStart model and benchmark it

After you have deployed your recommended configuration, the next step is to validate its performance under controlled conditions. Benchmarking confirms that the endpoint meets your latency and throughput requirements before serving production traffic. The SDK makes this straightforward: deploy a JumpStart model and run a synthetic load test in only a few lines of code.

import time, uuid
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder, start_benchmark
from sagemaker.train.configs import Compute
from sagemaker.serve import InferenceFramework, PerformanceTarget

uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
ep_name = f"demo-bench-ep-{uid}"
model_name = f"demo-bench-model-{uid}"

# Build and deploy a JumpStart endpoint
mb = ModelBuilder.from_jumpstart_config(
    jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
    compute=Compute(instance_type=INSTANCE_TYPE),
    role_arn=ROLE,
)
core_model = mb.build(model_name=model_name)
core_endpoint = mb.deploy(endpoint_name=ep_name)

# Benchmark with a synthetic workload
job = start_benchmark(
    endpoint=core_endpoint,
    tokenizer="google/gemma-4-e2b-it",
    concurrency=1,
    request_count=10,
    prompt_input_tokens_mean=32,
    output_tokens_mean=32,
    streaming=True,
    role=ROLE,
    wait=True,
)
result = job.show_result()

Reading benchmark results

After the benchmark completes, you need to understand whether your endpoint meets your service-level objectives. The benchmark returns a typed result object with a metrics accessor that gives you programmatic access to throughput, latency percentiles, and token-level timing. IDE autocomplete works on all fields.

# Well-known shortcuts — fully typed, IDE autocomplete works
print(f"Throughput avg: {result.metrics.request_throughput.avg} req/sec")
print(f"TTFT p99: {result.metrics.time_to_first_token.p99} ms")
print(f"E2E latency p90: {result.metrics.request_latency.p90} ms")

# Any metric AIPerf produced, by raw key
ott = result.metrics.get("output_token_throughput")
if ott:
    print(f"Output token throughput p90: {ott.p90} {ott.unit}")

Benchmark results can similarly be represented as a Python data frame.

import pandas as pd

pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", 60)

result = job.show_result()

rows = []
for name, m in result.metrics.all_metrics.items():
    rows.append({
        "metric": name,
        "unit": m.unit,
        "avg": m.avg,
        "p50": m.p50,
        "p90": m.p90,
        "p99": m.p99,
    })
df = pd.DataFrame(rows).set_index("metric")
print(df)

unit avg p50 p90 p99

metric

request_throughput requests/sec 3.841439 NaN NaN NaN

request_latency ms 256.288407 206.586304 257.917252 656.735558

request_count requests 10.000000 NaN NaN NaN

time_to_first_token ms 91.770545 23.977319 93.899204 636.935233

time_to_second_token ms 3.035445 3.400013 3.611624 3.670301

inter_token_latency ms 4.987234 5.518454 5.689482 5.695788

output_token_throughput tokens/sec 130.608919 NaN NaN NaN

output_token_throughput_per_user tokens/sec/user 1036.753582 181.210181 1041.238702 7969.310858

output_sequence_length tokens 34.000000 34.000000 35.000000 35.000000

input_sequence_length tokens 32.000000 32.000000 32.000000 32.000000

output_token_count tokens 34.000000 34.000000 35.000000 35.000000

inter_chunk_latency ms 5.307028 5.921907 6.133938 6.599335

total_output_tokens tokens 340.000000 NaN NaN NaN

benchmark_duration sec 2.603191 NaN NaN NaN

total_isl tokens 320.000000 NaN NaN NaN

total_osl tokens 340.000000 NaN NaN NaN

http_req_sending ms 0.504841 0.230543 0.527351 2.798082

http_req_waiting ms 91.130199 23.709262 93.193301 633.795432

http_req_data_received KB 7.954883 7.956055 7.970801 7.971592

http_req_connecting ms 0.257011 0.000000 0.257011 2.338797

http_req_connection_reused ratio 0.900000 1.000000 1.000000 1.000000

http_req_blocked ms 0.000000 0.000000 0.000000 0.000000

http_req_dns_lookup ms 0.042871 0.000000 0.042871 0.390126

http_req_chunks_received count