SageMaker HyperPod における LLM 推論のための分離型プリフィルとデコード

When prefill and decode share a GPU, long prompts stall token generation for every concurrent request. Disaggregated Prefill and Decode (DPD) removes this interference by running each phase on separate GPU pools connected through Elastic Fabric Adapter (EFA) with Remote Direct Memory Access (RDMA). Large language model (LLM) inference has two fundamentally different phases. Prefill is compute-bound. It processes the entire input prompt in parallel to generate the initial key-value (KV) cache. Decode is memory-bound. It generates one token at a time and requires substantial memory bandwidth to access model weights and the growing KV cache. By disaggregating these into specialized engines, you can assign different parallel strategies to each phase. With this separation, you can tune time to first token (TTFT) and inter-token latency (ITL) independently, control tail latency more reliably than chunked prefill tuning, and keep long-context prefills from blocking ongoing decode requests. vLLM improves single-node efficiency through continuous batching and PagedAttention. However, organizations that deploy at scale still face challenges when they orchestrate multi-node deployments and optimize routing.

In this post, we show how to implement DPD with vLLM on Amazon SageMaker HyperPod using the HyperPod Inference Operator.

When to use disaggregated inference

Disaggregating prefill and decode delivers the strongest gains for long-context, high-concurrency streaming workloads: chat assistants, agentic pipelines, document-analysis endpoints, and Retrieval Augmented Generation (RAG) with large retrieved contexts. In these cases, a single long prompt on a colocated GPU stalls in-flight decode for every other request, causing per-token latency spikes that DPD removes by construction.

Consider DPD when your workload has:

• Input prompts that regularly exceed 4,096 tokens.

• Multiple concurrent users or requests.

• Streaming responses where consistent token delivery matters.

• Mixed traffic with both long and short prompts.

A colocated deployment is the simpler choice when GPU contention is not a real concern: batch or offline workloads optimizing for TTFT, low-concurrency deployments, or short-prompt-only traffic. Below the routing threshold, the fixed cost of transferring KV cache over EFA RDMA outweighs the benefit of isolating decode. The DPD router sends those requests straight to a decoder. A single endpoint therefore handles mixed long and short traffic automatically, without manual routing logic.

DPD requires at minimum one prefill node and one decode node with RDMA-capable EFA networking. For supported instance types, see the Deploy a DPD model endpoint to your HyperPod cluster section.

Architecture

The HyperPod DPD implementation is built on the vLLM Production Stack router, with LMCache providing the KV cache transfer layer over NIXL and EFA. The deployment has three components plus a transport stack.

Intelligent router

The router is the control plane. It tokenizes each prompt and applies a configurable token threshold to decide whether the request takes the disaggregated path or runs end-to-end on a decoder. Long-context prompts go through a prefiller then a decoder. Short prompts skip the prefiller, avoiding cross-GPU KV transfer that isn’t worthwhile. For disaggregated requests, it directs the prefiller to compute and push KV cache to a decoder, then forwards the request to that decoder for generation. It also supports per-prefiller routing strategies (prefixaware, kvaware, session, roundrobin) through intelligentRoutingSpec.routingStrategy to maximize cache locality across replicas.

Prefiller pod

The prefiller is a vLLM worker with LMCache as its KV connector through LMCacheConnectorV1. It computes KV cache for long prompts and pushes it to the chosen decoder via LMCache’s PD sender backend layer-by-layer, overlapping compute and transfer to keep its GPUs saturated. LMCache also gives each prefiller an L1 CPU cache. When a prefix recurs (system prompts, multi-turn history, retrieval contexts), it serves from CPU memory without GPU recomputation. This produces significant TTFT gains. Activating DPD on an InferenceEndpointConfig provisions both the connector and the cache automatically.

Decoder pod

The decoder is a vLLM worker with LMCache as its receiver. It reserves GPU memory (the PD buffer, sized by PD_BUFFER_SIZE) for incoming KV transfers. It runs full CUDA graphs for the decode kernel and starts generation as soon as the transfer completes. Because it never executes prefill, decode latency stays stable under concurrency and adding a long-context request never disturbs tokens already streaming.

KV transfer

KV cache transfer uses a four-layer stack (LMCache PD → NIXL → libfabric → EFA) that HyperPod composes end-to-end. LMCache’s PD backend orchestrates the prefiller-side put and decoder-side retrieval. NIXL provides a unified memory abstraction across GPU, CPU, and remote peers and selects the right RDMA operation. The libfabric provider exposes EFA as kernel-bypass, GPU-Direct RDMA, keeping the host CPU off the data path. This makes transfer cost negligible relative to prefill compute: on ml.p5.48xlarge with 3,200 Gbps of EFA, an 8,000-token transfer for Llama 3.3 70B takes single-digit milliseconds. HyperPod ships the stack pre-integrated, so you select a DPD-supported worker image and the operator wires up the connector, NIXL, and EFA on every pod.

Deployment overview

Disaggregated Prefill and Decode (DPD) is a functionality implemented by the HyperPod Inference Operator. This section covers prerequisites, installation of the inference operator and deployment of an inference endpoint that uses DPD for efficiently serving a Llama 70B model.

Prerequisites and HyperPod Inference Operator installation

Make sure you have the AWS Command Line Interface (AWS CLI), kubectl access to your HyperPod cluster, a HuggingFace token, and sufficient service quota. Set up your local kubectl configuration to connect to your HyperPod cluster. For more information, see Disaggregated Prefill and Decode for HyperPod inference.

DPD requires HyperPod Inference Operator version 3.2 or later. The operator is installed by default on new HyperPod EKS clusters. For installation, setup, and upgrade instructions, see Unlock efficient model deployment: simplified Inference Operator setup on Amazon SageMaker HyperPod.

Verify your operator version by running:

kubectl get deployment hyperpod-inference-operator-controller-manager \
  -n hyperpod-inference-system \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="manager")].image}{"\n"}'

The output is the full container image reference. The tag at the end encodes the version, for example:

XXXXXXXXXXX.dkr.ecr.us-east-2.amazonaws.com/hyperpod-inference-operator:v3.2

If your operator version is not up-to-date, upgrade before continuing by following the upgrade instructions in the HyperPod Inference Operator Release Notes.

Deploy a DPD model endpoint to your HyperPod cluster

In this example, we deploy the Meta Llama 3.3 70B model on two ml.p5.48xlarge instances. Verify that the instances are available in an instance group within the HyperPod cluster before proceeding. For DPD inference deployments, choose instance types that support both NVLink and EFA. EFA needs to support RDMA in read and write mode. This includes the P5 and P6 instance families on AWS. Note that instances are required to be located within the same Availability Zone (AZ) for EFA high-bandwidth communication. Although G6, G6e, and G7e instance families do support EFA with RDMA read/write, performance on multi-GPU instances is bottlenecked by GPU-to-GPU communication over PCIe.

The worker image for the inference deployment must include vLLM, LMCache, NVIDIA NIXL, and the EFA libfabric provider. At the time of writing, we support two image options:

• Open source LMCache: lmcache/vllm-openai v0.4.3.

• SageMaker Deep Learning Container (DLC): vllm:server-hyperpod-cuda-v1.1.

Model checkpoint location

The HyperPod Inference Operator supports a broad range of checkpoint loading sources, including Amazon Simple Storage Service (Amazon S3) buckets, Amazon FSx file systems, and direct pulling from HuggingFace and the instance NVMe storage. For this post, we load the model checkpoint from an Amazon S3 bucket.

Verify that you have downloaded your preferred model checkpoint to an S3 bucket in the same Region as your HyperPod cluster. If you haven’t done so yet, set up your bucket name and HuggingFace token, then download Meta Llama 3.3 70B Instruct from HuggingFace and sync it to Amazon S3. To achieve high-bandwidth networking to Amazon S3, we recommend that you run this from an Amazon Elastic Compute Cloud (Amazon EC2) instance.

export MODEL_BUCKET=
export MODEL_PREFIX=Llama-3.3-70B-Instruct
export AWS_REGION=
export HF_TOKEN=
pip install -U "huggingface_hub[cli]" "huggingface_hub[hf-transfer]"
HF_HUB_ENABLE_HF_TRANSFER=1 hf download meta-llama/Llama-3.3-70B-Instruct \
  --local-dir ./$MODEL_PREFIX \
  --token "$HF_TOKEN"
aws s3 sync ./$MODEL_PREFIX \
  s3://$MODEL_BUCKET/$MODEL_PREFIX/ \
  --region "$AWS_REGION"

Prepare the model deployment manifest and change the environment variables as needed:

export DEPLOYMENT_NAME="dpd-test-deployment"
export ENDPOINT_NAME="dpd-test"
export MODEL_NAME="meta-llama-3-3-70b"
export NAMESPACE="default"
export INSTANCE_TYPE="ml.p5.48xlarge"
export GPUS_PER_NODE="8"
export MODEL_IMAGE="lmcache/vllm-openai:v0.4.3"

For the full deployment YAML, see Deploy a DPD endpoint.

DPD-relevant fields in the deployment manifest

Most InferenceEndpointConfig fields are shared with non-DPD endpoints and documented in the Inference Operator documentation. The fields below are required or have different semantics for DPD.

spec.pdSpec: Declares the prefill/decode topology and specifies arguments. Presence of this field is what makes the endpoint disaggregated: the operator creates separate Deployment objects for prefill and decode and wires them together through the router and LMCache PD backend.

• replicas: Scale prefill and decode independently.

• resources: Applied to the role’s pod spec. Top-level worker.resources is ignored for DPD pods. Per-role values override.

• routingThreshold: Token length threshold above which requests use the disaggregated path. Below the threshold, requests bypass the prefiller and go directly to the decoder.

• args: vLLM flags specific to that role, merged into worker.args at startup. Flags already in worker.args are replaced with the per-role value, and flags not present are appended.

spec.worker.environmentVariables: These environment variables are applied identically to both the prefiller and decoder containers. There is no per-role environment-variable field today. For per-role behavior, use pdSpec.{prefillSpec,decodingSpec}.args instead.

More details about environment variables are in Deploy a DPD endpoint.

Apply the manifest and validate the deployment

kubectl apply -f inference_endpoint_dpd_config.yaml

The operator creates two Deployment objects in your namespace and a router Deployment in hyperpod-inference-system. Image pull and model load take a few minutes. The pods first enter ContainerCreating, then become Running as containers come up. List the pods across both namespaces:

kubectl get pods -A \
  | grep -E "prefill-${DEPLOYMENT_NAME}|decode-${DEPLOYMENT_NAME}|${DEPLOYMENT_NAME}-${NAMESPACE}-router"

NAMESPACE                   NAME                                           READY   STATUS    RESTARTS   AGE
default                     prefill-dpd-test-deployment-XXXX               3/3     Running   0          7m
default                     decode-dpd-test-deployment-XXXX                3/3     Running   0          7m
hyperpod-inference-system   dpd-test-deployment-default-router-XXXX        2/2     Running   0          7m

Each model pod has 3 containers: the vLLM worker, an Nginx reverse proxy, and an OpenTelemetry collector. The router pod has 2 containers (router, otel). The IEC condition reports readiness:

kubectl get inferenceendpointconfig ${DEPLOYMENT_NAME} -n ${NAMESPACE} \
  -o jsonpath='{.status.conditions[0].message}{"\n"}'

DPD prefill and decode deployments are ready

Invoke the endpoint and verify KV transfer

Once the endpoint is ready, send a short and a long prompt to exercise both routing paths. Check the prefiller and decoder logs to confirm KV cache is being transferred over EFA. The following commands assume the IEC was deployed with ${NAMESPACE}, ${ENDPOINT_NAME}, and ${DEPLOYMENT_NAME} set as in the preceding manifest.

Get the required pod names and router url:

PREFILL_POD=$(kubectl get pod -n ${NAMESPACE} \
  -l 'inference.sagemaker.aws.amazon.com/dpd-role=prefill' \
  -o jsonpath='{.items[0].metadata.name}')
DECODE_POD=$(kubectl get pod -n ${NAMESPACE} \
  -l 'inference.sagemaker.aws.amazon.com/dpd-role=decode' \
  -o jsonpath='{.items[0].metadata.name}')
ROUTER_POD=$(kubectl get pods -n hyperpod-inference-system -o name \
  | grep -- "${DEPLOYMENT_NAME}-${NAMESPACE}-router" | head -1)
ROUTER_URL=http://${DEPLOYMENT_NAME}-${NAMESPACE}-routing-service.hyperpod-inference-system.svc.cluster.local:443/v1/chat/completions

Short prompt (below threshold, direct-to-decoder path)

Run a short prompt through a pod within the cluster that invokes the endpoint:

kubectl run curl-short --rm -it --image=curlimages/curl --restart=Never -- \
  curl -s -k -X POST "$ROUTER_URL" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "/opt/ml/model",
      "messages": [{"role": "user", "content": "What is disaggregated prefill-decode in one sentence?"}],
      "max_tokens": 80,
      "temperature": 0.0
    }'

{
  "id": "chatcmpl-7-...",
  "object": "chat.completion",
  "model": "/opt/ml/model",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "Disaggregated prefill-decode is an inference architecture that ..."},
    "finish_reason": "stop"
  }]
}

Long prompt (above threshold, DPD path)

A prompt above the 4,096-token routing threshold is routed through the prefiller, then to the decoder for token generation. The following example builds a roughly 6,000-token prompt by repeating a sentence:

kubectl run curl-long --rm -it --image=curlimages/curl --restart=Never -- sh -c '

ROUTER="'"$ROUTER_URL"'"

LONG=""

i=0; while [ $i -lt 600 ]; do LONG="${LONG}The quick brown