vLLM のルーティングと KV キャッシュに関する解説

vLLM routing and KV

The vLLM real-world lab models mixed production traffic instead of a single throughput number. FCK generated the request mixes, ran the scheduler and routing profiles, captured build outcomes, and emitted the evidence used for the charts below. The lab covers six request classes: interactive chat, RAG with repeated prefixes, long-prefill requests, agent tool loops, batch summarization, and slow streaming clients.

The adapter matrix compares vLLM V1, SGLang, llama.cpp, and TGI through the same OpenAI-compatible contract. The serving sweep covers the profiles operators actually tune: one balanced pool, a large-token pool, a small interactive pool, prefix-cache routing, slow-client isolation, and class-aware routing.

Main result: one global vLLM pool is a poor default for mixed traffic. The best profile was vllm-v1/class-aware-router, which gave the strongest combined tradeoff across first-token latency, token cadence, slow-reader isolation, and useful throughput.

The practical advice is to split lanes before changing kernels. Keep interactive traffic on a smaller max_num_batched_tokens budget with moderate max_num_seqs, then move long-context and batch work to a separate pool with a larger token budget.

Chunked prefill needs the same treatment. Keep max_long_partial_prefills below max_num_partial_prefills so short prompts can still enter the scheduler while long prompts are being processed. Treat token budget, sequence count, partial-prefill limits, and stream interval as workload controls.

The scale run made the routing result clearer. The single shared pool failed the TTFT/ITL gate under the larger workload, while class-aware routing accepted nearly all requests. Prefix-cache-only and slow-client-only profiles helped their own lanes, but neither worked as a whole-system strategy.

The source-build run added useful deployment checks. Debian needed a larger rootfs and then hit a GCC version gate. Fedora 44 with GCC 16, Python 3.14, VLLM_TARGET_DEVICE=cpu, MAX_JOBS=4, and numactl-devel built the extension modules successfully. The build also surfaced a deployment note worth keeping: install tcmalloc for better runtime performance.

The hybrid KV lab covers the PagedAttention rewrite path. It keeps vLLM's block ownership, prefix sharing, refcounts, partial blocks, eviction, and copy-on-write semantics, then exposes compact logical spans to future kernels instead of a long per-token block table.

The correctness oracle compares each layout against a plain reference across odd lengths, MQA/GQA, ALiBI, sliding windows, prefix reuse, copy-on-write, partial final blocks, and FP8-style scaling. The first run passed 30/30 cases and ranked virtual-contiguous and hybrid-prefix-shared as the first layouts to profile on hardware.

Profile Stress Result

600k Request Stress Result

120k requests/profile, 30-minute simulated production windowclass-aware-router66.48 good rpsaccepted 99.72% | p99 TTFT 387.94 ms | p99 queue 310.07 msprefix-cache-rag26.89 good rpsaccepted 40.34% | p99 TTFT 1766.03 ms | p99 queue 1475.72 msslow-client-isolation7.04 good rpsaccepted 10.56% | p99 TTFT 2293.08 ms | p99 queue 2001.69 msglobal-large-token-budget4.31 good rpsaccepted 6.46% | p99 TTFT 1071.02 ms | p99 queue 888.14 mssingle-pool-balanced0.00 good rpsaccepted 0.00% | p99 TTFT 2455.03 ms | p99 queue 2203.19 ms

Profile Matrix

ProfileGoodputTTFTITL

class-aware-router66.48387.9447.33

prefix-cache-rag26.891766.03105.07

slow-client-isolation7.042293.0885.29

global-large-token-budget4.311071.0297.34

single-pool-balanced0.002455.03120.47

Tuning Read

• One global pool is not a safe production default for mixed traffic. Under the 600k-request stress, it produced zero gated goodput.

• A larger global token budget is not a router. It improved the single-pool control but still missed the gate with p99 TTFT over one second.

• Prefix-cache routing is valuable but incomplete. It protected RAG and agent-tool-loop requests while starving ordinary interactive, batch, long-prefill, and slow-client classes.

• Slow-client isolation fixed slow-reader penalty but did not solve long-prefill queueing. Use it as an edge/proxy lane inside a broader router.

• The winning shape is class-aware: short interactive, prefix-heavy, long-prefill, batch, and slow-stream lanes with different token budgets and sequence limits.

Source Build Lab

Build Outcomes

Bar length = build command wall time; failed setup cases are shown as diagnostic runsDebian 12 default rootfs1.5 GiB catalog rootfs exhausted at 150.5/190.2 MB of torch-2.11.0+cpu.setupDebian 12 + diskGiB 12CMake reached the x86 backend and rejected GCC/G++ 12.2.3.00sFedora 44, no NUMA headersCompiled 428 targets before csrc/cpu/utils.cpp needed numactl-devel.712.00sFedora 44 + numactl-develBuilt _C, _C_AVX512, and _C_AVX2 shared modules with MAX_JOBS=4.853.00sFedora CUDA negative controlVLLM_TARGET_DEVICE=cuda without CUDA runtime metadata failed with Unknown runtime environment.1.26s

Fedora source-build recipe

This was the passing path in the lab: Fedora 44, GCC 16, Python 3.14, 12 GiB rootfs, numactl-devel, and limited parallelism.

dnf install -y git gcc gcc-c++ cmake ninja-build python3.14 python3.14-devel numactl-devel
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 1c607d7b2cd4fb572b919c6053f19d0577203495
python3.14 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip setuptools wheel
VLLM_TARGET_DEVICE=cpu MAX_JOBS=4 python -m pip install -v -e .

Debian source-build preflight

Debian 12 exposed two checks before a successful build: disk space for the PyTorch wheel and GCC/G++ version. Use a larger rootfs and a compiler at or above the x86 backend gate.

apt-get update
apt-get install -y git cmake ninja-build python3 python3-venv python3-dev libnuma-dev
# Debian 12 GCC/G++ 12.2 failed the current x86 backend gate.
# Install a newer GCC/G++ toolchain first, then set CC/CXX.
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 1c607d7b2cd4fb572b919c6053f19d0577203495
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip setuptools wheel
CC=gcc-13 CXX=g++-13 VLLM_TARGET_DEVICE=cpu MAX_JOBS=4 python -m pip install -v -e .

Hybrid KV Rewrite Lab

Paged Blocks To Virtual Spans

Lower estimated decode-read cost is better; all rows are simulator/reference evidencevirtual-contiguousread amp 1.08 | locality 90.43 | prefix reuse 15.63%2649.08hybrid-prefix-sharedread amp 1.12 | locality 86.41 | prefix reuse 16.99%3173.11hot-tail-contiguousread amp 1.28 | locality 71.89 | prefix reuse 14.27%5493.56packed-block-tableread amp 1.42 | locality 62.02 | prefix reuse 14.27%7215.82classic-pagedread amp 1.70 | locality 46.82 | prefix reuse 14.27%10368.30

Layout Metrics

LayoutRead AmpWasteCost

virtual-contiguous1.081.56%2649.08

hybrid-prefix-shared1.121.58%3173.11

hot-tail-contiguous1.281.49%5493.56

packed-block-table1.421.49%7215.82

classic-paged1.701.49%10368.30

Allocator Model

• KV memory is split into fixed-size physical blocks instead of one contiguous allocation per request.

• Requests with token-identical prefixes can point at the same prefix blocks and raise refcounts.

• When a shared sequence diverges, copy-on-write gives the diverging request private blocks without corrupting the shared prefix.

• The hybrid path keeps those allocator semantics but emits compact logical spans so kernels can read longer runs.

Top Profiles

ProfileTTFTITLGoodput

vllm-v1/class-aware-router254.5933.2922.32

sglang/class-aware-router234.2230.6320.98

vllm-v1/prefix-cache-rag359.9445.8020.80

vllm-v1/global-large-token-budget614.2748.4020.61

sglang/prefix-cache-rag331.1442.1319.56

sglang/global-large-token-budget565.1344.5319.38

vllm-v1/single-pool-balanced465.3552.0418.91

vllm-v1/slow-client-isolation353.6734.7818.53

Workload Mix

Bubble size = request share, x = p99 prompt tokens, y = shared prefix rateagent-tool-loopbatch-summaryinteractive-chatlong-prefillrag-shared-prefixslow-stream-client- ## How vLLM Works

async engine

scheduler

KV manager

block pool

PagedAttention op

model runner

Step 1 of 6: 1. API admission

1. API admission

The OpenAI-compatible server accepts a request, parses sampling parameters, tokenizes or receives token IDs, and hands work to the async engine.

TTFT pressurequeueing, tokenization, long prefill

ITL pressurenot active yet

Tuning surfaceroute by workload class before the engine

Restaurant Kitchen Analogy

Requests enter like order tickets. Nothing is visible to the diner yet; the system is deciding what work enters the next step.

Plain-English Mapping

Order tickets are user requests waiting for the scheduler.

• Prep work is prefill: reading the prompt before any answer appears.

• The first plated item is TTFT: the first token the user can see.

• Plating one item at a time is decode: repeated next-token work.

• The prep table is KV cache: stored context that avoids redoing work.

• Servers carrying plates are streaming clients; slow tables can back up service.

Traffic Intersection View

One traffic light for every lane lets a long prefill hold the intersection while short chats queue behind it.

Memory Hotel Map

A request checks into KV memory by taking rooms: fixed-size blocks that hold prompt and generated-token state.

Token Timeline Scrubber

At 0 ms: queue; TTFT waiting; ITL not active; decoded tokens 0; KV 18%.

Latency Read

• Queue is time before the scheduler admits the request into an engine step.

• Prefill processes prompt tokens and usually dominates TTFT for long contexts.

• First token is the user-visible TTFT boundary.

• Decode advances active requests token by token; this is where ITL shows up.

• Streaming can stretch visible cadence even when the engine is healthy.

Scheduler Budget Explorer

Decode owns most of the step. Keep the interactive token budget small and predictable.

High-Level Model

• Requests enter the OpenAI-compatible API and wait for an engine step.

• The scheduler spends each step's token budget on prefill and decode work.

• Prefill is prompt processing; long prompts need chunking so decode is not starved.

• Decode advances many active requests by the next token through continuous batching.

• KV cache blocks are the memory constraint; prefix reuse helps only when tokens match.

• Streaming turns engine cadence into client-visible TTFT and inter-token gaps.

Findings

Class-aware routing beats one global vLLM pool for mixed traffic

Best profile vllm-v1/class-aware-router cut p99 TTFT by 45.29% and p99 inter-token gap by 36.03% versus baseline.

• Use one small-token-budget interactive pool and one long/batch pool before tuning kernels.
• Treat max_num_batched_tokens as a per-lane control, not a cluster-wide constant.

A larger token budget improves long-prefill capacity but hurts interactive TTFT

The global-large-token-budget strategy keeps long prompts moving but is dominated by the routed profile for user-facing latency.

• Use max_num_batched_tokens=8192 or higher only in long-context lanes.

Slow streaming clients should be isolated from engine cadence

The best profile reduced slow-client penalty by 75.99% versus baseline.

• Put slow SSE readers behind a proxy or worker lane with explicit write deadlines.
• Do not interpret socket backpressure as a scheduler or kernel regression without this test.

Large profile stress punishes narrow fixes

Across 600,000 requests, class-aware routing was the only profile that kept the whole workload inside the TTFT/ITL gates.

• Promote class-aware routing to the first accelerator experiment.
• Keep prefix-cache and slow-client isolation as lanes inside the router, not standalone serving strategies.

Hybrid KV is now a testable kernel-rewrite candidate

The lab preserved PagedAttention semantics while changing the kernel-facing view. It simulated classic paged blocks, packed tables, hot-tail contiguous allocation, virtual-contiguous spans, and hybrid prefix sharing, then checked each layout against the same reference attention path before ranking virtual-contiguous and hybrid-prefix-shared first.

• Keep block ownership, refcounts, prefix sharing, eviction, and copy-on-write in the KV manager.
• Emit compact span-table records to kernels: logical start, length, physical run, sharing state, and quantization scale metadata.
• Measure read amplification, memory waste, locality, and estimated decode-read cost before writing a hardware kernel.
• Validate with real L2/DRAM counters, occupancy, CUDA graph behavior, TTFT, and ITL before claiming speed.

GPU Deploy Template

This is the deployment shape used by FCK for the lab: vLLM runs behind a small router, chunked prefill is enabled, and the scheduler limits are explicit so each serving lane can be changed and measured independently.

services:
  vllm:
    image: vllm/vllm-openai:latest
    command:
      - --model=<your-model>
      - --host=0.0.0.0
      - --port=8000
      - --enable-chunked-prefill
      - --max-num-batched-tokens=4096
      - --max-num-seqs=128
      - --max-num-partial-prefills=2
      - --max-long-partial-prefills=1
      - --long-prefill-token-threshold=4096
    ports: ["8000:8000"]
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: ["gpu"]

  router:
    image: nginx:stable
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    ports: ["8080:8080"]
    depends_on: [vllm]

References