DeepSeek-V4-Pro の高速配信技術と H20 GPU での実装知見

1. Introduction

DeepSeek-V4-Pro is a 1.6-trillion-parameter Mixture-of-Experts (MoE) model released with both FP8 and FP4 weights. Models at this scale naturally benefit from accelerators such as NVIDIA Blackwell GPUs, which offer more HBM, higher compute throughput, and native FP4 Tensor Cores. Yet H20 GPUs remain widely deployed, despite lacking those advantages.

Hardware constraints do not relax serving requirements. Long-context prefill must still control time to first token (TTFT). Interactive decode must satisfy the time-per-output-token (TPOT) target of each service tier. Sustained traffic must balance aggregate throughput against KV-cache capacity. Short inputs, long contexts, latency-sensitive requests, and high concurrency stress the system in different ways; no universal configuration can serve all of them well.

One model needs multiple serving profiles. Workload characteristics, service-level objectives (SLOs), and measured hardware behavior jointly inform the deployment topology and execution path:

• Match serving profiles to the workload. In the configurations evaluated here, prefill selects between PP2 and PP4 based on the measured context-length range, while decode uses profiles optimized for different latency, throughput, and KV-capacity targets.

• Optimize the prefill path. We optimize Attention-CP8 → MoE-TP8 and context-parallel communication, then tune for the real routing shapes produced by long- and short-context workloads.

• Optimize the decode path. We optimize the DSpark speculative-decoding path, refine execution, expert routing, and communication–computation overlap for distinct decode SLOs.

Push the latency frontier. At batch size 1, the single-node H20-141GB reference reaches 271 output tokens/s, compared with the 383.7 tokens/s reported on B300. Despite the substantial hardware gap, workload-specific system optimization narrows the observed decode performance ratio to 1.42×. Detailed benchmark settings and the log-based throughput extraction methodology are provided in Appendix B.3.

Cover the serving envelope. The latency result represents only one edge of the system. Across the broader profile family, optimized prefill reaches 8.45k input tokens/s per node and processes a 1M-token prompt in 43.7 seconds. For throughput-oriented decode, the DP16-EP16 efficiency reference reaches 4.67k output tokens/s per node, corresponding to an average TPOT of 27.4 ms. These results intentionally come from different profiles, each selected and optimized for a different combination of context length, latency, throughput, and capacity constraints.

The contribution is a methodology, not a single benchmark. Scenario-specific serving allows each workload to move toward a better measured operating point among the profiles evaluated on the available hardware. We hope the deployment choices, optimization methods, and measurements presented here provide a practical reference for teams serving frontier models under compute, memory, bandwidth, or interconnect constraints.

2. From Hardware Constraints to Serving Profiles

2.1 Hardware Constraints and Serving Roles

Blackwell offers raw performance; H20 offers deployable scale. B300 provides native FP4 Tensor Cores, much higher FP8 throughput, and substantially more HBM. H20 cannot match its compute capability, but it remains available at scale and provides high memory bandwidth and 900 GB/s NVLink. Each node in this study contains eight GPUs connected by NVLink. Prefill does not retain long-lived per-request state, so its hardware choice is governed primarily by TTFT, compute, and communication efficiency. Decode must retain the KV cache of every active request throughout generation, making HBM capacity a direct limit on context length and concurrency. For the deployment studied here, this led us to use H20-141GB for decode and H20-96GB—whose capacity was sufficient for our prefill workloads—for prefill.

2.2 Capacity Choices

Serving capacity ultimately comes from a shared HBM budget: model weights and per-request KV state compete for the same memory. We define full-token capacity as the maximum number of full-attention KV tokens that each rank can hold after model weights and runtime buffers have been allocated. It is a memory ceiling rather than a direct guarantee of admissible batch size.

Reducing Weight Footprint with Humming MXFP4AFP8

Reduce the weight footprint first. Humming MXFP4AFP8 uses MXFP4 expert weights with online FP8 activations to reduce weight footprint and memory traffic on H20 GPUs, which lack native FP4 Tensor Cores. The SGLang integration is available in sglang#23754. We will cover the Humming/SGLang integration in a dedicated follow-up post. Model-level accuracy results and public reference measurements are provided in Appendix D.2.

Expanding KV Capacity with Online C128

Give the KV cache room to grow. The Offline C128 baseline retains per-index state for each compressed page. Online C128 instead maintains a compact aggregate state, releasing more HBM to the KV-cache pool. It introduces additional state maintenance and speculative-verification work, but we observed no TPOT regression in our tests.

Combined Capacity Gains

Capacity gains compound across weights and KV state. By reducing the weight footprint, Humming MXFP4AFP8 expands full-token capacity to 1.71× the Baseline FP8 + Offline C128 configuration for DP32-EP32 and 4.47× for PP2-TP8. Online C128 then reduces the C128 auxiliary-state footprint, providing another 2.268× increase on top of Humming. Combined, the two techniques raise capacity to 3.88× the baseline for DP32-EP32 and 10.14× for PP2-TP8. Appendix D.1 provides the complete data.

2.3 Scenario-Specific Serving Profiles

Prefill Profiles

The right pipeline depth depends on how much work there is to pipeline. PP2-CP8-TP8 and PP4-CP8-TP8 share the same Attention-CP8 → MoE-TP8 execution path. At the topology level, their primary difference is pipeline depth: PP2 distributes the model across two stages, while PP4 uses four.

Short contexts favor lower pipeline overhead; long contexts expose more parallelism. Short inputs produce fewer chunks, leaving a deeper pipeline underfilled and making fill, drain, and cross-stage transfer costs more prominent. Long contexts provide enough chunks to keep four stages busy; with fewer layers per stage, the additional nodes translate into more prefill parallelism. In our deployment, these characteristics led us to use PP2-CP8-TP8 for shorter contexts and PP4-CP8-TP8 for long-context workloads.

Low-Latency Decode Profiles

Low latency starts with the shortest execution path. Single-node TP8 and PP2-TP8 share the same Attention-TP8 → MoE-TP8 execution path; the difference is whether the model is partitioned across nodes. Single-node TP8 places all layers on one H20-141GB node and avoids cross-stage communication and synchronization. PP2-TP8 partitions the model across two pipeline stages.

The fastest topology is not always the most serviceable one. Single-node TP8 has the shorter execution path, but model weights and serving state share the HBM of one node, leaving limited room for the KV cache. It cannot simultaneously support long contexts and larger batch sizes. PP2-TP8 pays additional pipeline overhead but distributes the model weights across two nodes, releasing more HBM for KV state. For our latency and capacity targets, we use single-node TP8 as the batch-size-1 latency reference and PP2-TP8 as the low-latency serving profile.

High-Throughput Decode Profiles

High-throughput decode scales data and expert parallelism together. Both profiles use the Attention-DP → MoE-EP execution path. DP16-EP16 is the smallest deployment unit; DP32-EP32 expands both DP and EP within the same topology.

Scale-out prioritizes request capacity over per-GPU throughput. A larger EP group distributes expert weights across more GPUs, releasing HBM for the KV cache and admitting more concurrent requests. At the same time, a smaller fraction of MoE traffic remains within each node, while a larger fraction crosses nodes, which can reduce per-GPU efficiency. In the profiles evaluated here, we use DP16-EP16 as the smallest deployment unit and efficiency reference, and DP32-EP32 to expand request capacity.

3. Prefill: Balancing Compute and Communication

Prefill performance is a system problem. Expert imbalance, context-parallel communication, and production routing shapes jointly determine TTFT; optimizing an isolated kernel is not enough.

3.1 Why MoE-TP Instead of MoE-EP

Less traffic can still take longer. MoE-EP exchanges only routed tokens, but real prefill traffic exhibits significant expert skew. Ranks that own hot experts perform more computation and become stragglers; all other ranks wait for the slowest path at the combine step. Lower communication volume does not translate into lower TTFT.

Balance compute before minimizing traffic. For the H20 prefill workloads evaluated here, both PP2 and PP4 use MoE-TP. Full-sequence all-gather and reduce-scatter introduce more communication, but the traffic remains on high-bandwidth NVLink and has stable, predictable cost. All TP ranks execute tensor-parallel computation over the same routed tokens, preventing expert skew from becoming a rank-level long tail. For this workload, predictable communication is cheaper than unpredictable imbalance. The implementation is available in sglang#24947.

3.2 Accelerating and Fusing Prefill Collectives

Build a reusable collective fast path. MoE-TP replaces unpredictable expert imbalance with predictable collective traffic, making communication efficiency the next bottleneck. We made symmetric memory reusable across TP and CP, allowing AllReduce, AllGather, and ReduceScatter to share registered-buffer fast paths and applicable Hopper acceleration. The supporting upstream work spans memory-pool ownership, communicator registration, MoE-TP collective buffers, and the CP Attention and KV-cache buffer paths.

Then shorten the Prefill critical path. Faster collectives alone do not remove the boundaries between communication and computation. For the 32K single-chunk case, we built a fused path that overlaps a copy-engine-driven AllGather with fused FP8 quantization and shared-expert GEMM, then combines TopK reduction, shared-expert addition, and ReduceScatter in a second Triton kernel. This reorganizes seven operators into three execution groups and reduces TTFT by approximately 3.5% in a matched PP4 A/B.

3.3 Tuning Humming for Real Routing Shapes

Generic tuning misses the shapes that matter. Prefill routing distributes tokens unevenly across 384 experts, so the effective M dimension clusters into a small set of discrete values. W13 and W2 also operate on different shapes, so a single generic heuristic cannot optimize both paths.

Tune from production routing. We extract high-frequency shapes from real routing histograms, build separate exact-shape configurations for W13 and W2, and validate them at the kernel, pipeline-stage, and matched A/B levels. The optimization target is not a synthetic range of M, but the routing distribution we actually serve. In a matched PP4 A/B at 32K, selected MoE kernel latency falls by approximately 21%, translating into an 11.35% end-to-end TTFT reduction.

4. Decode: Optimizing Speculation and MoE Execution

Decode optimization is profile-specific in our implementation. PP2-TP8 requires coordination across speculative pipeline stages, while DP32-EP32 focuses on optimizing the refinement step and expert routing at high concurrency. Humming fusion and overlap improve the shared MoE hot path beneath these serving topologies.

4.1 Low-Latency PP2-TP8: Extending DSpark Across Pipeline Stages

Pipeline parallelism splits the speculative loop. In PP2-TP8, target execution spans two pipeline stages, while the DSpark drafter resides only on the final stage. Stage 0 sends target hidden states to Stage 1, which performs verification, accepts tokens, and generates candidates for the next round.

Make two stages advance as one. Every speculative round crosses the pipeline boundary. We coordinate both stages and the required intermediate transfers under one execution protocol, preventing the stages from entering different rounds while avoiding redundant synchronization. The PP-specific DSpark integration is being upstreamed in sglang#32281.

4.2 High-Throughput DP32-EP32: Removing High-Concurrency Bottlenecks

The matched A/B results in this subsection use DP32-EP32 at 4K with 32 concurrent requests per DP rank.

Choose the right execution shape for refinement. The refinement step applies a full-vocabulary projection to rescore DSpark's candidate set. At high concurrency, the row-wise dot-reduce repeatedly reads the vocabulary weights for every active row, creating a persistent tail in each decode step. We combine active rows into one transposed GEMM, reducing redundant memory traffic and shortening the refinement path. Per-GPU throughput improves by 22.8%.

Place experts from measured routing. DSpark traffic also exhibits significant expert skew. We record routing affinity from representative requests and use it to configure expert-parallel load balancing (EPLB) and redundant experts, preventing a small number of hot experts from repeatedly extending the critical path. Per-GPU throughput improves by 13.5%.

4.3 Humming Decode Hot Path: Fusion and Overlap

These optimizations sit below the serving topology and can be reused by Humming-based decode profiles. The matched results below use DP32-EP32 at 4K with 32 concurrent requests per DP rank.

Remove the extra quantization pass. We fuse the SwiGLU activation with quantization so that the fused kernel directly produces the data and scale required by W2. This eliminates repeated access to an intermediate buffer and removes the standalone quantization pass, allowing W2 to start earlier. In the matched DSpark A/B, per-GPU throughput improves by 44.0%.

Overlap communication with W2. We adapt the Single-Batch Overlap (SBO) mechanism from our previous work (sglang#9660) into Humming-Aware SBO. Per-tile signals allow DeepEP to begin the corresponding combine send as soon as a W2 output tile completes, without waiting for the entire GEMM. In an earlier matched non-spec A/B at the same operating point, SBO recovers 4.12% throughput relative to the FP8-transport tier.

5. Evaluation: System Gains and Profile Trade-offs

5.1 Prefill: Cumulative Gains and Context-Length Trade-offs

PP2 strengthens the short-context profile. PP2 improves at all nine input lengths, with a geometric-mean throughput gain of 36.5% and a peak total input throughput of 16,900 tokens/s. Its shallower pipeline reduces fill-and-drain overhead for short requests, allowing PP2 to maintain lower TTFT with fewer resources.

PP4 carries the gains into long context. PP4 delivers a geometric-mean throughput gain of 31.8% across the same nine points. As context length grows, the deeper pipeline has enough work to amortize its fixed cost: total input throughput reaches 25,860 tokens/s at 512K and remains 23,970 tokens/s at 1M.

Context length shifts the PP2/PP4 trade-off. Relative to PP4, PP2 lowers TTFT by 16.7% at 4K and 19.5% at 32K. The two profiles remain within 2% at 8K, 16K, and 64K. PP4 establishes a decisive advantage from 128K onward, reducing TTFT relative to PP2 by 26.2%, 33.3%, 42.1%, and 44.8% at 128K, 256K, 512K, and 1M, respectively. We therefore treat the routing boundary as an operating policy derived from the measured context-length range rather than a universal crossover point.

Appendix A.1–A.2 provide the complete TTFT and total-input-throughput results.

5.2 Low-Latency Decode: Performance and Capacity Trade-offs

Optimized DSpark resets the latency baseline. Across the four input lengths shown in Figure 15, Optimized DSpark reduces peak TPOT by 74.8%–78.0% at batch size 1. At the largest batch size shared by each pair of measurements, the reduction remains 52.2%–60.0%. The gain holds from 8K through 1M rather than being confined to short contexts or single-request execution.

Observed serving performance is much closer than peak-compute ratios alone suggest. Across the four input lengths shown in Figure 16, Optimized DSpark on PP2-TP8 reaches 150–174 tokens/s at batch size 1. The single-node TP8 reference reaches 183–271 tokens/s. For the precisions used by the actual execution paths, B300 has approximately 45.6× the peak Tensor Core compute of H20-141GB (B300 FP4 versus H20 FP8) and 1.67× its memory bandwidth. Yet the highest observed generation rates are 383.7 tokens/s on B300 and 271 tokens/s on H20-141GB, respectively—a ratio of 1.42×. Even against this much stronger hardware reference, workload-specific optimization brings the H20-141GB reference substantially closer in observed serving performance.

Capacity favors PP2-TP8 for our production targets. Single-node TP8 is faster, but at a 1M context it has enough KV-cache capacity only for batch size 1. It cannot admit a larger batch or more concurrent requests. By distributing model weights across two pipeline stages, PP2-TP8 supports batch sizes 4, 8, and 16 at 1M, 512K, and 256K, respectively. With Online C128, its full-token capacity reaches 11.04M tokens/rank. For context-length and concurrency targets similar to ours, we recommend retaining single-node TP8 as the latency reference and using PP2-TP8 as the low-latency serving profile. Appendix B and Appendix D.1 provide the complete performance and capacity data.

5.3 High-Throughput Decode: Frontier Gains and Profile Trade-offs

Figure 17 shows how the throughput–interactivity frontier evolves with the system. The horizontal axis is interactivity in tokens/s/user, and the vertical axis is throughput in tokens/s/GPU. In these DP/EP profiles, each DP rank maps to one GPU; interactivity is per-GPU throughput divided by the number of concurrent requests per DP rank. Points farther toward the upper right provide a better combination of user-visible generation speed and GPU efficiency. The four curves represent cumulative system evolution rather than the isolated gain of any optimization in Section 4.

MTP denotes multi-token prediction; the (3, 1, 4) configuration uses three speculative steps, top-k 1, and four draft tokens.

System optimization moves the entire frontier. At 4K with 32 concurrent requests per DP rank, per-GPU throughput rises from 319.92 tokens/s/GPU to 703.15 tokens/s/GPU, a 2.20× increase. At 1M with one request per DP rank, it rises from 27.05 tokens/s/GPU to 66.82 tokens/s/GPU. The first three system milestones can each process only one request per DP rank at 1M; the final system supports four and reaches 177.48 tokens/s/GPU. The expanded operating envelope comes from both faster execution and greater capacity.

Smaller deployment units preserve efficiency at selected high-concurrency operating points. In our earlier work on serving DeepSeek-V3/R1 on H20, we found that a smaller EP deployment unit can keep a larger fraction of MoE traffic within each node. DeepSeek-V4-Pro shows the same advantage at the operating points plotted in Figure 18: with 16 and 32 concurrent requests per DP rank, DP16-EP16 delivers approximately 3.6%–20% higher per-GPU throughput than DP32-EP32. The full sweep is not monotonic across every concurrency level, so we use DP16-EP16 as an efficiency reference rather than a universal replacement for DP32-EP32.

Capacity shifts the preferred high-throughput profile. DP16-EP16 is more efficient per GPU, but DP32-EP32 distributes expert weights across more ranks and releases additional HBM for the KV cache. At 256K, 512K, and 1M, the maximum concurrent requests per DP rank increase from 8, 4, and 2 to 16, 8, and 4, respectively—a consistent 2× expansion. For deployments with long-context concurrency targets similar to ours, this additional capacity favors DP32-EP32 as the capacity-oriented high-throughput profile, while DP16-EP16 remains useful as an efficiency reference. Appendix C and Appendix D.1 provide the complete data.

6. Conclusion

One model does not require one compromise profile. We built a scenario-specific serving stack for DeepSeek-V4-Pro on H20. Prefill switches between PP2 and PP4 according to context length. Decode uses PP2-TP8 for low latency and DP32-EP32 for high throughput. By co-designing capacity, deployment topology, and execution path, H20 can sustain 1M-token contexts and meet multiple serving SLOs despite limited compute and the absence of native FP4 Tensor Cores.

The transferable result is a scenario-driven methodology. Serving profiles should not be selected from hardware specifications or isolated benchmarks alone. We recommend starting from the workload, SLO, context length, and concurrency, then using profiling to identify the binding resource and translate it into concrete topology and execution-path decisions. We hope this methodology helps AI infrastructure teams build practical frontier-model serving systems under diverse resource constraints—whether the bottleneck is compute, memory capacity, memory bandwidth, or interconnect—and share those lessons with the broader open-source ecosystem.

Acknowledgements

We would like to thank the SGLang Team and Community for their outstanding work on the SGLang framework. We also thank the following teams and collaborators for their support and contributions:

• Ant Group SCT Team: Yongfei Xu, Qianyu Zhang, Zekai Gu, ZhiLin Huang, Fakang Wang, Jianhao Fu, Zhuoxuan Du, Xia Zhan, Chun Huang, Qi Liu, Xi Chen, Yuhan Mao, Peipeng Cheng, Hanlin Gao, Jinghua Yao

• Ant Group Venus Team: Jinzhen Lin

• SGLang Community: Peng Zhang

Appendix A. Prefill Results

A.1 Humming PP2 Prefill: Baseline vs. Final Profile

A.2 Humming PP4 Prefill: Baseline vs. Final Profile

Appendix B. Low-Latency Decode Results

B.1 Peak TPOT Across Input Lengths and Batch Sizes

B.1.1 No-Spec PP2-TP8

B.1.2 Optimized DSpark PP2-TP8

B.2 Batch-Size-1 Output Throughput

B.3 Benchmark Settings

Hardware: one 8× H20-141GB decode node.

Decode server

--tp-size 8 \
--mem-fraction-static 0.91 \
--max-running-requests 1 \
--cuda-graph-max-bs 1 \
--cuda-graph-bs 1 \
--moe-runner-backend humming \
--moe-a2a-backend none \
--speculative-algorithm DSPARK \
--speculative-num-draft-tokens 7 \
--speculative-dspark-block-size 7 \
--speculative-moe-runner-backend triton \
--speculative-moe-a2a-backend none

Client benchmark

python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 8000 \
  --model <MODEL_PATH> \
  --dataset-name random \
  --dataset-path <DATASET_PATH> \
  --random-input-len 262144 \
  --random-output-len 4096 \
  --random-range-ratio 1.0 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --warmup-requests 0 \
  --seed 1

Output throughput is extracted from the server's TP0 Decode batch log lines; we discard the highest and lowest 20% of samples and average the remainder. The B300 number follows the setup reported in the linked source.

Appendix C. High-Throughput Decode Results

C.1 DP32-EP32 with FP8 + MTP (3, 1, 4)

C.2 DP32-EP32 with FP8 + Optimized MTP (3, 1, 4)

C.3 DP32-EP32 with FP8 + DSpark

C.4 DP32-EP32 with Humming MXFP4AFP8 + Online C128 + DSpark

C.5 DP16-EP16 with Humming MXFP4AFP8 + Online C128 + DSpark

Appendix D. Capacity Results

D.1 Decode Capacity Scaling

D.2 Humming Accuracy Validation

We evaluated the DP16-EP16 Humming MXFP4AFP8 + Online C128 + DSpark profile on GSM8K1000. The profile achieves 95.5% exact-match accuracy, with one invalid response and zero system errors, passing our 95.0% acceptance threshold.

As a public reference, the upstream SGLang Humming integration reports the following results on a 200-example GSM8K evaluation of DeepSeek-V4-Flash:

No accuracy degradation is visible for Humming MXFP4AFP8 in this public comparison. Because it uses DeepSeek-V4-Flash rather than DeepSeek-V4-Pro, we treat it as an external reference rather than a matched accuracy-loss measurement for our serving profile.