NVIDIAの極端なハードウェア・ソフトウェア協調設計がSarvam AIの主権モデルに大規模な推論向上をもたらした方法

As global AI adoption accelerates, developers face a growing challenge: delivering large language model (LLM) performance that meets real-world latency and cost requirements. Running models with tens of billions of parameters in production, especially for conversational or voice-based AI agents, demands high throughput, low latency, and predictable service-level performance. For startups building sovereign AI models from scratch, these challenges are amplified by the need to balance model scale and accuracy with infrastructure efficiency—while also maintaining data sovereignty and cost control.

Sarvam AI, a generative AI startup based in Bengaluru, India, set out to build large, multilingual, multimodal foundation models that serve its country’s diverse population, support nearly two-dozen languages, and keep model development and data governance fully under India’s sovereign control. To meet strict latency targets and improve inference efficiency for its flagship Sovereign 30B model, Sarvam AI collaborated with NVIDIA to co-design hardware and software optimizations.

This collaboration delivered a 4x speedup in inference performance on NVIDIA Blackwell over baseline NVIDIA H100 GPUs, and established a path for deployment on the next-generation NVIDIA Blackwell architecture. The end-to-end performance boost was achieved through kernel and scheduling optimizations on NVIDIA H100 SXM GPUs that contributed a 2x speedup. That was combined with the powerful compute capabilities of Blackwell, along with NVFP4 weight quantization, for an additional 2x speedup, with an even bigger performance gain of 2.8x seen at higher interactivity points.

NVIDIA engineers helped Sarvam AI build 3B, 30B, and 100B foundational models, and optimize a new family of sovereign foundation models that were trained using NVIDIA Nemotron libraries, including the NVIDIA NeMo Framework and NVIDIA NeMo-RL. These models support 22 Indian languages, English, math, and code. They demonstrate how developer teams can leverage NVIDIA’s full-stack AI platform—from data to deployment—to achieve state-of-the-art performance and localized AI capabilities.

This post walks through the joint engineering effort and shares benchmarks for the speed-ups achieved on the NVIDIA H100, the largest-deployed NVIDIA GPU in India. We also provide an early look at how these workloads are being adapted for the NVIDIA Blackwell architecture.

Making multilingual sovereign AI scalable with MoE

To deliver sovereign-scale intelligence with high efficiency, the Sarvam AI models employ a sophisticated heterogeneous mixture-of-experts (MoE) architecture tailored for deep reasoning and linguistic density. These models were pretrained from scratch across 3B, 30B, 100B using the NVIDIA NeMo framework and NVIDIA Megatron-LM. Furthermore, Nemo-RL was used for post-training workflows for these models including long-context reasoning.

Sarvam 30B utilizes a 19-layer depth (1 dense + 18 MoE) with 128 experts and a top-6 routing strategy, relying on grouped query attention (GQA) to balance memory bandwidth with generation quality.

Sarvam 100B scales this design to 32 layers (1 dense + 31 MoE) and employs top-8 routing over 128 experts with a larger MoE FFN hidden size of 2048. Additionally, the 100B model adopts multi-head latent attention (MLA)—similar to DeepSeek-V3—to aggressively compress the Key-Value (KV) cache, enabling massive context windows without the memory penalties of standard attention.

Both models feature a shared expert design where a dedicated expert handles common features while routed experts tackle specialized tasks. This combination of high active parameter counts (via top-6/top-8 routing) and complex memory access patterns created a unique serving challenge, necessitating the deep kernel optimizations on NVIDIA Hopper and NVIDIA Blackwell GPUs detailed below.

The performance challenge: SLAs and baseline configuration on NVIDIA H100

Optimizing the Sarvam 30B model wasn’t just about raw speed; it was about maximizing density under strict latency constraints. For the applications served by this model—voice-to-voice agents—we established the following service level agreements (SLAs):

P95 (95th percentile) time to first token (TTFT): < 1000 ms

P95 (95th percentile) inter-token latency (ITL): < 15 ms

P95 (95th percentile) in inference performance testing measures latency, indicating that 95% of served requests are completed faster than this threshold, while the slowest 5% take longer. It is a critical tail-latency metric used to evaluate user experience and system stability, ensuring that even under load, most users face no more than a specific delay. The engineering goal was to maximize the inference server’s token throughput (concurrently served requests) without breaching these P95 targets.

For the initial performance analysis, the Sarvam AI and NVIDIA teams selected the SGLang inference engine for their initial performance analysis. Unlike standard serving frameworks that treat the KV cache as a linear buffer, SGLang implements RadixAttention—a mechanism that manages the KV cache as a radix tree. This was critical for the Sarvam 30B architecture; RadixAttention enables automatic prefix sharing, allowing the shared expert context and system prompts to be computed once and reused across concurrent requests. Furthermore, SGLang’s Cache-Aware Scheduler maximizes the hit rate of these shared prefixes, significantly reducing redundant memory operations during the prefill phase.

The Sarvam AI and NVIDIA teams modeled a production traffic profile characterized by an average input sequence length (ISL) of 3,584 tokens and an output sequence length (OSL) of 128 tokens. Guided by internal simulation data, we deployed the model on two NVIDIA H100 SXM GPUs with a specific parallelism strategy designed to balance the distinct memory and compute requirements of the MoE layers:

Expert parallelism (EP=2) for the expert weights. This configuration utilizes Grouped GEMM kernels to maximize compute density and ensures that the massive expert weights reside in HBM, reducing the cost of expert routing.

Data parallelism (DP=2) for the attention weights with –enable-dp-attention. This enabled us to parallelize attention computation across parallel batches, significantly boosting the aggregate throughput of the prefill phase.

While this configuration provided a robust functional baseline, profiling revealed that satisfying the sub-second TTFT at high concurrency required deeper optimization – leading us to the specific kernel and precision strategies detailed below.

From profiling to performance: eliminating MoE bottlenecks

Simulation data indicated that a concurrency range of 32 to 64 requests would offer the best chance of meeting SLA requirements. To identify the precise bottlenecks limiting token throughput in this concurrency range, the NVIDIA and Sarvam AI teams utilized NVIDIA Nsight Systems to capture execution traces of both the prefill and decode phases at a concurrency of 32 requests. We then processed the traces to extract the microsecond-level latency contribution of every kernel within a single transformer layer.

The profiling revealed that while the heavy General Matrix Multiplication (GEMM) operations (experts and attention) were performing well, significant latency bubbles existed in the non-compute-intensive operations—specifically in the MoE routing logic and positional embedding calculations. These operations were suffering from kernel launch overheads and redundant memory reads.