How did Sarvam Optimize and Serve Large MOEs on NVIDIA GPUs?

Recently there was a talk presented jointly by Ashwin from Sarvam and Utkarsh Uppal from NVIDIA at the Sarvam epoch conference explaining where does the $0.80 per million tokens comes from. Not from a cleverer model but from serving engineering.

The problem?

The framing slide was titled “Entering Era of Test Time Compute.” When models started reasoning generating long chains of thought before answering output lengths exploded, and every assumption behind conventional serving stopped holding. NVIDIA laid out four specific challenges:

1. More compute and memory required. A large MoE doesn’t fit on one GPU. You need model-parallel strategies to split it across many, and every split adds communication overhead.

2. Scaling techniques don’t scale symmetrically. The talk gave a striking ratio: 40 nodes for decode, 4 nodes for prefill. To understand why that’s lopsided, you need the two phases:

• Prefill : the model reads your entire prompt at once. Highly parallel, compute-bound, over quickly.
• Decode : the model generates the response one token at a time, each token depending on the last. Sequential, memory-bandwidth-bound, and where nearly all the wall-clock time goes.

Ten times more hardware for decode than prefill. That ratio is the single most important number for anyone sizing an inference cluster.

3. Inefficient routing. Naive round-robin load balancing sends requests to whichever GPU is free ignoring which GPU already holds the relevant KV cache. The KV cache is the model’s working memory of the conversation so far; if you route a follow-up request to a GPU that doesn’t have it, that GPU recomputes everything from scratch. You’ve just paid twice for the same tokens.

4. Dynamically changing workloads. Request lengths vary enormously and unpredictably. A cluster provisioned for the average is wrong most of the time.

Disaggregated serving

The core answer: stop running prefill and decode on the same GPUs.

In monolithic serving, both phases compete for the same resources — a long prefill stalls every decode stream sharing that GPU. Disaggregated serving splits them into separate pools, each scaled and tuned independently. Hence 40 decode nodes and 4 prefill nodes.

NVIDIA’s slide listed when to reach for it:

• Model size is large and spans multiple GPUs
• You need flexible resource allocation tailored to each phase
• Traffic involves long inputs relative to outputs
• Reducing time-to-first-token latency matters
• You want to avoid bottlenecks where prefill and decode compete for the same resources
• High-concurrency use cases i.e, many users per model instance, less latency-sensitive

That last bullet is the honest caveat. Disaggregation adds a network hop for KV cache transfer between pools. If you’re serving one latency-critical user, it’s overhead. At Sarvam’s concurrency, it’s a large win.

Three pieces of infrastructure that make it work

KV-Cache Block Manager (KVBM). GPU memory (HBM) is the fastest and most expensive storage in the building, and KV cache eats it. KVBM offloads cache down a tiered hierarchy: HBM → host CPU memory → local SSD → shared network storage. Cold conversation state moves to cheap storage; hot state stays on the GPU. More concurrent users on the same hardware.

Picture of a slide taken at the conference

NVIDIA Inference Transfer Library (NIXL). Once cache lives in four different places, something has to move it fast. NIXL is a single API over DRAM, HBM, file, and object storage, with pluggable backends like UCX, GDS, S3, custom. Optimised specifically for inference data movement, consistent across heterogeneous data paths. Unglamorous, and the thing that makes disaggregation viable rather than theoretical.

NVIDIA Dynamo with SLO-based Planner. The orchestration layer that ties it together. Dynamo combines disaggregated serving, a KV-aware router (routing to the GPU that already holds your cache — fixing challenge 3), the KV block manager, low-latency transfer via NIXL, and topology-optimised serving via Grove. On top sits an SLO-based planner that does initial system profiling and time-series request forecasting, then allocates GPUs between prefill and decode to hit TTFT (time to first token — how long before the reply starts) and ITL (inter-token latency — how fast it streams after that).

It runs across PyTorch, SGLang, TensorRT-LLM, and vLLM, targeting GB200 NVL72 a rack of 72 Blackwell GPUs connected by NVLink switches that behaves as one enormous GPU. The slide showed decode and prefill partitions within the rack, with NVLink switches between them.

That’s the punchline of hardware–software codesign: the rack was built so a model can be split across it without the interconnect becoming the bottleneck, and the software was built to exploit exactly that.

Why it matters

Sarvam had already worked with NVIDIA on inference codesign before Epoch; kernel and scheduling optimisation on H100 for a 2x gain, then Blackwell plus NVFP4 weight quantisation for another 2x, roughly 4x end to end over the H100 baseline. Sarvam now says it operates India’s largest Blackwell cluster.

The economic argument closes here. Sarvam isn’t undercutting GPT-5.4 Mini by 5.5x because its model is cheaper to run in some abstract sense. It’s doing it because the model is sparse (MoE), the precision is aggressive (NVFP4), the serving is disaggregated, the routing is KV-aware, the cache is tiered, and the whole thing runs on a rack designed for this shape of workload.

Cheap tokens are an engineering achievement, not a pricing decision.