llm-d Is Now in the CNCF Sandbox. Here’s What Disaggregated LLM Serving Actually Means

Most teams running LLMs on Kubernetes have the same setup: vLLM behind a load balancer, requests distributed round-robin, GPU nodes as fungible compute. It works. Until you look at your time-to-first-token at scale, or your KV cache hit rate, and realize you’ve been leaving a lot of performance on the table.

llm-d is the project trying to fix that. It just landed in the CNCF Sandbox, donated by IBM Research, Red Hat, and Google Cloud at KubeCon Europe 2026. Backed by NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, and Mistral AI. The version shipping today is v0.7, with predicted-latency scheduling now GA.

Here’s what it actually does — and why the architecture matters more than the announcement.

What “Disaggregated Serving” Means in Practice

Standard LLM inference has two distinct phases: prefill (processing the input tokens, building the KV cache) and decode (generating output tokens one at a time). These phases have very different resource profiles.

Prefill is compute-bound: lots of parallel GEMM operations, high GPU utilization for a short burst. Decode is memory-bandwidth-bound: you’re reading the KV cache repeatedly for each token, and throughput depends on how fast you can move data off GPU DRAM.

Running both phases on the same GPU is a compromise. The compute-heavy prefill and the memory-bandwidth-heavy decode are competing for different things on the same hardware.

Disaggregated serving splits them: prefill workers handle the input processing on compute-optimized nodes, then hand off the KV cache to decode workers on memory-bandwidth-optimized nodes. On a 16×16 B200 prefill/decode topology, llm-d benchmarks show ~50,000 output tokens/second with order-of-magnitude TTFT reduction versus round-robin baseline — approximately 3,100 tokens/second per B200 decode GPU.

That’s not a tuning improvement. That’s an architectural difference.

The Round-Robin Problem: Why Your KV Cache Hit Rate Is Low

Here’s the part that stings when you think about it. If you’re running four vLLM instances behind a round-robin load balancer, every incoming request has a 75% chance of landing on a node that doesn’t have its prefix cached. You’re recomputing the same KV cache repeatedly across nodes.

Prefix caching works at the node level. The cache is local. Without intelligent routing, multi-node setups destroy the cache hit rate that makes prefix caching valuable.

llm-d solves this with prefix-cache-aware routing: the scheduler knows which decode workers hold which KV cache blocks. When an incoming request shares a prefix with a recently cached request — a shared system prompt, a RAG context, a multi-turn conversation — the scheduler routes it to the node that already has that cache. Less redundant prefill, lower TTFT, higher throughput.

This is the thing round-robin can’t do. The load balancer has no idea what’s in each node’s KV cache.

What llm-d Adds on Top of Kubernetes

llm-d doesn’t replace vLLM — it orchestrates it. The architecture has three layers:

The inference pods run vLLM (prefill and decode workers as separate deployments). The v0.7 kustomize-first approach means you’re working with standard Kubernetes manifests, not a proprietary operator DSL.

The scheduler sits between the gateway and the inference pods. It tracks KV cache state, predicted decode latency, and worker load — and routes accordingly. Predicted-latency scheduling hit GA in v0.7, which means this is now production-ready.

Hierarchical KV cache offloading moves cold cache blocks from GPU DRAM → CPU DRAM → disk or remote storage, using LRU eviction. Active blocks stay hot. Old blocks get offloaded without being dropped. This extends effective KV cache capacity significantly without adding GPUs.

Additionally, v0.7 introduced a batch gateway (experimental) for handling burst workloads, active-active HA, and UCCL-based transport resilience. Nightly CI now runs against OpenShift, GKE, and CoreWeave — the compatibility surface is real.

The CNCF Sandbox Signal

When a project gets donated to the CNCF, it goes through a due diligence process: governance, security posture, community health, vendor neutrality. The Sandbox stage means CNCF has accepted it as worth investing in — not that it’s production-hardened, but that it’s on a trajectory toward that.

For infrastructure teams, the practical implication is: this isn’t going to be quietly deprecated by a single vendor. IBM Research, Red Hat, and Google Cloud are co-invested. NVIDIA and CoreWeave are founding partners. The project has enough industry weight behind it to develop a stable API surface.

The CNCF Sandbox also puts llm-d on the same trajectory as projects like Argo CD, KEDA, and OpenTelemetry — projects that started as sandbox and became foundational.

What to Do With This Today

You don’t need to rip out your existing vLLM setup. The path is additive:

1. Audit your KV cache hit rate. If you’re running multi-node vLLM with round-robin, check your prefix cache metrics. A low hit rate (under 40%) on shared-prefix workloads is a strong signal for intelligent routing.
2. Try the v0.7 kustomize base. The project docs have a kustomize-first deployment guide that layers onto existing EKS or GKE clusters. It’s less invasive than it looks.
3. Evaluate the prefill/decode split for your workload. If your average input is long (RAG, multi-document, agent traces) and your decode steps are short, disaggregated serving helps more. If your workloads are short-context chat, the overhead may not be worth it yet.
4. Watch the batch gateway. It’s experimental in v0.7 but the design is right for burst inference workloads. That’s the piece most teams will want for handling traffic spikes without over-provisioning.

Key Takeaways

• Disaggregated serving separates prefill (compute-bound) from decode (memory-bandwidth-bound), enabling hardware specialization at each phase
• Round-robin load balancing kills KV cache hit rates at scale — prefix-cache-aware routing is what fixes it
• llm-d benchmarks show order-of-magnitude TTFT improvement over round-robin baseline on B200 hardware
• The CNCF Sandbox donation means multi-cloud governance and no single-vendor lock-in
• v0.7’s kustomize-first deployment lowers the barrier to trying this on existing EKS/GKE clusters

The right question isn’t “should we use llm-d?” It’s “at what scale does the routing intelligence pay for the operational complexity?” For most teams, that crossover is earlier than you’d expect.

If you’re running inference at scale and want to compare notes on KV routing strategies, happy to connect.