How We Solved the KV Cache Bottleneck in LLM Inference with CXL Shared Memory

LLM inference has quietly shifted from a compute problem into a memory problem.

A few years ago, serving a model meant fitting weights onto a GPU and running matmuls fast enough. Today, context windows have stretched from a few thousand tokens to hundreds of thousands — sometimes millions. Agents replay long histories, RAG pipelines stuff in entire document sets, and chat sessions stay alive across thousands of turns. Every one of those tokens leaves behind a Key/Value vector that the model needs again on the next step.

That’s the KV Cache. And it has grown into the dominant memory consumer of modern inference.

Take LLaMA3–70B with a 128K context window. The KV Cache alone eats 40 GB of GPU VRAM — nearly an entire A100’s worth of memory, not for model weights, not for activations, but simply for caching previous tokens. Push to 1M context and you’re looking at 320 GB. For a cache.

The fact that TurboQuant — a KV cache quantization technique — blew up recently is itself proof of the problem. Compression breakthroughs rarely get this kind of attention unless the underlying data has become so painfully large that the entire industry starts paying attention. The hype around shrinking KV cache only confirms how desperately everyone needs massive storage for it.

Two Problems That Won’t Go Away

Problem 1 — capacity: where do you store gigabytes of KV cache?

GPU VRAM is expensive and finite. Offloading is obvious. The question is to where.

• CPU DRAM (80–140 ns): Fast, but a few hundred GB per server doesn’t cut it when you’re sharing KV Cache across dozens of instances.
• SSD (20–50 us): Plenty of capacity, but 100x slower than DRAM. Your TTFT takes a direct hit.

We needed DRAM-like speed with SSD-like capacity. That tier didn’t exist.

Problem 2: How do you transfer KV Cache without killing latency?

LLM serving doesn’t run on a single instance. Two architectures make KV Cache transfer unavoidable:

Context Caching — Multiple requests share the same prefix (system prompts, few-shot examples). Cache that prefix once, reuse it everywhere. But the cache needs to be accessible from any instance.

Prefill-Decode Disaggregation — Prefill is compute-heavy, decode is memory-heavy. Splitting them into separate instances makes sense for utilization — but the KV Cache from prefill has to reach the decode node. Fast.

Both directly impact TTFT (Time To First Token) — one of the metrics that matters most in production.

What we wanted: a KV Cache store that’s large, fast, and directly accessible by multiple instances without copying.

Enter CXL

CXL (Compute Express Link) is an open industry-standard interconnect built on PCIe. The key protocol is CXL.mem — it lets you access device-attached memory with load/store instructions, exactly like local DRAM.

Plug a CXL memory expander into a PCIe slot, and the system treats it as additional DRAM. No special APIs — just mmap and go.

If you’re new to CXL, check out our previous post, CXL Unpacked, where we explain the core protocols, device types, memory expansion, pooling, and sharing. Here, we focus on why those capabilities matter for KV Cache.

CXL also scales beyond a single host. Starting with CXL 2.0, switches enable multiple hosts to share the same memory pool. CXL 3.0 takes this further with fabric-level connectivity — multiple memory devices and multiple hosts connected through a CXL fabric, forming a shared memory pool accessible by any node in the cluster. This is what makes CXL fundamentally different from a simple memory expander: it’s a shared memory interconnect.

Why it solves both problems at once

For capacity: CXL Type-3 memory expanders provide hundred-GB scale per device. XCENA’s device supports 256 GB DIMMs for up to 2 TB per unit. Add memory independently of CPU DRAM slots — no server redesign. With fabric-attached CXL, this extends to multi-TB shared memory pools accessible across multiple nodes.

For latency: 170–250 ns on a single host. 300–400 ns fabric-attached. Slightly slower than local DRAM (80–140 ns), but same nanosecond ballpark. Compare that to RDMA at 2–4 us or SSD at 20–50 us.

And the critical part: no data copying. Multiple CPUs and GPUs can directly access the same physical CXL memory. Instead of transferring KV Cache over a network, every instance just reads from and writes to the same pool.

CXL creates a new tier between DRAM and the network — exactly where KV Cache storage belongs.

Maru: The CXL-based Storage Engine We Built for Open Ecosystem

Maru is a KV Cache storage engine built on CXL shared memory. The core principle: data doesn’t move.

Traditional KV Cache sharing copies data across the network. Maru flips this — data is written to CXL memory once and stays there. Only metadata (keys and location info) travels between components.

Control Plane vs. Data Plane

Maru separates two concerns:

Data Plane — where KV Cache bytes live. Each inference instance’s Maru Handler directly accesses CXL memory. No intermediate server ever touches the data. Zero-copy.

Control Plane — where metadata lives. A lightweight MaruServer manages KV key → storage location mappings. Handlers exchange only these pointers — tens of bytes per operation.

Plugging Into the Ecosystem

A storage engine is only useful if it works with the tools people already use.

vLLM + LMCache

If you’re running LLMs in production, you probably know vLLM — the most widely adopted open-source inference engine. And LMCache — a dedicated KV cache management system developed by the LMCache Lab at the University of Chicago and integrated with vLLM, handling serialization, compression, token matching, and multi-tier storage. Together they form the core of the vLLM Production Stack.

LMCache’s pluggable storage architecture makes adding new backends straightforward. Maru implements LMCache’s backend interface as the maru:// scheme. This is already merged into the official LMCache repo (PR #2705).

vLLM Direct

MaruKVConnector — a direct implementation of vLLM’s KVConnectorBase_V1.

Performance

We benchmarked Maru against LMCache’s P2P backend (data transfer via TCP) in a single-node setup with a CXL Type-3 memory expander.

Metric: Query TTFT P50 — time to first token when reusing KV Cache cached in the other instance.

The results: Maru’s advantage grows dramatically with context length. TCP transfer scales linearly with data size — more data, more time copying. Maru’s mmap-based access has virtually no copy overhead.

The fundamental difference:

• TCP P2P: GPU → CPU → TCP transfer → CPU → GPU
• Maru: GPU → CXL → GPU

What’s Next

Maru today runs on single-node CXL memory. Here’s where we’re heading:

Multi-node sharing — KV Cache sharing across nodes using CXL switches and RDMA, with benchmarks to validate cross-node latency.

Near Data Processing (NDP) — CXL’s coherency protocol keeps memory access consistent between host and device. That means compute cores embedded directly in CXL memory can safely operate on the same data the host sees. XCENA’s CXL devices include RISC-V cores that enable KV Cache operations like compression and quantization directly on-device.

SSD-backed Memory Expansion — XCENA’s CXL device natively supports SSD-based DRAM expansion: SSD-scale capacity at near-DRAM latency. Pair that with PCIe 6.0 bandwidth and proactive prefetching, and you get a virtually unlimited memory tier for KV Cache — what we call InfiniteMemory.

CXL memory isn’t just “slower DRAM.” With built-in compute and storage tiering, it becomes an intelligent memory layer purpose-built for KV Cache workloads.

Wrapping Up

As LLM inference scales, KV Cache has evolved from a simple buffer into a large-scale data management problem. GPU VRAM is too expensive for it, CPU DRAM is too small, SSDs are too slow, and network transfer adds too many copies.

CXL sits in the sweet spot — DRAM-class latency, TB-scale capacity, zero-copy shared access. Maru makes CXL practical as a KV Cache storage engine, with native integration into vLLM, LMCache, and SGLang.

The data doesn’t move. The instances just read.

https://xcena.com https://xcena-dev.github.io/maru