The Memory Pool Nobody Saw Coming

The AI memory hierarchy has a missing tier. CXL 4.0 is building it.

A Gap Between Tiers

In our analysis of the STX storage architecture, we described how NVIDIA created a G3.5 context memory tier, a flash-based layer managed by BlueField-4 DPUs, to hold KV cache at petabyte scale when GPU HBM overflows. That tier solves the problem of where to put evicted context. But there is a second memory problem, closer to the GPU and more fundamental to inference economics, that STX does not address: the gap between GPU HBM and everything else.

HBM4 on a Rubin GPU operates at nanosecond latency with terabytes-per-second bandwidth. CPU system RAM (LPDDR5X) offers microsecond latency at hundreds of gigabytes per second. The next tier, NVMe SSDs, whether local or in a CMX enclosure, operates at millisecond latency.

Between the microsecond DRAM tier and the millisecond flash tier, there is a 200–500x latency gap and no intermediate layer purpose-built for AI workloads.

CXL 4.0 Fills the Gap

CXL 4.0, released by the CXL Consortium in November 2025, addresses that gap directly. It creates what amounts to a G2.5 tier: cache-coherent, pooled DRAM accessible across multiple servers and racks at 200–500 nanosecond latency. That is 200–500x faster than NVMe, 4–5x cheaper per gigabyte than GPU VRAM, and for the first time in the CXL lineage, scalable to 100+ terabytes across multi-rack fabrics through bundled ports delivering 1.5 TB/s per logical connection.

Early results validate the approach. At Supercomputing 2025, XConn Technologies and MemVerge demonstrated CXL memory pooling for KV cache offload integrated with NVIDIA Dynamo and NIXL, achieving greater than 5x performance improvement versus SSD-based caching and 3.8–6.5x speedup versus RDMA-based alternatives. Microsoft launched its first CXL-equipped cloud instances in November 2025. SK Hynix demonstrated a memory-centric AI machine connecting multiple servers and GPUs through CXL pooled memory without traditional networking.

The CXL market is projected to reach $15 billion by 2028, with over $12 billion of that in DRAM behind CXL controllers. The memory hierarchy for AI inference is being rewritten, and the new tier sits in a latency/cost sweet spot that neither HBM nor flash can occupy.

Figure 1 — The complete AI memory hierarchy with both new tiers. G2.5 (CXL 4.0 pooled memory) fills the latency gap between system DRAM and NVMe with cache-coherent, shared memory at 200–500ns. G3.5 (CMX/STX) fills the capacity gap between node-local NVMe and enterprise storage with petabyte-scale flash optimized for ephemeral KV cache. Together, they give Dynamo’s KV block manager a five-tier hierarchy where every evicted prefix has a right-sized home.

What CXL 4.0 Actually Specifies

CXL 4.0 is worth understanding in detail because the specification changes are not incremental. They are what make multi-rack memory pooling for AI workloads architecturally viable for the first time.

Bandwidth and Bundled Ports

The headline improvement is bandwidth: CXL 4.0 doubles the transfer rate from 64 GT/s (CXL 3.x on PCIe 6.0) to 128 GT/s on PCIe 7.0, with zero added latency. The most significant new feature is the bundled port. It aggregates multiple physical CXL device ports into a single logical entity, allowing a host and a Type 1 or Type 2 device to combine connections while presenting a single device to system software.

A bundled port using an x16 link at 128 GT/s delivers 768 GB/s in each direction, for a total of 1.536 TB/s of full-duplex bandwidth between CPU and accelerator. For context, HBM3e on an H200 delivers 4.8 TB/s. A bundled CXL 4.0 connection at 1.5 TB/s represents roughly 30% of that bandwidth, which is sufficient for memory expansion use cases where capacity matters more than peak bandwidth.

Multi-Rack Reach and Protocol Continuity

CXL 4.0 also introduces native x2 link widths for increased fan-out, meaning more devices per switch port, and support for up to four retimers to extend the physical reach of PCIe signals. That extended reach is what enables multi-rack configurations.

The specification retains backward compatibility with CXL 3.x, 2.0, 1.1, and 1.0, and preserves the 256-byte FLIT format introduced in CXL 3.x. The three CXL protocol layers remain intact: CXL.io for discovery and configuration, CXL.cache for device-to-host coherent access, and CXL.mem for host-to-device coherent memory access. CXL.mem is the decisive layer for memory pooling.

The practical result: CXL 4.0 allows a cluster of AI servers to access a shared pool of over 100 terabytes of DRAM with hardware-enforced cache coherency across racks, at latencies 200–500x lower than NVMe and at a cost per gigabyte that is 4–5x lower than GPU VRAM.

Figure 2 — CXL 4.0 memory pooling architecture for AI inference. Multiple GPU servers and analytics nodes connect through a CXL 4.0 switch fabric to a shared memory pool exceeding 100 TiB. The pool holds warm KV cache, context assembly results, agent intermediate state, and elastic overflow — all at 200–500ns latency with hardware cache coherency. This is the G2.5 tier that sits between local DRAM and NVMe in the memory hierarchy.

Disaggregated Inference and the KV Transfer Problem

The Network Bottleneck in Prefill/Decode

The relevance of CXL 4.0 to the inference problems we have been tracking across this series is direct. In our analysis of the disaggregated inference stack, we described how Dynamo separates prefill (compute-bound) and decode (memory-bound) across separate GPU pools. The operational challenge with disaggregated serving is that when prefill completes on one GPU, the KV cache it produced must be transferred to a different GPU for decode.

Today, that transfer happens via RDMA over Ethernet or via CPU system memory as a staging area. Both approaches add latency and consume network bandwidth.

Turning a Network Operation into a Memory Operation

CXL 4.0 memory pooling offers a fundamentally different path. The prefill GPU writes its KV output directly to the shared CXL pool at hardware-coherent DRAM speeds, and the decode GPU reads it from the same pool without any network transfer, serialization, or copy.

The XConn/MemVerge demonstration at Supercomputing 2025 proved this works in practice. Disaggregated prefill/decode with Dynamo and NIXL running against a shared CXL memory pool achieved the 3.8–6.5x speedup over RDMA reflected in the figures above.

The implication for inference serving architectures is significant. CXL memory pooling turns KV cache transfer between prefill and decode pools from a network operation into a memory operation, eliminating an entire class of latency and bandwidth bottleneck from the disaggregated serving pipeline.

For the GPU utilization problem we analyzed earlier in this series, this matters because every microsecond a decode GPU spends waiting for KV data to arrive from the prefill GPU is a microsecond of idle compute. At $4 million per rack, those microseconds accumulate quickly.

Figure 3 — CXL specification evolution from 1.0 to 4.0. Each generation added a critical capability: 1.x introduced coherent attach, 2.0 added pooling and switching, 3.x enabled fabric-scale with multi-level switching and memory sharing, and 4.0 doubles bandwidth to 128 GT/s while adding bundled ports for 1.5 TB/s connections and retimers for multi-rack reach. The progression from single-device coherent attach to 100+ TB multi-rack memory pools took six years.

What MPP Databases Figured Out Thirty Years Ago

A Familiar Architecture

There is a parallel here that enterprise database architects will recognize immediately. CXL 4.0 memory pooling is solving a problem that MPP databases solved thirty years ago.

A production MPP database manages a distributed memory hierarchy across hundreds of nodes: each node has its own local buffer pool (analogous to GPU HBM), the buffer pool overflows into local disk (analogous to NVMe), and intermediate query results, spool space in MPP terminology, must be shared across nodes during parallel joins, aggregations, and sorts.

The techniques these systems use to manage this hierarchy are structurally identical to what Dynamo’s KV block manager does with the new memory tiers. Hot data stays in fast local memory, warm data migrates to a shared intermediate tier, cold data drops to durable storage, and the placement engine makes continuous decisions about what belongs where based on access frequency, query plan, and cost.

Convergence Confirmed by Research

The CXL-powered database research emerging from Tsinghua, SAP HANA, and Alibaba confirms this convergence. Papers published in 2024–2025 describe hybrid buffer pools where CXL-attached memory serves as a dynamically expandable tier between DRAM and SSD, with access-pattern-aware migration policies that promote hot pages to faster tiers and demote cold pages to cheaper ones. That is exactly the buffer management logic that production MPP databases have refined for decades.

The Memstrata system manages a three-tier hierarchy, local DRAM at 50ns, CXL memory at 200ns, local SSD at 10μs, with policies that are formally equivalent to the hot/warm/cold data placement that MPP workload management has performed at scale since the 1990s.

Figure 4 — The structural parallel between MPP database buffer management and AI inference memory tiering. Buffer pool = GPU HBM. Spool space = CXL pooled memory. Persistent storage = CMX/NVMe. Placement intelligence = Dynamo KV block manager. The problems are isomorphic; the AI infrastructure world is converging on solutions that MPP databases refined over thirty years.

The Lesson for Enterprise Teams

The lesson for enterprises is straightforward. CXL 4.0 does not require inventing new memory management paradigms. It requires applying well-understood tiered memory management, the kind that has been running production analytical workloads for a generation, to the AI inference memory hierarchy.

Sources: CXL Consortium “CXL 4.0 Specification Release” (November 18, 2025) · CXL Consortium “Introducing CXL 4.0” White Paper (November 2025) · XConn Technologies & MemVerge “CXL Memory Pool for KV Cache Using NVIDIA Dynamo” SC25 demonstration (November 2025) · CXL Consortium “Overcoming the AI Memory Wall: How CXL Memory Pooling Powers the Next Leap” (November 2025) · Blocks & Files “CXL 4.0 doubles bandwidth and stretches memory pooling to multi-rack setups” (November 2025) · SDxCentral “CXL Consortium debuts 4.0 spec to push past bandwidth bottlenecks” (November 2025) · Guo et al. “A CXL-Powered Database System: Opportunities and Challenges” (ICDE 2025) · Weisgut “CXL Memory Performance for In-Memory Data Processing” (VLDB 2025) · Ahn et al. “An Examination of CXL Memory Use Cases for In-Memory Database Systems” (VLDB 2024) · Zhong et al. “Memstrata: Buffer Management in CXL-Enabled Systems” (OSDI 2024)