Moving on up with Scale-up

This post does not reflect the views of current, past, or future employers. The opinions in this article are my own.

Last time we talked about Ultra Ethernet which is a scale-out technology, this week I thought it’d be fun to take a quick tour of scale-up technologies.

Scale-up, shared memory, and GPU-to-GPU

Our story for scale-up really begins with memory. Ever since the beginning of computers memory has played a vital role. While CPUs may be the workhorses for performing computation, it’s memory that provides the data to be processed for those computations. It follows that application and system performance directly correlates to how efficiently memory is accessed by the CPU. For so-called “memory-bound” applications, performance can depend on memory access more than anything else. We’ve already seen one answer to optimizing performance in memory caches, and the principles of tiered memory are applicable.

Recently, GPUs have been thrown into the mix and have turned everything upside down. GPUs are the processing workhorses for AI/ML. There a number of different characteristics than CPUs that motivate a rethink of memory access:

• GPUs offer a lot of processing parallelism. The data processed by GPUs is multi-dimensional and voluminous, and the nature of the data makes it highly parallelizable which means we can throw a bunch of GPUs at the problem (i.e. there can be way more GPUs than CPUs in a system).
• Initially, GPUs were confined to a single server. Then people wanted to pool together all the GPUs in a rack, and nowadays people are looking at connecting racks of tightly coupled GPUs in clusters.
• GPUs are memory hogs. They need very high memory bandwidth to process data, especially in training for AI. But it’s not just that GPUs are passively using memory, it’s that the are actively sharing memory via *Remote Memory Access, or RMA* (a generalization of RDMA).
• GPUs need high memory throughput and super-low latency. At the high end we’re taking 1.6Tbps throughput and <1 µs latency.

The memory wall

The *memory wall* refers to the increasing gap between processor speed and memory bandwidth, where the rate of improvement in processor performance outpaces the rate of improvement in memory performance due to limited I/O and decreasing signal integrity. This disparity limits the overall system performance. The memory wall is a significant challenge in performance-intensive computing, such as AI and HPC communications.

The memory wall is a longstanding problem, but it’s exacerbated by AI and GPUs that need to access a lot of memory quickly. Improving the hardware memory path is one part of the solution we’ll talk about today, but there are others like restructuring AI applications to be more efficient or paring down the amount of memory consumed.

GPU-to-communications

Multi-GPU nodes and clusters are being commonly deployed to meet the needs of large training models and other AI applications. GPUs on the same node are connected through dedicated networks, with bandwidths up to a few terabits per second. Direct GPU-to-GPU communications are an essential part of training. One GPU may write to the memory of another, or GPUs may share atomic variables. GPU-to-GPU communications require both high throughput and very low latency.

Putting the “scale-up” in scale-up

So now we see what really puts the “scale-up” in scale-up. Adding GPUs to a compute cluster is a scale-up optimization, accordingly the memory accessed by GPUs needs to scale up and so does GPU-to-GPU communications. Scaling is not necessarily linear, for instance we may need O(N²) communication links between GPUs. Anyway you slice it, a super efficient scale-up networking is challenging, or to put it more optimistically, it’s a very active area of research and development. For today’ we’ll look at three contender technologies for scale-up networking:

• CXL
• UALink
• Scale Up Ethernet

I will note that NVLink isn’t on this list. While NVLink is a widely deployed scale-up interconnect, it’s a proprietary Nvidia technology and for this discussion we’re only considering open protocols and standards .

CXL

Compute Express Link, or CXL, is a protocol built on top of the PCIe physical layer, enabling cache and memory coherency across devices. CXL leverages the wide availability of PCIe interfaces, and it allows memory to be shared across various kinds of hardware: CPUs, NICs & DPUs, GPUs & other accelerators, SSDs, and memory devices.

CXL enables the following use cases:

• Memory expansion: CXL can help increase memory bandwidth and capacity for servers.
• Memory pooling: CXL can create memory pools where memory is disaggregated from the CPU, which, in theory, could massively increase DRAM utilization rates.
• Heterogeneous compute: ASIC’s are far more efficient than general purpose CPUs. CXL can help enable heterogeneous compute by providing a low latency cache coherent interconnect between ASIC’s and general purpose compute.
• Composable server architectures: Servers are broken apart into their various components and placed in groups where these resources can be dynamically assigned to workloads on the fly.

Before AI really emerged as the driver of data center technology, CXL was getting all the buzz as the next generation fabric. But it really hasn’t caught on as a scale-up technology for AI. There are several reasons for this, both technical and non-technical. One of the drawbacks is that CXL uses PCIe SerDes which offers lower throughput than an Ethernet SerDes which is used in the other technologies. Additionally, CXL is geared to be a memory interconnect, and not so much designed for GPU-to-GPU communications. That being said, the buzz for CXL is once may be picking up again as it could be an alternative for scale-out in cases where there’s a tradeoff for losing some performance and getting features of a consistent and cache coherent memory fabric.

Block diagram of CXL.

UALink

Ultra Accelerator Link, or UALink, for short, is an open industry standard for a memory semantic, scale-up fabric protocol developed by the UALink Consortium. UALink is touted as the industry’s answer to NVLink. As one web site put it, “Everyone Except Nvidia Forms Ultra Accelerator Link (UALink) Consortium”.

UALink is a protocol that enables load, store, and atomic operations between hundreds of GPUs, or generally accelerators. It optimizes the protocol stack to minimize end-to-end latency, reduce valuable die area on GPUs and switches, and reduce interconnect and switching power consumption. UALink is designed for very high throughput and very low latency GPU-to_GPU communications.

The UALink stack includes four layers:

• Physical Layer
• Data Link Layer
• Transaction Layer
• Protocol Layer

Physical Layer (PL)

The UALink PHY is based on 802.3 Ethernet PHY. UALink is defined for one, two, or four serial lanes running at a serial rate of 212.5G (200GBASE-KR1/CR1, 400GBASE-KR2/CR2, 800GBASE-KR4/CR4).

Data Link Layer (DL)

The Data Link packs 64-byte Flits (a small, fixed-size unit of data that is used to transmit information across a link) from the transaction layer into 640 Bytes Flits for the Physical Layer. The Data Link also provides a message service between link partners that originates and terminates at the Data Link layer.A Link level replay (LLR) is provided on a 640 Byte Flit basis. A 32-bit CRC is computed, checked, and included as part of the 640 Byte Flit.

Transaction Layer (TL)

The Transaction Layer (TL) is responsible for converting protocol messages from the inbound channels of the two protocol layer interfaces (Originator and Completer) into outbound (TX) TL Flits. The TL also converts TL Flits received from the inbound (RX) DL back into protocol layer messages on the protocol interfaces.

The Transaction Layer implements compressed addressing, streamlining data transfer with up to 95% protocol efficiency under real workloads. It also enables direct memory operations such as read, write, and atomic transactions between accelerators, preserving ordering across local and remote memory spaces.

Protocol Layer (PL)

The protocol layer for UALink is called UALink Protocol Level Interface (UPLI). UPLI defines a logical signaling interface and a protocol by which devices can exchange data and control information through a set of Request and Response messages. The format of the Request and Response messages is customizable by the application protocol.

UALink overview.

Scale Up Ethernet

As workloads continue to expand in complexity in areas such as machine learning and AI inference, the need for parallel processing power increases significantly. Scaling up a GPU (or more generically an XPU, to include various custom ML accelerators) cluster to a rack or multi-rack level is needed to enhance its ability to process larger datasets and improve overall system efficiency. Ethernet offers many advantages for designing such a fabric, from industry leading high-speed links and high-capacity switches to a well-developed ecosystem and well-understood operational methods.

Scale Up Ethernet, or SUE, provides a framework to provide low latency, high bandwidth connectivity for XPU scale up networks based on Ethernet. The intent of SUE is to provide the transport and ethernet datalink to move memory between XPU (GPUs, CPUs, TPUs, NPUs, etc.).

The SUE spec is aptly called a “framework”. It’s only twenty-seven pages long (compared to 253 pages in the UALink spec and 562 pages in the UEC spec). Unlike UALink, the SUE spec describes just the transport layer aspects of the protocol and defers much of the details about the higher and lower layers. For instance, it mentions that data encryption is the purview of the upper layer protocols. For congestion control, SUE suggests to use the link layer credit based flow control being defined by UEC.

The SUE stack is built to scale efficiently across up to 1024 XPUs, supporting strict and unordered packet flows over 800G, 400G, and 200G Ethernet ports. By utilizing new constructs like AI Fabric Headers (AIFH), Credit-Based Flow Control (CBFC), and Link-Level Retry (LLR), the system delivers lossless transport, congestion control, and simple go-back-N recovery. The modular architecture supports mesh and switched topologies, offering high configurability per application. It also integrates with Ethernet-based fabrics using low-latency SerDes, a couple of packet encapsulation options, and optional FEC tuning. SUE includes several layers of queues and the concept of virtual channels to support QoS.

Scale Up Ethernet protocol stack.

As we mentioned in our discussion of Ultra Ethernet, the UET protocol is primarily in UDP in IPv4 or IPv6 . Scale Up Ethernet defines a similar UDP encapsulation, however it also defines a protocol encapsulation directly over Ethernet. There are three encapsulation options:

• Standard Ethernet with IP{v4,v6}– Ethernet-IPv4/IPv6-UDP
• AI Forwarding Header Gen 1– compatible with Ethernet MAC but uses fewer bits from DA for forwarding
• AI Forwarding Header Gen 2– only use a few bits from Eth addresses, rest of bits in Ethaddrs are repurposed (including entropy, traffic class)

The last two options are divergences from standard Ethernet, and the last one especially would break compatibility with existing Ethernet switches. Since scale-up is basically a closed network this is feasible.

Broadcom was a founding member of the UALink Consortium but subsequently published SUE which conceivably would compete with SUE. There’s quite a bit of speculation as to why they did this. One possibility is that SUE is compatible with existing Ethernet switches allowing a shorter time to market.

Tail of the tape

We summarize the features of our three contenders plus Ultra Ethernet in the table below. As for predicting a winner in scale-up, it’s really hard to tell at this point… stay tuned I suppose!