How a Packet Travels from a NIC into GPU Memory: GPUDirect RDMA Explained

Story so far, kind of :)

For the last few months of my internship at NI/Emerson, I’ve been getting IQ samples out of a software-defined radio (SDR) and straight into GPU memory, without the CPU ever laying a finger on them. This is the first of a new series where I unpack what I actually built, one piece at a time.

Los geht’s!

Some context first —

SDR throws out I/Q samples very, very fast. The obvious way to get them to a GPU is: NIC → kernel buffer → CPU copy → GPU. Every arrow in that sentence is your latency budget quietly leaking away.

RDMA offers a different deal: skip the CPU and the kernel almost entirely, let the NIC write straight into memory. Pair that with Nvidia’s GPUDirect RDMA, and “memory” can mean GPU memory. The NIC and the GPU start talking like two coworkers who’ve stopped cc’ing the CPU on every email.

[embed]NIC to CPU every, single, time :P

So, for visual representation, this is how RDMA looks as compared to a TCP communication —

Source — Everything You Wanted to Know About RDMA But Were Too Proud to Ask

Setup for context (skip if you just want the mechanics) — A server, Nvidia RTX A4000, a Mellanox NIC doing RoCEv2 (RDMA over Converged Ethernet v2), with libibverbs handling the plumbing underneath.

Before we jump to what happens when one packet lands on the GPU, we should come across some terminologies used in RDMA that will be used in future blog posts as well —

• Queue pairs — To draw an analogy from everyday mail service, a queue pair (QP) defines the address of the communication endpoints, or equivalently, sockets in traditional socket-based programming. Each communication endpoint needs to create a QP to talk to each other

• Work requests — Instructions posted by applications to Queue pairs to initiate data transfers(send/receive)

• Receive queue — A circular buffer in memory, managed by the NIC, that holds work requests pointing to pre-allocated buffers for incoming messages

• Send queue — A circular buffer in memory, managed by NIC, that holds work requests pointing to pre-allocated buffers for outgoing messages

• Completion queue — A queue consisting of completion events for each send/receive operation completed

• SGE buffers — data structures used to define memory buffers for data transfers; used to represent non-contiguous memory locations on host memory or device memory (GPU memory). Each SGE inside a WR can be of different sizes. An SGE can belong both to a receive queue (RQ) and a send queue (SQ).

1. So what actually happens when one packet lands? —

I’ll be honest, I went into this internship thinking GPUDirect RDMA was basically “NIC writes to GPU, magic happens.” Turns out it’s a very deliberate four-stage hardware pipeline, and once you see it laid bare, the magic evaporates and gets replaced by mild awe at how many components had to agree to cooperate.

[embed]

Stage 1 — Packet parsing at the NIC: The NIC pulls the source IP, destination IP, and queue pair (QP) number out of the incoming RoCEv2 packet. The QP number is the important bit; it’s basically the packet’s forwarding address, telling the NIC exactly which memory region this payload is meant for.

Stage 2 — PCIe TLP generation: The NIC’s PCIe controller repackages that payload into Transaction Layer Packets (TLPs), the basic unit of anything moving across a PCIe fabric. Each TLP carries an address, a command (a memory write, here), and the data itself, like a tiny courier with very specific delivery instructions and zero small talk.

Stage 3 — Memory controller routing. Here’s the clever part. The memory controller looks at the destination address on the TLP. If that address falls inside a GPU BAR region (because nvidia-peermem registered it via ibv_reg_mr), the TLP gets routed to the GPU instead of host DRAM. Same PCIe mechanism the NIC would use to write to host memory, just quietly redirected somewhere else entirely; no one else in the building even notices.

Stage 4 — GPU BAR1 write to VRAM: The GPU’s own PCIe controller sees the address falls within its BAR1 aperture and writes the payload straight into VRAM. At this point your raw I/Q samples are sitting in GPU memory, ready for a CUDA kernel, and the CPU has, at no point, been informed that any of this happened. Rude, honestly. But fast.

[embed]Sorry CPU

Small but important detail — BAR1 isn’t infinite, at least in my setup. That number quietly decides how many SGE buffers you can register at once, a fact I learned the slightly hard way, staring at a very confusing error message. :’)

3. One thing that quietly deprecated itself while I wasn’t looking —

nvidia-peermem is the kernel path I described in Stage 3, and it's how most GPUDirect RDMA writeups (including a bunch I read before starting) explain the mechanism. Newer kernels are moving to dma_buf instead, a generalized file-descriptor-based framework for sharing buffers between device drivers, originally introduced in Linux 3.3 for entirely different purposes and now living a quiet second life here. Worth checking which path your own kernel is actually taking before you go debugging the wrong layer for three hours as a certain someone may have. 🤔 (me ?)

That’s the plumbing. Next post picks up right where the packet lands: I’ll walk through where the actual latency hides.

Ciao!! :)