GPU Direct RDMA: Eliminating the CPU from the Data Path

The Standard (Slow) Path

In a conventional server, when GPU A on Server 1 needs to send data to GPU B on Server 2, the data takes a long, CPU-mediated journey:

1. GPU A copies data from its HBM into CPU System RAM (via PCIe).
2. The CPU reads the data from System RAM.
3. The CPU copies the data into the Network Interface Card (NIC) transmit buffer.
4. The NIC sends the data over the network.
5. On the receiving side, the process reverses: NIC → CPU → System RAM → GPU B's HBM.

This path involves multiple data copies, CPU interrupts, and OS kernel context switches. Each copy adds latency and consumes CPU cycles that could be doing useful work.

RDMA: Remote Direct Memory Access

RDMA is a technology that allows one machine's network card to read from or write to another machine's memory without involving the remote CPU. The CPU on the receiving machine is not interrupted — the data just appears in memory.

GPU Direct RDMA extends this concept to GPU memory: the NIC can read data directly from the GPU's HBM and write data directly into the GPU's HBM, bypassing both the CPU and System RAM entirely.

Standard Path:  GPU HBM → PCIe → CPU RAM → CPU → NIC → Network
GPU Direct RDMA: GPU HBM → PCIe → NIC → Network

The benefits are significant:
- Zero extra copies: Data moves from GPU HBM to the wire in one step.
- Lower latency: No OS kernel involvement, no CPU scheduling delays.
- Lower CPU overhead: The CPU is free to do other work (or sleep, saving power).

Hardware Requirements

GPU Direct RDMA requires specific hardware alignment:

1. RDMA-capable NIC: NVIDIA ConnectX series (InfiniBand or RoCE), or equivalent.
2. PCIe topology alignment: The NIC and the GPU must be on the same PCIe root complex (same PCIe switch subtree). If the GPU is on PCIe switch A and the NIC is on PCIe switch B, data must cross the CPU — defeating the purpose.
3. Driver support: The nvidia-peermem kernel module must be loaded to enable the GPU Direct RDMA path.

Verifying GPU Direct RDMA Is Active

Two checks confirm whether the fast path is actually active. First, whether the kernel module is loaded at all — checking the module list for nvidia_peermem. Second, whether the GPU and NIC are topologically close enough to benefit — the GPU interconnect topology matrix reports each GPU/NIC pair's connection type, and a PIX between a GPU and NIC means they share a PCIe switch (GPU Direct RDMA is enabled); a PHB means the path still crosses the CPU, disabling the fast path.