The Multi-Node Problem
NVLink solves intra-node GPU communication. But training at scale requires multiple servers, each with multiple GPUs. Data must travel from GPU A on Server 1 to GPU B on Server 2 over a network (InfiniBand or RoCE).
The Naive Path (without GPUDirect):
GPU A (Server 1)
→ GPU VRAM → PCIe → CPU RAM (Server 1) [GPU-to-CPU copy]
→ CPU RAM → NIC (Server 1) [CPU-to-NIC DMA]
→ Network
→ NIC (Server 2) → CPU RAM (Server 2) [NIC-to-CPU DMA]
→ CPU RAM → PCIe → GPU VRAM (Server 2) [CPU-to-GPU copy]
GPU B (Server 2)
This involves four DMA operations, two PCIe crossings, and significant CPU involvement. The CPU must orchestrate each step, consuming CPU cycles and adding latency.
GPUDirect RDMA:
GPUDirect RDMA (Remote Direct Memory Access) allows the NIC to read from and write to GPU VRAM directly, without staging data through CPU RAM:
GPU A (Server 1)
→ GPU VRAM → PCIe → NIC (Server 1) [Direct GPU-to-NIC DMA]
→ Network
→ NIC (Server 2) → PCIe → GPU VRAM (Server 2) [Direct NIC-to-GPU DMA]
GPU B (Server 2)
The CPU is reduced to an orchestrator — it sets up the RDMA operation and is notified when it completes, but it does not touch the data. This eliminates two memory copies and frees CPU cycles for other work.
Requirements for GPUDirect RDMA:
- NVIDIA GPU with GPUDirect support (Kepler architecture or later).
- Compatible RDMA-capable NIC (Mellanox/NVIDIA ConnectX series, or RoCE-capable NICs).
- The nvidia-peermem kernel module loaded on the host (provides the peer memory interface between the GPU driver and the RDMA subsystem).
- InfiniBand or RoCE network fabric (standard Ethernet TCP/IP does not support RDMA).
Confirming GPUDirect RDMA is active means checking that the nvidia-peermem kernel module — the peer-memory interface between the GPU driver and the RDMA subsystem — is loaded, and cross-checking NCCL's debug output to confirm it selected the GPUDirect RDMA transport rather than falling back to a slower path.