The Physical Path Matters
On a single-socket workstation, all PCIe slots connect to the same CPU, and data movement is straightforward. On a dual-socket server — the kind used in production ML training clusters — the topology becomes critical.
Each CPU socket in a dual-socket server has its own PCIe controller. PCIe slots are physically wired to one socket or the other. The two sockets are connected to each other via a high-speed interconnect called UPI (Ultra Path Interconnect, Intel) or xGMI (AMD). This interconnect is fast, but it adds latency and is a shared resource.
The NUMA Problem in Practice:
Imagine a 2-socket server where GPU 0 and GPU 1 are wired to Socket 0, and GPU 2 and GPU 3 are wired to Socket 1. Your Python training process is scheduled by the OS onto cores on Socket 1. When it tries to copy a training batch to GPU 0:
System RAM (Socket 1 DIMM) → CPU Socket 1 → UPI Link → CPU Socket 0 → PCIe Switch → GPU 0
Every byte of data crosses the UPI link. Compare this to the optimal path:
System RAM (Socket 0 DIMM) → CPU Socket 0 → PCIe Switch → GPU 0
The UPI crossing adds roughly 50–100 ns of latency per hop and reduces effective bandwidth by 30–50% compared to local PCIe access. In a training loop that moves hundreds of gigabytes per minute, this compounds into significant throughput loss.
Diagnosing and Fixing NUMA Affinity:
nvidia-smi topo -m shows which CPU cores are local to each GPU, and numactl --hardware shows the system's NUMA topology. Armed with that mapping, the fix is to pin the training process to the CPU cores and memory node local to the GPU it's feeding, so the CPU staging a batch and the GPU consuming it sit on the same socket rather than crossing the UPI link.
PyTorch's DataLoader also accepts a pin_memory=True argument, which allocates CPU-side data buffers in pinned (page-locked) memory. Pinned memory cannot be swapped out by the OS, which allows the GPU's DMA engine to transfer it directly without the CPU copying it first — another PCIe efficiency win.