Not All Memory Paths Are Equal
In modern high-end servers used for AI, there isn't just one CPU. There are often two or four large CPUs on a single motherboard, each with dozens of cores. This design is called a multi-socket system.
This introduces a specific memory architecture called Non-Uniform Memory Access (NUMA). The name describes the core insight: not all memory accesses take the same amount of time.
- Each CPU socket has its own bank of RAM physically attached to it via high-speed memory channels. Accessing this local RAM is extremely fast — typically 50–100 nanoseconds.
- The CPU sockets are connected to each other via an interconnect (Intel's UPI, AMD's Infinity Fabric). A CPU can access RAM attached to another socket, but it must travel across this link. This remote access is 1.5–3× slower and adds latency.
The GPU Connection
Crucially, GPUs are also connected via PCIe slots that are physically wired to specific CPU sockets. The PCIe root complex — the hardware that manages PCIe traffic — is integrated into each CPU socket.
- GPU 0 might be wired to CPU Socket 0.
- GPU 1 might be wired to CPU Socket 1.
This is not a software configuration — it's determined by the physical traces on the motherboard.
The Affinity Problem in Practice
Before data can be processed by the GPU, it must be pre-processed by the CPU and loaded into system RAM. This is the job of your data loading workers (e.g., PyTorch DataLoader worker processes).
If your data loading workers run on CPU Socket 0 cores, they load data into CPU Socket 0's local RAM. If you then try to feed that data to GPU 1 (which is wired to CPU Socket 1), the data must take the slow path:
CPU 0 RAM → CPU 0 → NUMA Interconnect → CPU 1 → PCIe → GPU 1
This added latency can starve the GPU of data, causing it to sit idle waiting for the next batch. In practice, this can reduce GPU utilization by 10–30% on data-intensive workloads — a significant and completely invisible bottleneck if you don't know to look for it.
GPU-CPU Affinity is the practice of ensuring that the CPU processes preparing data for a specific GPU are running on the same physical CPU socket that the GPU is wired to. This ensures the fast, local path:
CPU 0 RAM → CPU 0 → PCIe → GPU 0 ✓ (fast, local)
Diagnosing Your NUMA Topology
Before you can fix affinity, you need to understand your server's topology. The lstopo tool (from the hwloc package) gives you a visual map of sockets, cores, and memory banks, and nvidia-smi's topology matrix reports which NUMA node — and which CPU core range — each GPU is wired to (e.g., GPU 0 on node 0 with cores 0–23, GPU 1 on node 1 with cores 24–47).
Enforcing Affinity
Once you know the topology, numactl pins a process to the correct NUMA node: its --cpunodebind flag restricts which CPU cores the process may run on, and --membind restricts which node's RAM its allocations come from. Running a GPU 0 training script with both flags set to node 0 (and GPU 1's with both set to node 1) guarantees the fast local path for all data movement to each GPU.
A common interview scenario: "Your GPU utilization is only 60% even though your model is compute-bound and you've verified there are no software bottlenecks. What would you investigate?" NUMA affinity is a strong answer here. Explain that you'd run nvidia-smi topo -m to check the topology, then use numactl to pin data loading workers to the same NUMA node as the target GPU. This demonstrates knowledge of hardware-level bottlenecks that most candidates miss entirely.
Key Point: NUMA affinity matters most for workloads with high data throughput — large batch sizes, high-resolution images, or video data. For small models with tiny inputs, the NUMA penalty may be negligible. Always profile first with nvidia-smi dmon (device monitoring) to confirm GPU utilization is actually suffering before spending time on NUMA tuning. Premature optimization of NUMA topology on a workload that's already compute-bound wastes engineering time.