The Superhighway
Training large models requires distributing work across multiple GPUs. After each forward/backward pass, GPUs must synchronize their gradient updates — a collective communication operation called All-Reduce. In All-Reduce, every GPU sends its gradients to every other GPU and receives the sum. The bandwidth of the interconnect between GPUs directly determines how fast this synchronization completes, and therefore how fast training can proceed.
PCIe Peer-to-Peer (P2P):
GPUs on the same PCIe switch can transfer data directly without going through CPU RAM, but they still share the PCIe bus bandwidth. PCIe Gen4 x16 provides ~32 GB/s per direction. With 8 GPUs sharing a PCIe switch, effective per-GPU bandwidth for All-Reduce drops significantly.
NVLink:
NVLink is NVIDIA's proprietary high-bandwidth interconnect, implemented as physical bridges between GPU cards (or, in the case of NVSwitch-based systems like DGX A100, as a dedicated switching fabric).
| Interconnect | Bandwidth (bidirectional) | Latency |
|---|---|---|
| PCIe Gen4 x16 | ~64 GB/s | ~1–2 µs |
| NVLink 3.0 (A100) | ~600 GB/s | ~1 µs |
| NVLink 4.0 (H100) | ~900 GB/s | <1 µs |
The bandwidth difference is not marginal — NVLink provides roughly 10–14× more bandwidth than PCIe for GPU-to-GPU communication. For a model like GPT-3 (175B parameters), the All-Reduce gradient synchronization involves moving ~700 GB of data per training step. At PCIe speeds, this takes seconds. At NVLink speeds, it takes milliseconds.
Verifying NVLink is in Use:
NCCL (NVIDIA Collective Communications Library), used by PyTorch's DistributedDataParallel, automatically detects and prefers NVLink. You can confirm which transport it actually chose by checking the GPU connectivity topology — where an NV# code means a live NVLink connection — and by enabling NCCL's debug logging, which names the transport it selected for each communication.
If NCCL falls back to PCIe P2P or, worse, to network (for intra-node communication), training throughput will be severely degraded. Common causes: NVLink bridges not physically installed, driver not detecting them, or NCCL version mismatch.
A common interview question: "Your 8-GPU training job is running at 30% of expected throughput. nvidia-smi shows all GPUs at 100% utilization. What do you check?" The answer involves checking the communication topology. High GPU utilization with low throughput often means GPUs are spending most of their time waiting for All-Reduce to complete — a communication bottleneck, not a compute bottleneck. Check nvidia-smi topo -m for NVLink connectivity, run with NCCL_DEBUG=INFO to see which transport NCCL chose, and use nsys profile to see the ratio of compute time to communication time.