The Problem: One GPU Is Not Enough
When a model is too large to fit in a single GPU's HBM (e.g., a 140 GB model on an 80 GB GPU), you must split the model across multiple GPUs — a technique called Model Parallelism or Tensor Parallelism. GPU 1 holds the first half of a layer's weight matrix; GPU 2 holds the second half. During every forward pass, they must exchange intermediate results.
This communication happens constantly — for every layer, for every token. The interconnect between GPUs becomes the critical path.
PCIe (Peripheral Component Interconnect Express) is the standard bus that connects GPUs to the CPU and to each other in a normal server:
- PCIe Gen4 x16: ~64 GB/s bidirectional
- PCIe Gen5 x16: ~128 GB/s bidirectional
Compare this to HBM bandwidth at 3,350 GB/s. Using PCIe for inter-GPU communication during tensor parallelism is like routing traffic from a 64-lane highway through a single-lane bridge. The GPUs would spend most of their time waiting for data from their neighbors.
NVLink: The Direct Wire
NVLink is NVIDIA's proprietary high-speed interconnect that connects GPUs directly to each other, bypassing the CPU and PCIe entirely.
| Generation | GPU | NVLink Bandwidth (bidirectional) |
|---|---|---|
| NVLink 3 | A100 | 600 GB/s |
| NVLink 4 | H100 | 900 GB/s |
| NVLink 5 | B100/B200 | 1,800 GB/s |
At 900 GB/s, NVLink is roughly 14x faster than PCIe Gen4. This makes tensor parallelism across NVLink-connected GPUs practical — the communication overhead is small relative to the compute time.
NVSwitch: The Chassis-Level Router
Connecting 8 GPUs with NVLink in a full mesh (every GPU directly wired to every other GPU) would require 28 cables and create a wiring nightmare. NVSwitch solves this with a dedicated switching chip inside the server chassis.
All 8 GPUs connect to one or more NVSwitch chips. The switch provides all-to-all connectivity — any GPU can communicate with any other GPU at full NVLink speed simultaneously. From the software's perspective, the 8 GPUs behave like a single unified device with a shared pool of 640 GB of HBM (8 × 80 GB), accessible at NVLink speeds.
NVIDIA's DGX H100 server uses 4 NVSwitch chips to connect 8 H100 GPUs, providing 900 GB/s of all-to-all bandwidth. This is the building block of large AI clusters.