Scaling Beyond One Server
NVLink and NVSwitch handle communication inside a single server. But training GPT-4-scale models requires thousands of GPUs spread across hundreds of servers. You need a network fabric that connects these servers together.
The choice of inter-node network is one of the most consequential infrastructure decisions in AI cluster design.
The Contenders
Ethernet is the universal networking standard — the same technology that connects your laptop to the internet.
- Pros: Inexpensive, ubiquitous, large ecosystem of management tools, most engineers already know it.
- Cons: Designed for bursty, latency-tolerant traffic (web requests). Standard Ethernet drops packets under congestion and relies on TCP retransmission — catastrophic for AI workloads where every GPU in a 1,000-node cluster waits for the slowest packet.
- Evolution: RoCE (RDMA over Converged Ethernet) adds RDMA capabilities to Ethernet, dramatically reducing latency. NVIDIA's Spectrum-X platform combines RoCE with adaptive routing to approach InfiniBand performance on Ethernet hardware.
InfiniBand (IB) was designed from the ground up for high-performance computing (HPC) and supercomputers.
- Pros: Extremely low latency (sub-microsecond), lossless by design (hardware-level flow control prevents packet drops), very high bandwidth (400 Gb/s per port with HDR, 800 Gb/s with NDR).
- Cons: Expensive (switches and NICs cost significantly more than Ethernet equivalents), requires specialized operational knowledge, smaller vendor ecosystem.
The Practical Decision
| Criterion | InfiniBand | High-Speed Ethernet (RoCE) |
|---|---|---|
| Latency | ~1 µs | ~2–5 µs |
| Packet loss under congestion | None (lossless) | Possible without careful tuning |
| Cost | High | Moderate |
| Operational complexity | High | Moderate |
| Best for | Tightly-coupled training (All-Reduce) | Inference fleets, mixed workloads |
For the largest AI training clusters (Meta's RSC, NVIDIA's Selene), InfiniBand remains the dominant choice because the lossless, low-latency properties directly translate to faster All-Reduce operations and higher GPU utilization. For inference serving fleets and enterprise clusters where cost matters, high-speed Ethernet with RoCE is increasingly viable.
A common interview question is: "Why does packet loss matter so much for AI training but not for web traffic?" The answer is the synchronization barrier. In distributed training, all GPUs must complete the All-Reduce step before any GPU can proceed to the next training step. A single dropped packet that triggers TCP retransmission stalls the entire cluster — 1,000 GPUs wait for one retransmit. Web traffic tolerates retransmission because each request is independent. AI training is a synchronized collective operation where tail latency equals cluster latency.