Why DDP Is Wasteful — and How ZeRO Fixes It
Standard Data Parallelism replicates the entire model state on every GPU. "Model state" means more than just the weights — it includes:
- Model Parameters (W): The weights themselves. For a 7B model in fp16: ~14 GB.
- Gradients (∇W): Same size as parameters: ~14 GB.
- Optimizer States: Adam keeps two additional tensors per parameter (momentum and variance), stored in fp32: ~56 GB.
Total for a 7B model: ~84 GB per GPU. With 100 GPUs, you store 100 identical copies — 8.4 TB of redundant data.
ZeRO (Zero Redundancy Optimizer), developed by Microsoft DeepSpeed and also implemented as PyTorch's FSDP (Fully Sharded Data Parallel), eliminates this redundancy by sharding the model state across GPUs instead of replicating it.
ZeRO has three progressive stages:
| Stage | What is Sharded | Memory Reduction |
|---|---|---|
| ZeRO-1 | Optimizer States | ~4× |
| ZeRO-2 | Optimizer States + Gradients | ~8× |
| ZeRO-3 / FSDP | Optimizer States + Gradients + Parameters | ~N× (linear with GPU count) |
How ZeRO-3 works at runtime:
Each GPU permanently owns 1/N of the parameters. When GPU 1 needs to run a forward pass through Layer 5 (owned by GPU 3), it broadcasts a request. GPU 3 sends the Layer 5 weights to all GPUs that need them. After the computation, those weights are immediately discarded from GPU 1's memory. This "gather-compute-discard" pattern means peak memory per GPU is only slightly above 1/N of the total model state.
The trade-off: ZeRO-3 introduces more communication than DDP (parameters must be gathered before each layer's computation). On fast interconnects (NVLink, InfiniBand HDR), this overhead is acceptable. On slow networks, ZeRO-2 is often preferred as a compromise.
A common interview question: "What is the difference between ZeRO-3 and Tensor Parallelism? Both shard model parameters." The key distinction: TP shards the computation within a layer (the matrix multiply itself is split), requiring synchronization after every layer with high bandwidth. ZeRO-3 shards the storage of parameters but gathers the full parameter tensor before computing — each GPU runs the full layer computation on its own data slice. TP is compute-parallel; ZeRO-3 is memory-parallel.