Train Faster by Cloning the Model
Data Parallelism solves Scenario A — training is too slow. The core idea is simple: if one GPU takes 3 years, give 100 GPUs the same model and let each one work on a different slice of the data simultaneously.
Here is the exact sequence of events in one training step:
- Replicate: Copy the entire model weights to every GPU (called a worker or rank).
- Scatter: Split your global batch (e.g., 3,200 images) into equal mini-batches (e.g., 400 images per GPU across 8 GPUs).
- Forward + Backward: Each GPU independently runs a forward pass (prediction) and a backward pass (gradient computation) on its own mini-batch. No communication happens here — this is the fast part.
- All-Reduce: Before updating weights, every GPU broadcasts its gradients to all other GPUs and receives theirs. The gradients are averaged. Every GPU ends up with the same averaged gradient and applies the same weight update. The model stays in sync.
The All-Reduce operation is the critical communication step. The most common algorithm for it is Ring All-Reduce, where GPUs are arranged in a logical ring and pass gradient chunks around the ring in two phases (scatter-reduce, then all-gather). This scales efficiently because communication cost is constant regardless of the number of GPUs — each GPU only ever talks to its two neighbors.
The hard constraint: The entire model must fit inside the VRAM of a single GPU. A 70B parameter model in fp16 requires ~140 GB. A single H100 has 80 GB. DDP cannot start.
Interviewers often ask: "What is the communication complexity of All-Reduce in DDP?" The answer is O(2(N-1)/N × M) where N is the number of GPUs and M is the message size — meaning communication cost is roughly constant per GPU regardless of cluster size. This is why DDP scales well. A follow-up question is: "What breaks DDP at scale?" Answer: gradient synchronization becomes a bottleneck when the model is large (many parameters = large gradient tensors) and the network is slow.