Why You Need All Three
No single parallelism strategy solves every problem. Real large-scale training jobs combine all three into what practitioners call 3D Parallelism:
- Tensor Parallelism (TP): Handles layers too large for one GPU. Constrained to fast intra-node links.
- Pipeline Parallelism (PP): Handles models with too many layers. Works across nodes on InfiniBand.
- Data Parallelism (DP) with ZeRO: Handles throughput — more GPUs processing more data per second.
The total GPU count is the product of all three dimensions:
Total GPUs = TP × PP × DP
How the dimensions map to hardware:
Cluster
├── Node 1 (8 GPUs, NVLink)
│ ├── GPU 0–3: TP Group A (Pipeline Stage 1)
│ └── GPU 4–7: TP Group B (Pipeline Stage 1)
├── Node 2 (8 GPUs, NVLink)
│ ├── GPU 0–3: TP Group A (Pipeline Stage 2)
│ └── GPU 4–7: TP Group B (Pipeline Stage 2)
...
The TP dimension always lives within a node (fast NVLink). The PP dimension spans nodes (moderate InfiniBand). The DP dimension spans independent pipeline replicas (low-bandwidth gradient sync, often compressed).
This hierarchy matches communication intensity to available bandwidth — the most communication-intensive operation (TP) gets the fastest link, and the least intensive (DP gradient sync) gets the slowest.
Practical defaults for large models (as of 2024):
- TP = number of GPUs per node (typically 8)
- PP = number of nodes in a pipeline group (typically 4–16)
- DP = remaining GPUs / (TP × PP)