The Problem
Training a neural network on a single GPU is like trying to fill a swimming pool with a garden hose — it works fine for small jobs, but scale up and you immediately hit two hard limits.
- Scenario A — Time: Your model fits on the GPU, but training on your huge dataset will take 3 years. You need to finish in 3 weeks.
- Scenario B — Memory: Your model is so large (e.g., Llama-70B at ~140 GB in fp16) it doesn't even fit in the VRAM of a single GPU to begin with. Training cannot start.
To solve these, we connect multiple GPUs together and divide the work. But how we split that work defines the Parallelism Strategy — the most fundamental architectural decision in AI infrastructure. Choose wrong and you either run out of memory, saturate your network, or leave GPUs sitting idle.
The rest of this lesson builds up each strategy from first principles, explains when to use it, and shows how they combine into the "3D Parallelism" approach used to train models like GPT-3 and Llama.