The Assembly Line
Pipeline Parallelism is the second way to split a model that doesn't fit on one GPU. Instead of slicing individual layers (like TP), we cut the model between layers and assign consecutive groups of layers to different GPUs. This is called inter-layer parallelism.
For a 96-layer model across 4 GPUs:
- GPU 1: Layers 1–24
- GPU 2: Layers 25–48
- GPU 3: Layers 49–72
- GPU 4: Layers 73–96
GPU 1 processes a batch, produces activations (intermediate outputs), and sends them to GPU 2. GPU 2 processes those activations and passes them to GPU 3, and so on. On the backward pass, gradients flow in reverse: GPU 4 → GPU 3 → GPU 2 → GPU 1.
The Pipeline Bubble Problem:
In a naive implementation, GPU 2 cannot start until GPU 1 finishes. GPU 3 cannot start until GPU 2 finishes. This creates large idle periods — called pipeline bubbles — where most GPUs are waiting. In the worst case, bubble overhead is (PP - 1) / PP of total time. With 4 GPUs, up to 75% of time could be wasted.
The fix: Micro-batching. Instead of sending one large batch through the pipeline, we split it into many small micro-batches. While GPU 2 is processing micro-batch 1, GPU 1 immediately starts on micro-batch 2. This keeps the assembly line full. The 1F1B schedule (one forward, one backward) is the standard scheduling algorithm that interleaves forward and backward passes to minimize bubble size to approximately (PP - 1) / (PP × M) where M is the number of micro-batches.
Communication cost is low compared to TP. GPUs only exchange activations at the layer boundaries — once per micro-batch, not once per layer. This makes PP suitable for slower inter-node connections like InfiniBand.