Model Parallelism: Tensor Parallelism (TP)

Fit Bigger Models by Slicing the Math

Tensor Parallelism solves Scenario B — the model is too large for one GPU. Instead of replicating the model, we split the model's internal computations across GPUs. Specifically, we slice the individual matrix multiplications that make up each layer.

Consider a single linear layer: Y = X × W, where W is a weight matrix of shape [8192, 8192]. In Tensor Parallelism:

• GPU 1 holds the left half of W (columns 0–4095) and computes Y₁ = X × W₁.
• GPU 2 holds the right half of W (columns 4096–8191) and computes Y₂ = X × W₂.
• The results are combined via an All-Reduce (for row-parallel) or All-Gather (for column-parallel) to produce the full output Y.

This is called intra-layer parallelism because the split happens inside a single layer. The Megatron-LM paper (from NVIDIA) formalized this approach for Transformer attention and MLP blocks, and it remains the standard implementation.

The bandwidth requirement is severe. After every single layer, GPUs must exchange partial results. For a 175B parameter model with 96 layers, that is 96 synchronization points per forward pass. If the interconnect is slow, GPUs spend more time waiting for data than computing.

The NVLink rule: Tensor Parallelism is almost always constrained to GPUs within a single server node, connected via NVLink (bandwidth: ~900 GB/s on H100 NVSwitch) or NVSwitch. Running TP across InfiniBand (~25 GB/s effective) or Ethernet is technically possible but typically 10–20× slower and rarely done in production.