Why Standard RAM Isn't Enough
Your laptop uses DDR (Double Data Rate) RAM. DDR is engineered for capacity and modularity — you can buy sticks of it cheaply and plug them into a motherboard. It connects to the processor via a relatively narrow 64-bit bus (think of a 4-lane highway).
- DDR5 bandwidth: ~50–90 GB/s
- NVIDIA H100 GPU compute throughput: ~3,350 GB/s
If you connected an H100 to DDR RAM, the GPU would spend roughly 97% of its time waiting for data. You would have bought a Formula 1 car and given it a garden hose for fuel delivery.
HBM Architecture
HBM (High Bandwidth Memory) solves this with a fundamentally different physical design:
-
Stacked dies: Instead of flat chips on a circuit board, HBM stacks multiple memory dies vertically — like floors in a skyscraper — directly on top of each other. This stack sits on the same silicon package as the GPU die, connected via thousands of microscopic vertical wires called Through-Silicon Vias (TSVs).
-
Massive bus width: DDR uses a 64-bit bus. HBM uses a 1024-bit bus per stack (and modern GPUs use multiple stacks). This is like replacing a 4-lane highway with a 64-lane superhighway running right next to the factory.
-
Result: The H100 achieves 3.35 TB/s of memory bandwidth — roughly 40–60x more than DDR5.
The trade-off is cost and capacity. HBM is extraordinarily expensive to manufacture and physically constrained by the package size. This is why flagship GPUs have "only" 80–96 GB of VRAM. It is not a design oversight — it is a physical limit of how much HBM can fit on a single package today.
Key Point
HBM capacity is a hard architectural ceiling for model serving. A 70B parameter model in float16 precision requires ~140 GB of VRAM just to hold the weights — before accounting for the KV cache needed during inference. This is why multi-GPU serving is not optional for large models; it is a physical necessity imposed by HBM capacity limits.