The Bottleneck Shift
Think of a factory assembly line. You can hire the fastest workers in the world, but if the conveyor belt delivering raw materials runs at walking speed, your workers spend most of their time standing idle. The bottleneck isn't the workers — it's the supply chain.
Modern AI clusters have exactly this problem. For decades, the industry focused on making processors faster — more GHz, more cores, more floating-point operations per second (FLOPS). That battle is largely won. A single NVIDIA H100 GPU can perform nearly 2,000 trillion floating-point operations per second (2 petaFLOPS). The compute cores are extraordinarily fast.
The new bottleneck is moving data. Training a large model like GPT-4 requires feeding terabytes of training examples into the GPUs and shuffling gigabytes of gradient updates between GPUs after every training step. If the compute cores are waiting for data to arrive — from memory, from another GPU, or from another server — your multimillion-dollar cluster is essentially idle. The expensive workers are standing at the conveyor belt, waiting.
AI Hardware Topology is the discipline of designing how these systems are wired together — internally (how memory connects to compute) and externally (how GPUs connect to each other across servers) — to keep the compute cores continuously fed. Understanding this topology is what separates engineers who can diagnose a 40% GPU utilization rate from those who just throw more hardware at the problem.