The I/O Bottleneck
Think of a GPU like a Formula 1 pit crew — extraordinarily fast at its job, but completely idle if the car isn't in the pit. In AI training, the "car" is your data. If data doesn't arrive fast enough, the GPU sits idle, burning money. This is the storage I/O bottleneck: the gap between how fast a GPU can consume data and how fast your storage system can deliver it.
An NVIDIA H100 GPU can perform roughly 2,000 TFLOPS of computation. A single enterprise NVMe SSD delivers around 7 GB/s of sequential read throughput. A single spinning hard disk delivers around 150 MB/s. The GPU can process a batch of images in microseconds; a slow disk might take tens of milliseconds to deliver the next batch. That mismatch is where training jobs go to die.
Storage for AI has three dimensions that all matter simultaneously:
- Capacity: How much data can you store? (Petabytes)
- Throughput: How fast can you move large chunks of data? (GB/s)
- IOPS: How many individual file operations can you perform per second? (Files/s)
Most storage systems excel at one or two of these but not all three. Choosing the wrong storage architecture for your workload is one of the most common and expensive mistakes in AI infrastructure. A $30/hour GPU cluster sitting at 20% utilization because of a $0.05/GB storage bottleneck is a failure of platform engineering, not a failure of hardware.
Interviewers at AI infrastructure companies frequently ask: "How do you diagnose a slow training job?" The correct first step is to check GPU utilization (via nvidia-smi or your monitoring stack). If utilization is below 80%, the bottleneck is almost certainly I/O or data preprocessing — not compute. Saying "I'd add more GPUs" is the wrong answer and signals a lack of systems thinking.