The Math of Failure
Imagine you are training a computer vision model on ImageNet — one of the most common benchmark datasets in the field.
- Dataset: 14 million images
- Average image size: 100 KB
- Total dataset size: ~1.4 TB
1.4 TB is not large by modern standards. A single NVMe SSD can hold it. A single spinning disk array can hold it. But reading it will be excruciatingly slow, and the reason is not bandwidth — it is the cost of opening each file.
Why Opening Files Is Expensive
Every time a program opens a file (open('img_001.jpg')), the operating system must:
- Look up the inode: The inode is a data structure that stores the file's metadata — permissions, owner, size, and the physical disk locations of its data blocks. Finding the inode requires reading from disk.
- Check permissions: The OS verifies the process has read access.
- Update access time (atime): By default, the OS writes the current timestamp to the inode every time a file is read. This turns every read into a write.
- Return a file descriptor: The OS allocates kernel resources for the open file.
On a local SSD, this overhead is roughly 0.1–1 ms per file. On a network filesystem (NFS, Lustre), it can be 1–10 ms because the metadata lookup crosses the network.
The arithmetic:
14,000,000 files × 1 ms per open = 14,000 seconds = ~3.9 hours
Just to open the files. Before reading a single byte of image data.
IOPS: The Hidden Ceiling
IOPS (Input/Output Operations Per Second) measures how many individual I/O operations a storage system can handle per second. This is distinct from throughput (GB/s), which measures how much data moves.
| Storage Type | Sequential Throughput | Random IOPS |
|---|---|---|
| 7200 RPM HDD | 150 MB/s | 100–200 IOPS |
| SATA SSD | 550 MB/s | 80,000 IOPS |
| NVMe SSD | 7,000 MB/s | 1,000,000 IOPS |
| Lustre (large cluster) | 50,000 MB/s | 500,000 IOPS |
To read 14 million files in one hour, you need: 14,000,000 / 3,600 = ~3,900 IOPS. That sounds achievable — but each file open also triggers metadata operations on the filesystem's inode table, and those operations serialize. In practice, a Lustre cluster capable of 50 GB/s of sequential bandwidth might deliver only 50 MB/s on a small-file workload because the Metadata Server is saturated.
The diagnostic signature of the small file problem:
- High IOPS utilization on storage monitoring
- Low bandwidth utilization (well below the storage system's rated throughput)
- GPU utilization stuck below 30%
- iostat -x showing high %util with a high reads-per-second (r/s) but low kilobytes-per-second (rkB/s) — that mismatch, small kB per read, is the small-file signature