Solving Small Files: Aggregation (TAR/TFRecord/Parquet)

Pack Them Up

The solution to the small file problem is conceptually simple: instead of storing millions of tiny files, pack them into a small number of large files. The training dataloader then reads large files sequentially, paying the file-open overhead once per shard instead of once per sample.

This technique is called sharding: dividing a dataset into a fixed number of large archive files, each containing thousands of individual samples.

Aggregation Formats

WebDataset (TAR shards) is the most widely adopted format for large-scale vision and multimodal training. It uses standard POSIX tar archives, which means any tool that reads tar files can inspect the data. Each shard is a tar file containing alternating sample files:

shard-000000.tar
├── 000000.jpg      # image
├── 000000.cls      # label
├── 000001.jpg
├── 000001.cls
...
├── 009999.jpg
└── 009999.cls

Creating shards is a matter of iterating over the source images and handing each one, along with its label, to a ShardWriter — the library handles rolling over to a new tar file once the configured sample count per shard is reached.

TFRecord is TensorFlow's native binary container format. It stores data as a sequence of serialized Protocol Buffer messages. It is efficient but TensorFlow-specific — reading TFRecords outside TensorFlow requires extra tooling.

Apache Parquet is a columnar binary format designed for tabular data. It is the standard for NLP datasets (text, tokens, embeddings) and structured metadata. Parquet supports predicate pushdown — you can read only the rows and columns you need without scanning the full file, simply by passing the subset of column names you need to the reader's columns argument.

The Throughput Math

Before sharding (14M individual files):

14,000,000 opens × 1ms overhead = 14,000 seconds of overhead
Effective read rate: ~50 MB/s (IOPS-bound)

After sharding (1,400 TAR files of 10,000 images each):

1,400 opens × 1ms overhead = 1.4 seconds of overhead
Effective read rate: ~5,000 MB/s (bandwidth-bound, NVMe)

The overhead drops from hours to seconds. The storage system now operates in its efficient sequential mode.

Rule of thumb: If your average file size is below 10 MB, aggregate into shards. Target shard sizes of 100 MB to 1 GB — large enough to amortize open overhead, small enough to distribute across workers efficiently.

Disabling atime updates is a complementary optimization that reduces write amplification on reads — remounting the filesystem (or setting it permanently in /etc/fstab) with the noatime option eliminates the access-time write that would otherwise happen on every single read.