Parallel Filesystems: Lustre & GPFS

Speed Through Striping

A standard filesystem is like a single-lane road: all traffic flows through one server, one disk controller, one set of spindles. No matter how wide your GPU cluster is, data arrives through that single lane. Parallel filesystems solve this by turning one road into a hundred-lane highway — every lane carries a piece of your data simultaneously.

Lustre and IBM GPFS (now called Spectrum Scale) are the two dominant parallel filesystems in HPC and AI supercomputing environments. They are the storage backbone of most national labs, university clusters, and large-scale AI training facilities.

How Striping Works

When you save a 100 GB file on Lustre, the filesystem doesn't write it to one disk. It breaks the file into fixed-size chunks (called stripe units, typically 1–4 MB each) and distributes those chunks across many Object Storage Targets (OSTs) — each OST is a separate physical disk or RAID group.

File: dataset.tar (100 GB)
Stripe size: 1 MB
Stripe count: 100 OSTs

Chunk 1  → OST-001
Chunk 2  → OST-002
Chunk 3  → OST-003
...
Chunk 100 → OST-100
Chunk 101 → OST-001  (wraps around)

When a training job reads dataset.tar, Lustre issues parallel read requests to all 100 OSTs simultaneously. If each OST delivers 500 MB/s, the aggregate throughput is 50 GB/s for a single file read. This is why Lustre is used in GPU supercomputers.

The Metadata Server (MDS) is a separate component that tracks where every chunk lives. It handles open(), stat(), readdir() — the bookkeeping operations. This is also Lustre's Achilles heel: the MDS is a single point of coordination. Under heavy small-file workloads, the MDS becomes the bottleneck, and under failure, the entire filesystem stalls.

Stripe count and size are configured per-directory before writing with lfs setstripe, and lfs getstripe reports the current settings (stripe count, stripe size, and offset) for any file or directory.

Trade-offs at a glance:

Senior engineers know that Lustre's stripe count should be tuned per workload. A large sequential read (video frames) benefits from high stripe counts. A workload with many small random reads (metadata-heavy) can actually perform worse with high stripe counts because the coordination overhead exceeds the parallelism benefit.