The HPC Problem
Some workloads don't need gigabytes per second — they need hundreds of gigabytes per second. Weather modeling, genomic sequencing, financial Monte Carlo simulations, and deep learning training jobs need to feed data to thousands of CPU/GPU cores simultaneously. Standard NFS (EFS) tops out at tens of GB/s and has per-operation latency in the milliseconds. That's too slow.
What Lustre Is
Lustre is an open-source parallel file system originally developed for supercomputers. The name is a portmanteau of "Linux" and "cluster." It is the file system running on the majority of the world's top 500 supercomputers. Instead of one server handling all file requests, Lustre splits data across many storage servers simultaneously — every client reads from multiple servers in parallel, achieving aggregate throughput that scales linearly with the number of storage nodes.
AWS manages the Lustre infrastructure for you through FSx for Lustre. You specify the storage capacity and throughput class; AWS provisions the cluster.
Key Characteristics
- Throughput: Up to hundreds of GB/s aggregate throughput and millions of IOPS.
- Latency: Sub-millisecond for metadata and data operations.
- Protocol: Lustre client (Linux kernel module). Not NFS or SMB — requires the Lustre client installed on compute nodes.
- S3 Integration (Data Repository Association): This is the killer feature. You link FSx for Lustre to an S3 bucket. The S3 objects appear as files in the Lustre namespace immediately — no copying required. When a compute node reads a file for the first time, Lustre fetches it from S3 and caches it locally ("lazy hydration"). Subsequent reads are served from Lustre at full speed. When processing is complete, you can export results back to S3 automatically. This link is created as a single Data Repository Association pointing the Lustre file system at an S3 prefix, with auto-import and auto-export policies (covering new, changed, and deleted objects) so that changes in either direction propagate automatically.
Deployment Types
| Type | Durability | Use Case |
|---|---|---|
| Scratch 1 | No replication | Temporary processing, maximum cost efficiency |
| Scratch 2 | No replication | Temporary processing, higher throughput than Scratch 1 |
| Persistent 1 | Data replicated within AZ | Long-running workloads, HA within AZ |
| Persistent 2 | Data replicated within AZ | Higher throughput, SSD-backed |
Scratch file systems are cheaper and faster but data is not replicated. If an underlying storage server fails, you lose data on that server. This is acceptable for jobs where the source data lives in S3 and you can re-run the job. Persistent file systems replicate data within the AZ, protecting against hardware failure.
Common Use Cases
- ML/AI training: Read a 50 TB training dataset from S3 at 500 GB/s into GPU instances.
- Media rendering (VFX): Render farms reading and writing massive frame files simultaneously.
- Genomics: Align and process whole-genome sequencing data across hundreds of compute nodes.
- Financial simulations: Monte Carlo simulations reading market data in parallel.
A common interview question: "Your ML training job is bottlenecked on data loading — GPUs are idle 40% of the time waiting for data. What do you do?" The answer involves FSx for Lustre with an S3 Data Repository Association. The training data lives in S3 (cheap, durable), but the training cluster mounts FSx for Lustre. On first epoch, data is lazily loaded from S3 into Lustre. On subsequent epochs, reads are served from Lustre at sub-millisecond latency. GPUs stay fed. This pattern is the standard AWS architecture for large-scale ML training.
Key Point — Cost Model: FSx for Lustre is priced per GB-month of provisioned storage plus throughput capacity. Unlike EFS, you provision a fixed size upfront (minimum 1.2 TB for Scratch). This means you pay for capacity whether you use it or not. For short-lived HPC jobs, the pattern is: provision → run job → delete file system → results are in S3. Don't leave Lustre file systems running idle.