The Infinite Bucket
Object storage is the filing cabinet of the cloud. Instead of a hierarchical directory tree (folders inside folders), object storage uses a flat namespace: every piece of data is an object identified by a unique key (a string like datasets/imagenet/train/n01440764/img_001.jpg), stored in a bucket. There are no real folders — the slashes in the key are just part of the name.
The access protocol is HTTP/REST: you PUT an object to upload it, GET an object to download it, DELETE to remove it. This simplicity is what makes object storage infinitely scalable — any server that speaks HTTP can serve objects, and you can add servers horizontally without limit.
Amazon S3, Google Cloud Storage (GCS), Azure Blob Storage, and the open-source MinIO are the dominant implementations. At petabyte scale, object storage costs roughly $20–$23 per TB/month in the cloud, compared to $200–$500/TB/month for high-performance parallel filesystem storage.
The Latency Problem
The HTTP protocol introduces overhead that POSIX filesystems don't have: TCP handshake, TLS negotiation, HTTP headers, and server-side request routing. A single GET request to S3 from an EC2 instance in the same region takes roughly 5–50 ms of latency before the first byte arrives. For a 500 MB video file, that 50 ms is negligible — you spend most of your time transferring data. For a 10 KB image file, that 50 ms dominates: you spend 99% of your time on overhead and 1% on actual data transfer.
The mitigation is parallelism: instead of one sequential stream of requests, you issue hundreds of concurrent requests. A thread pool of dozens of workers, each independently fetching a different object key, saturates S3's bandwidth even though each individual request pays the same fixed latency overhead.
The Modern Hybrid: Hot/Cold Tiering
No production AI shop uses a single storage tier. The standard architecture is:
- Cold tier (S3/GCS): Store the full dataset — 10 PB of raw data — at low cost. This is the source of truth.
- Warm tier (Parallel filesystem or high-speed NAS): When a training run starts, a data staging job copies the relevant dataset shard to a Lustre or GPFS volume. Training reads from here.
- Hot tier (Local NVMe on the training node): For datasets small enough to fit, cache the entire dataset on the node's local NVMe drives. Subsequent epochs read at full NVMe speed (7 GB/s) with zero network overhead.
The staging step is often automated by the job scheduler (Slurm, Kubernetes) as a pre-job hook. The training job doesn't start until the data is staged.