Two Different Problems
Think about the difference between a restaurant kitchen and a factory assembly line. A restaurant kitchen keeps the lights on all day — it's always ready to serve the next customer. An assembly line, by contrast, runs a specific job from start to finish as efficiently as possible, then moves to the next one. These are fundamentally different operational models, and they require different tools.
Computing has the same split. Web services (APIs, databases, dashboards) need to stay running indefinitely — they're the restaurant kitchen. Batch workloads (training a neural network, running a genome simulation, rendering a film) have a clear start and end — they're the assembly line. Trying to use the same scheduler for both leads to compromises that hurt both.
HPC (High-Performance Computing) clusters are built for the assembly-line model. They've existed since the 1960s, long before the cloud, and they solve a specific problem: how do you fairly and efficiently share thousands of expensive CPUs and GPUs across hundreds of researchers, each submitting jobs that might run for hours or days?
Slurm (Simple Linux Utility for Resource Management) is the dominant answer to that question. It was developed at Lawrence Livermore National Laboratory in 2002 and now runs on the majority of the world's top supercomputers — including many of the clusters used to train large language models like GPT-4. While Kubernetes dominates cloud-native workloads, Slurm remains the tool of choice for Foundation Model Training because it offers tighter control over hardware topology, near-zero scheduling overhead, and bare-metal GPU performance that containerized environments struggle to match.
| Dimension | Web/Cloud (Kubernetes) | HPC (Slurm) |
|---|---|---|
| Primary goal | Keep services up forever | Get jobs done as fast as possible |
| Scheduling unit | Pod (container) | Job (shell script) |
| Failure response | Restart the pod elsewhere | Fail the job (usually) |
| Optimization target | Availability | Utilization and throughput |
| Typical workload | APIs, microservices, databases | Model training, simulations, rendering |
Interviewers at AI infrastructure companies often ask: "Why would you use Slurm instead of Kubernetes for LLM training?" The answer is three-fold: (1) Slurm has near-zero overhead — no kubelet, no sidecar containers, no CNI plugin eating into your network bandwidth; (2) Slurm has deep hardware topology awareness, letting you pin jobs to specific racks or switches to minimize inter-node latency; (3) Slurm's srun provides synchronized multi-node process launch that NCCL collective operations depend on. Kubernetes is catching up with projects like Kubeflow and the Volcano scheduler, but Slurm still wins on raw performance for tightly-coupled distributed training.