Generic Resources (GRES)
Slurm was originally designed for CPUs. GPUs were added later as Generic Resources (GRES) — a plugin system for any resource that isn't a CPU or memory. This is why GPU requests look slightly different from CPU requests: a --gres directive names the resource type (gpu), optionally a specific model (a100, or a VRAM-specific variant like a100_80gb), and a count. The available GRES types are defined by the cluster administrator, and a per-node breakdown is available via sinfo.
The Critical Role of srun
Inside your job script, srun is the command that actually launches your workload across the allocated nodes. Without srun, your script runs only on the first node — the other nodes sit idle.
# This runs ONLY on node 1 — wrong for distributed training
python train.py
# This runs on ALL 4 nodes simultaneously — correct
srun python train.py
srun does several things automatically:
1. Synchronizes launch: All processes start at the same moment across all nodes. This is essential for NCCL collective operations (AllReduce, AllGather) which require all ranks to be present before communication begins.
2. Sets environment variables: Each process receives SLURM_PROCID (global rank), SLURM_LOCALID (local rank within the node), SLURM_NODEID, and SLURM_NTASKS — exactly what PyTorch Distributed and DeepSpeed need to initialize their process groups.
3. Handles I/O: Aggregates stdout/stderr from all nodes back to your log files.
Environment Variables for Distributed Training
A typical PyTorch training script reads its global rank from SLURM_PROCID, its GPU index on the local node from SLURM_LOCALID, and the total process count from SLURM_NTASKS — then passes those straight into torch.distributed's process-group initialization. Because srun populates these variables automatically, the training script needs no cluster-specific bootstrapping code beyond reading three environment variables.
CPU Affinity and NUMA Topology
On modern GPU servers, CPUs and GPUs are organized into NUMA (Non-Uniform Memory Access) domains. A GPU on PCIe slot 0 has faster access to CPU socket 0's memory than to CPU socket 1's memory. Slurm can enforce CPU affinity — a single directive binding tasks to cores along NUMA boundaries — to ensure your data-loading CPU cores are on the same NUMA node as your GPUs, reducing memory latency.
This is the kind of hardware-aware scheduling that Kubernetes cannot easily replicate, and it's why Slurm remains preferred for latency-sensitive distributed training.