How the Pieces Fit Together
Slurm uses a centralized master-worker architecture. There is one brain that knows the state of everything, and many workers that do the actual computation. This is conceptually similar to how a restaurant manager (who knows every table's status) coordinates with individual waitstaff (who serve the tables).
The four components you need to know:
1. slurmctld — The Controller Daemon
This is the brain of the cluster. It runs on a dedicated management node (not a compute node). It maintains a real-time map of every node's state: how many CPUs are free, how many GPUs are allocated, which nodes are down. When a job is submitted, slurmctld decides when it runs and where — it's the Tetris player fitting jobs into available hardware slots.
2. slurmd — The Node Daemon
This runs on every compute node (every GPU server). It's a lightweight agent that does three things: reports its node's health and available resources back to slurmctld, receives job assignments from the controller, and launches and monitors the actual processes. When a job's time limit expires, slurmd kills the processes.
3. slurmdbd — The Database Daemon
This is the accounting layer. It records every job: who submitted it, which nodes it used, how long it ran, how many GPU-hours it consumed. This data feeds the Fair Share scheduler (explained later) and lets administrators generate usage reports for billing or quota enforcement.
4. Login Node
Users never SSH directly into GPU compute nodes. Instead, they connect to a login node — a regular server that has the Slurm client tools installed. From the login node, users write scripts, submit jobs, and check queue status. The login node has no slurmd; it's purely a submission and monitoring interface.
The Job Lifecycle
User on Login Node
│
│ sbatch job.sh
▼
slurmctld (Controller)
│ "Node gpu-04 is idle. Assign job 12345 there."
▼
slurmd on gpu-04
│ Launches processes, monitors them
▼
Job completes → slurmd reports back → slurmctld marks node idle
│
▼
slurmdbd records accounting data
A critical architectural detail: slurmctld is a single point of failure. Most production clusters run a standby controller that takes over if the primary fails. The compute nodes continue running their current jobs even if the controller goes down — slurmd is autonomous enough to keep a running job alive. Only new job scheduling stops until the controller recovers.