Partitions: Grouping Nodes by Purpose
A partition is Slurm's term for a queue. The cluster administrator groups nodes into partitions based on hardware type, time limits, or access policy. Users submit jobs to a specific partition based on their needs.
A typical cluster might have:
Partition: debug
Nodes: gpu-[01-02] (2 nodes)
Max time: 00:30:00 (30 minutes)
Max nodes per job: 2
Priority: HIGH (fast turnaround for testing)
Purpose: Test your code before committing to a long run
Partition: production
Nodes: gpu-[01-64] (64 nodes)
Max time: 7-00:00:00 (7 days)
Max nodes per job: 32
Priority: NORMAL
Purpose: Real training runs
Partition: preemptible
Nodes: gpu-[01-64] (same nodes, shared)
Max time: 7-00:00:00
Priority: LOW (can be preempted by production jobs)
Purpose: Opportunistic use of idle capacity at lower cost
Available partitions can always be listed with a single sinfo summary command.
QoS: Fine-Grained Policy Control
Quality of Service (QoS) layers additional policies on top of partitions. While partitions control which nodes you can use, QoS controls how much and at what priority.
Preemption: A high-priority QoS can evict a lower-priority job from its nodes. The evicted job is either requeued (put back in the pending queue) or killed, depending on configuration. This is how research leads get immediate access to GPUs even when the cluster is full — a job submitted under a preemptible QoS can be evicted by one submitted under high_priority.
Fair Share Scheduling: Slurm tracks cumulative GPU-hours consumed by each user and team. If your team used 80% of the cluster last week, your fair share score drops, and your jobs are deprioritized this week relative to teams that used less. This prevents any single team from monopolizing the cluster indefinitely.
The fair share score decays over time (configurable, typically a half-life of days to weeks), so past heavy usage doesn't permanently penalize a team.
Association Limits: QoS can enforce hard limits:
Max running jobs per user: 4
Max GPUs per user at once: 64
Max wall time: 48 hours
These prevent a single user from accidentally (or intentionally) consuming the entire cluster.
A common interview question at HPC-adjacent companies: "How does Slurm handle priority when multiple jobs are waiting?" The answer involves three factors multiplied together: (1) Job priority set by the administrator or QoS; (2) Fair share score based on historical usage relative to allocation; (3) Age — jobs gain priority the longer they wait (to prevent starvation). The final priority score is a weighted sum of these factors, configurable via PriorityWeightFairshare, PriorityWeightAge, etc. in slurm.conf.