Writing Your First Job Script
To run a workload on Slurm, you write a standard bash script with a special twist: lines beginning with #SBATCH are not comments — they are directives to the scheduler. The Slurm client reads these lines before the script ever executes and uses them to determine resource requirements, time limits, and output handling.
This design is elegant: the same file is both a valid bash script (you can run it locally with bash job.sh for testing) and a complete job specification for the cluster.
A realistic job script for distributed LLM training would declare, via #SBATCH directives, the job's name, how many nodes and GPUs it needs, how many CPU cores and how much memory per task, a wall-clock time limit, which partition (queue) to submit to, where to write stdout/stderr, and who to email on completion or failure — followed by the actual environment setup (loading CUDA/NCCL modules, activating a virtualenv) and the training command itself, launched with srun so it runs across every allocated node rather than just the first one.
Anatomy of Key Directives
| Directive | What it controls |
|---|---|
--nodes=4 |
Total number of servers to allocate |
--ntasks-per-node=8 |
Processes per node (usually matches GPU count) |
--gres=gpu:a100:8 |
GPU type and count per node |
--cpus-per-task=8 |
CPU cores per process (for data pipeline parallelism) |
--mem=0 |
0 is a special value meaning "all memory on the node" |
--time=48:00:00 |
Format is DD-HH:MM:SS or HH:MM:SS |
--partition=production |
Which queue to submit to |
--output=logs/%j.out |
%j = job ID, %N = node name, %u = username |
The Essential CLI Commands
A handful of commands cover the entire daily workflow: sbatch submits a job script and returns its job ID; squeue shows the queue (everyone's jobs, or just your own with --user); scontrol show job gives full detail on one job; scancel cancels a job (or, with --state=PENDING, every pending job you own); sinfo reports node status and, with a custom --format, each node's GPU resources; and sacct queries historical job accounting — runtime, state, and GPU-hours consumed — for billing or auditing.
Reading squeue Output
squeue's output lists each job's ID, partition, owner, state, elapsed time, node count, and — for pending jobs — the reason it's waiting. The state codes are worth memorizing:
R= RunningPD= Pending (waiting in queue)CG= Completing (job finished, cleaning up)F= Failed