Decoding the Numbers
High load average tells you something is wrong. It does not tell you what. To diagnose the cause, you need to look at CPU state breakdowns, which top, vmstat, and iostat all provide.
The CPU state columns in top (press 1 to expand per-core):
| Column | Meaning |
|---|---|
us |
User space — your application code |
sy |
Kernel/system calls |
ni |
Processes with adjusted nice priority |
id |
Idle |
wa |
I/O Wait — CPU idle, waiting for disk/network |
hi |
Hardware interrupt handling |
si |
Software interrupt handling |
st |
Steal — time stolen by the hypervisor (cloud VMs only) |
Scenario A: CPU-Bound (The Math Cruncher)
top - load average: 4.00, 3.95, 3.90
%Cpu(s): 98.5 us, 1.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
Load is high because processes are actively computing. This is "good load" — the hardware is being used efficiently. The fix is either to optimize the code, add more CPU cores, or distribute the work.
Scenario B: I/O-Bound (The Disk Trasher)
top - load average: 4.00, 4.10, 4.20
%Cpu(s): 0.8 us, 0.5 sy, 0.0 ni, 2.1 id, 96.3 wa, 0.0 hi, 0.3 si, 0.0 st
Load is high but CPU is nearly idle. Processes are stuck in D-state waiting for disk. Adding more CPU cores will not help. The fix is faster storage, fixing a broken mount, or reducing I/O pressure.
Scenario C: Hypervisor Steal (The Cloud Surprise)
%Cpu(s): 5.0 us, 2.0 sy, 0.0 ni, 10.0 id, 0.0 wa, 0.0 hi, 0.0 si, 83.0 st
%st (steal) is high. The hypervisor is giving your VM's CPU time to other tenants. Your application is slow not because of your code or your disk — it's because the physical host is oversubscribed. The fix is to move to a different host, upgrade your instance type, or switch to a dedicated host.
Practical diagnosis workflow: start by checking load against core count, then look at the CPU state breakdown to see which column dominates. If %wa is high, drill into per-device I/O utilization and per-process I/O to find the culprit. Finally, cross-reference against which processes are actually sitting in D-state.
A classic SRE interview question: "A server's load average is 50 but CPU usage is under 5%. What's happening and how do you diagnose it?" The answer: processes are in D-state (Uninterruptible Sleep), almost certainly waiting on I/O — a slow disk, a hung NFS mount, or a storage network issue. Diagnose with iostat -x, iotop, and ps aux | awk '$8=="D"'. The fix depends on the root cause: it could be a dead NFS server, a failing disk, or a misconfigured storage path.