Triage Before You Diagnose
A doctor doesn't start with an MRI. They check pulse, blood pressure, and temperature first. When a server is "slow" or "broken," the same principle applies: check the machine's vitals before you look at application code. This prevents the classic mistake of spending 30 minutes reading logs when the real problem is a full disk.
The USE Method
Brendan Gregg's USE Method gives you a structured checklist for every hardware resource (CPU, memory, disk, network):
- Utilization: How busy is the resource? (e.g., CPU at 95% means it's nearly saturated.)
- Saturation: Is work queueing up beyond capacity? (e.g., Load Average of 32 on a 4-core machine means 28 tasks are waiting.)
- Errors: Are there hardware or kernel-level errors? (e.g.,
dmesgshowing disk read errors or OOM kills.)
Apply USE to each resource in order. The first resource where you find a U, S, or E anomaly is your starting point for deeper investigation.
The Rapid-Fire Vitals Toolkit
Five checks, each answering one question in under two seconds, rule out an entire category of failure before you ever look at application code:
- Load average — is the system overwhelmed? A load average above the CPU count means work is queuing up somewhere.
- Kernel errors — did the OOM Killer terminate your app, or did a disk report a hardware error? Kernel-level failures surface here first.
- Memory and swap — is the system swapping heavily? Growing swap usage points to a memory problem, not a CPU problem.
- Disk space — is any filesystem 100% full? A full disk silently breaks log rotation, database writes, and application temp files.
- A one-second CPU/memory/I-O snapshot — showing the run queue, how many processes are blocked on I/O, swap activity, and the percentage of time spent waiting on disk.
A full disk is the single most common "embarrassing" production incident, precisely because it fails silently. Checking it first takes seconds and has saved countless on-call engineers hours of misdirected investigation.