When Push Comes to Shove
What happens if RAM is full and swap is also full? The system has literally nowhere left to put data. The kernel cannot fulfill new memory requests. Without intervention, the entire system would freeze.
Enter the Out Of Memory (OOM) Killer — a kernel subsystem whose job is to save the system by sacrificing one or more running processes to free up memory. It's brutal, but the alternative is a complete system hang.
How does it choose a victim?
The OOM killer calculates an oom_score for every process. The score is visible at /proc/<PID>/oom_score and ranges from 0 to 1000. The process with the highest score is killed first.
The score is calculated roughly as:
oom_score ≈ (process RSS memory / total RAM) × 1000
Then adjusted based on:
- Root processes: Score is reduced (they're assumed more important).
- Process age: Longer-running processes get a slight reduction.
- oom_score_adj: A manual adjustment you can apply (covered next).
You can see the OOM killer's decisions in the kernel log via dmesg, filtered for OOM-related messages. A typical OOM kill log entry looks like:
Out of memory: Kill process 14234 (java) score 872 or sacrifice child
Killed process 14234 (java) total-vm:4194304kB, anon-rss:3670016kB
What triggers the OOM killer?
The OOM killer is a last resort. It's triggered when:
1. A memory allocation fails and cannot be satisfied by reclaiming page cache or swapping.
2. The kernel has exhausted all other reclaim strategies.
In containerized environments, the OOM killer is also triggered by cgroup memory limits — when a container exceeds its memory limit, the kernel OOM-kills a process within that cgroup, even if the host has plenty of RAM. This is why Kubernetes pods show OOMKilled as a termination reason.