Protecting the VIPs
Sometimes the OOM killer gets it wrong. It might decide your critical database server is the best victim because it uses the most RAM — even though killing it would cause far more damage than killing a batch job.
You can influence the OOM killer's decision by adjusting oom_score_adj for a specific process — readable and writable directly through /proc/<pid>/oom_score_adj, alongside the calculated /proc/<pid>/oom_score itself. This value ranges from -1000 to +1000 and is added to the process's calculated score.
Practical adjustments:
oom_score_adj |
Effect |
|---|---|
-1000 |
Process is immune to the OOM killer. Used for sshd, systemd, critical daemons. |
-500 to -100 |
Process is strongly protected. Good for databases, monitoring agents. |
0 |
Default. No adjustment. |
+500 to +999 |
Process is a preferred sacrifice. Good for batch jobs, disposable workers. |
Systemd integration: If your process is managed by systemd, set OOMScoreAdjust in the [Service] section of the unit file instead of writing to /proc directly — the value is applied automatically on every start, which is the more durable and auditable way to protect (or sacrifice) a specific service.
Setting -1000 is dangerous. If a process marked -1000 has a memory leak, the OOM killer cannot touch it. The system will kill everything else first, potentially leaving you with a frozen system where only the leaking process survives. Reserve -1000 for truly critical, well-understood processes like sshd (so you can still SSH in to diagnose the problem).
Interviewers frequently ask: "A Kubernetes pod keeps getting OOMKilled even though the node has plenty of RAM. What's happening?" The answer: the pod is hitting its cgroup memory limit, not the node's total RAM. The OOM killer is invoked at the cgroup level. The fix is either to increase the pod's memory limit, reduce the application's memory usage, or investigate a memory leak. This is distinct from a node-level OOM event.