Making Way for VIPs
QoS determines what gets killed on an overloaded Node. But what if the entire cluster is full and a critical Pod cannot be scheduled anywhere? This is a different problem — not about killing processes on a Node, but about deciding which already-running Pods should be evicted to make room for a new, more important one.
This is solved by PriorityClasses and Preemption.
PriorityClass is a cluster-scoped Kubernetes object that maps a human-readable name to an integer priority value. Higher integers mean higher priority. You create PriorityClasses once — a production-critical class at a high value, a background-batch class at a low value — and reference them by name in a Pod spec's priorityClassName field.
The Preemption Logic
When a high-priority Pod is stuck in Pending because no Node has enough capacity, the scheduler runs the preemption algorithm:
- Find candidate Nodes: Identify Nodes where, if lower-priority Pods were removed, the pending Pod would fit.
- Simulate evictions: For each candidate Node, calculate the minimum set of lower-priority Pods that must be evicted to free enough resources.
- Choose the least disruptive Node: Prefer the Node where preemption kills the fewest Pods, and among those, the Pods with the lowest priority values.
- Evict and schedule: The selected lower-priority Pods receive a graceful termination signal (respecting their
terminationGracePeriodSeconds). Once enough space is freed, the high-priority Pod is scheduled.
Critical distinction: Preemption respects PodDisruptionBudgets (PDBs). If evicting a Pod would violate its PDB (e.g., it's the last replica of a service that requires at least one replica running), the scheduler will not evict it and will look for another candidate. This prevents preemption from accidentally taking down a service entirely.
Priority values to know:
- System-critical Pods (like kube-apiserver, etcd) use values ≥ 2,000,000,000.
- User workloads should stay below 1,000,000,000.
- Kubernetes reserves the range 1,000,000,000–2,000,000,000 for system use.
Interviewers often ask: "What's the difference between QoS-based eviction and priority-based preemption?" The key distinction: QoS eviction is triggered by node resource pressure (the kubelet acts when a Node is running out of memory or disk) and uses QoS class to choose victims. Priority preemption is triggered by scheduling failure (a Pod cannot be placed anywhere) and uses PriorityClass values to choose victims. They are independent mechanisms that can interact — a Guaranteed Pod with low priority can still be preempted by a BestEffort Pod with high priority.