Disruption & Consolidation: Cleaning Up

The Fragmentation Problem

Scaling up is straightforward. Scaling down is where clusters silently hemorrhage money.

Over time, as workloads shift, a cluster becomes fragmented: many nodes running at low utilization. You might have 10 nodes each at 15% CPU — effectively paying for 10 nodes while using the equivalent of 1.5. Neither CAS nor Karpenter can prevent this entirely, but Karpenter has a far more aggressive and configurable response.

Karpenter Consolidation

Karpenter's consolidation loop runs continuously. It evaluates every node and asks: "Could all the pods on this node fit on other existing nodes?" If yes, it initiates a disruption sequence:

1. Cordon the node (mark it unschedulable so no new pods land on it).
2. Drain the node (evict all pods, respecting PodDisruptionBudgets).
3. Terminate the EC2 instance.

Karpenter also performs consolidation by replacement: "Could I replace two half-empty nodes with one smaller, cheaper node?" This is more aggressive than simple deletion and can significantly reduce node count over time.

Disruption Budgets

Consolidation is powerful but dangerous if unconstrained. Evicting pods causes restarts, which causes latency spikes or brief unavailability. Disruption budgets in the NodePool spec act as guardrails:

disruption:
  consolidationPolicy: WhenUnderutilized   # or WhenEmpty
  consolidateAfter: 30s                    # how long a node must be underutilized
  budgets:
    - nodes: "10%"                         # max 10% of nodes disrupted at once
    - schedule: "0 9 * * 1-5"             # during business hours...
      duration: 8h
      nodes: "0"                           # ...disrupt nothing

The schedule field uses cron syntax. This lets you freeze consolidation during peak traffic windows (e.g., Black Friday, end-of-month batch runs) while allowing aggressive cleanup during off-hours.

Key insight for senior engineers: Karpenter respects PodDisruptionBudgets (PDBs) defined on your Deployments. If a PDB says "always keep at least 2 replicas running", Karpenter will not evict a pod if doing so would violate that budget. This means your application-level availability guarantees are the primary control plane for disruption safety — not Karpenter's own settings. Always define PDBs for production workloads.