Why Etcd Needs Care
Think of Etcd as the brain of your Kubernetes cluster — it holds every piece of state: which Pods exist, which Nodes are registered, what Secrets are stored, what the desired configuration looks like. If the brain goes down, the cluster stops functioning. New Pods can't be scheduled, existing workloads keep running but can't be changed, and the API server returns errors.
In small development clusters, Etcd usually runs fine without intervention. But at scale — thousands of Pods, constant deployments, frequent ConfigMap and Secret updates, high churn from autoscalers — Etcd requires active, deliberate maintenance. Three categories of problems emerge:
- Disk fragmentation: Etcd's storage engine accumulates "holes" over time, inflating the database file until it hits a hard quota and the cluster goes read-only.
- Member failures: Losing a node without proper surgery can break the consensus algorithm and make the cluster unable to accept writes.
- Catastrophic data loss: If a majority of nodes fail simultaneously, the only path back is a point-in-time snapshot restore.
Unlike a stateless web server, you can't just restart Etcd to fix these problems. Each requires a specific, ordered procedure. Getting the order wrong can turn a recoverable situation into a full outage. This lesson covers all three, from first principles to the exact commands.