The Single Source of Truth
etcd is a distributed key-value store purpose-built for storing configuration data that must be consistent and highly available. In Kubernetes, it is the only persistent storage layer. Everything else — the API Server, the scheduler, the controllers — is stateless and derives its understanding of the world from etcd.
What lives in etcd:
- The desired state of every object: Deployments, Services, ConfigMaps, Secrets, Namespaces, RBAC rules, and more.
- The current reported state of nodes and Pods.
- Cluster metadata like API discovery information.
Because etcd is the single source of truth, losing etcd data without a backup means losing your cluster entirely. The API Server can be restarted and will immediately resume serving from etcd. The scheduler can be restarted and will re-read pending Pods from etcd. But etcd itself has no upstream source to recover from — it is the source.
In production, etcd backup is non-negotiable. The standard tool for taking a point-in-time snapshot is etcdctl, authenticated against the cluster's etcd certificates.
How etcd Achieves High Availability: The Raft Algorithm
A single etcd instance is a single point of failure. Production clusters run etcd as a cluster of 3 or 5 nodes (always an odd number — explained below).
etcd uses the Raft consensus algorithm to keep all nodes consistent. Here is how it works at a high level:
- Leader Election: One etcd node is elected the leader. All write requests go to the leader.
- Log Replication: When the leader receives a write, it appends it to its log and replicates it to all follower nodes.
- Commit: Once a quorum (majority) of nodes acknowledge the write, it is committed and the leader responds to the client.
- Fault Tolerance: If the leader fails, the remaining nodes elect a new leader automatically.
Why odd numbers? Quorum requires a majority: (n/2) + 1 nodes must agree. With 3 nodes, quorum is 2 — you can lose 1 node and still operate. With 4 nodes, quorum is still 3 — you can still only lose 1 node, but you're paying for an extra machine. With 5 nodes, quorum is 3 — you can lose 2 nodes. Adding a 4th node buys you nothing in fault tolerance, which is why even-numbered clusters are avoided.
| Cluster Size | Quorum Needed | Nodes That Can Fail |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
A common interview question is: "Why does Kubernetes recommend an odd number of etcd nodes?" The answer is about quorum efficiency. An even number of nodes (e.g., 4) provides the same fault tolerance as the next lower odd number (3) because quorum still requires a majority. You pay for an extra node without gaining any additional failure tolerance. Always run 3 or 5 etcd nodes in production.