The Ultimate Safety Net
Defragmentation and member replacement both assume you still have quorum — at least a majority of nodes are healthy. But what if you lose 2 out of 3 nodes simultaneously? A datacenter power failure, a botched OS upgrade that bricks multiple machines, a cascading disk failure — these scenarios leave you with no quorum and no path to recovery through normal operations. The only way back is a snapshot restore.
A snapshot is a point-in-time binary dump of the entire Etcd database. It captures every key, every value, and the revision at which the snapshot was taken. Restoring from a snapshot creates a brand-new Etcd cluster initialized with that historical state. Any changes made after the snapshot was taken are permanently lost — this is why snapshot frequency matters.
Creating a Snapshot
Taking a snapshot is a single etcdctl snapshot save command (writing to a timestamped file with the member's endpoint and TLS credentials) and should be automated as a scheduled job.
Always verify the snapshot immediately after saving, using etcdctl snapshot status on the saved file. This reports the snapshot's hash, revision, total key count, and total size. A zero revision or zero total keys indicates a corrupt snapshot. Do not rely on it.
Restoring a Snapshot
Restoration is more involved than saving. You are not "loading" data into a running cluster — you are creating a new cluster from scratch using historical data.
Step 1: Stop the Kubernetes API server. If the API server keeps running during restore, it will detect the Etcd disruption and may write error states or attempt reconnections that interfere with the process. On kubeadm clusters, moving the API server and Etcd static Pod manifests out of the watched manifest directory stops both Pods.
Step 2: Move the corrupt data directory out of the way. Never restore on top of a running data directory — rename it aside instead.
Step 3: Run the restore command. etcdctl snapshot restore takes the snapshot file plus the new member's name, its initial cluster peer list, a fresh --initial-cluster-token, its advertise peer URL, and the target data directory. The --initial-cluster-token must be a new, unique value different from the original cluster token. This prevents the restored cluster from accidentally communicating with any surviving fragments of the old cluster.
Step 4: Restart Etcd and the API server. Moving the manifests back into the watched directory. The kubelet will detect the manifest files and start both Pods. Etcd initializes from the restored data directory. The API server reconnects and resumes serving requests. The cluster state reflects the moment the snapshot was taken.
A common interview scenario: "Your Etcd cluster is completely down. Walk me through recovery." The answer must include: (1) confirm quorum is truly lost — not just a network partition, (2) locate the most recent verified snapshot, (3) stop the API server before touching Etcd data, (4) restore with a new --initial-cluster-token to avoid split-brain, (5) verify the cluster is healthy before restarting the API server. Skipping step 3 or step 4 are the two most common mistakes candidates make.
Key Point: The recovery time objective (RTO) for an Etcd disaster is directly proportional to how recently you took a snapshot and how quickly you can execute the restore procedure. Teams that have never practiced a restore in a staging environment will be slow and error-prone during a real incident. Treat snapshot restore as a fire drill — run it quarterly.