Maintaining Quorum
Etcd uses the Raft consensus algorithm to keep all members in sync. Raft requires that a majority of members — called a quorum — agree before any write is committed. This majority requirement is what makes Etcd fault-tolerant: a minority of nodes can fail without data loss.
The quorum formula is: floor(n/2) + 1, where n is the total number of members.
| Cluster Size | Quorum Required | Tolerated Failures |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
The cluster size Raft uses is the registered member count, not the number of currently healthy members. This distinction is the source of the most dangerous operational mistake in Etcd management.
The Dangerous Mistake: Adding Before Removing
When a node fails permanently, the instinct is to provision a replacement and add it to the cluster. But if you add the new node before removing the dead one, you increase the registered member count from 3 to 4. Quorum for a 4-node cluster is 3. You now have only 2 healthy original nodes plus 1 new node with no data — 3 participants, but the new node cannot vote until it has synced. Depending on timing, you may drop below quorum and freeze the cluster.
The correct sequence is always: remove first, then add.
Step 1: Identify the Dead Member
Listing the cluster's members shows each one's ID, status, name, and peer/client URLs. A member whose status reads unstarted is the dead one.
Step 2: Remove the Dead Member
Removing that member by its ID drops the registered cluster size from 3 to 2. Quorum is now 2, which you already have with the two healthy nodes. The cluster is stable.
Step 3: Register the New Member
Adding a new member by name and peer URL tells the existing cluster to expect a new peer at the given address. The output includes environment variables (ETCD_INITIAL_CLUSTER, ETCD_INITIAL_CLUSTER_STATE=existing) that you must pass to the new Etcd process when starting it. The new member will start in learner mode, sync the full snapshot from the leader, and then join as a voting member.
Key Point: Never run etcdctl member add without capturing and using the output. The ETCD_INITIAL_CLUSTER_STATE=existing flag is what tells the new node "you are joining an existing cluster" rather than "you are bootstrapping a new one." Starting a new node with new instead of existing will cause it to form its own single-node cluster and corrupt your data.