Tuning Consensus in Production
Understanding consensus algorithms theoretically is necessary but not sufficient. Production deployments require understanding several practical dimensions.
Election Timeout Tuning
The election timeout is the duration a follower waits before starting an election. It must be tuned carefully:
- Too short: Followers start unnecessary elections when the leader is merely slow (e.g., under GC pause or high load). This causes leadership churn and temporary unavailability.
- Too long: When a leader genuinely fails, the cluster takes a long time to recover. During this window, no writes can be committed.
A common rule of thumb: the election timeout should be at least 10x the typical round-trip time between nodes. For nodes in the same data center (RTT ~1ms), 150–300ms is typical. For nodes across regions (RTT ~50–150ms), timeouts of 1–5 seconds may be appropriate.
Write Latency
Every committed write in a Raft cluster requires at least one round trip from the leader to a quorum of followers. This means write latency is bounded by the network latency to the slowest quorum member. For a 3-node cluster with one follower in a distant region, every write will be as slow as the cross-region RTT. This is a fundamental constraint — you cannot commit faster than your network allows.
Cluster Membership Changes
What happens when you need to add or remove nodes from a running cluster? This is surprisingly dangerous. If you naively add 2 nodes to a 3-node cluster simultaneously, there is a brief window where the old quorum (2 of 3) and the new quorum (3 of 5) can both be satisfied by disjoint sets of nodes — enabling split-brain.
Raft handles this with a joint consensus approach (or the simpler single-server changes approach): membership changes are themselves replicated as log entries, and the cluster transitions through a joint configuration where both the old and new quorum must agree before the change is finalized. In practice, tools like etcd abstract this away — you use etcdctl member add and etcdctl member remove rather than managing it manually.