A Cluster Divided
What happens if a network failure cuts your cluster in half? This is called a network partition, and it is one of the most dangerous failure modes in distributed systems.
Imagine a 5-node cluster. A network switch fails, isolating 2 nodes from the other 3:
* Partition A: Has Nodes 1, 2, 3 (3 nodes).
* Partition B: Has Nodes 4, 5 (2 nodes).
Without a robust consensus algorithm, both partitions might think the other side is dead. Both might elect their own leader and start accepting writes from clients. This is a split-brain scenario. You now have two different, diverging versions of the "truth." When the network heals, you have two conflicting histories with no automatic way to determine which is correct. Data is corrupted or lost.
Split-brain is not a theoretical concern — it has caused real-world data loss incidents at major companies. It is the primary failure mode that consensus algorithms exist to prevent.
How Consensus Fixes It
Consensus algorithms like Raft and Paxos prevent split-brain by requiring a quorum for every decision:
* Partition A (3 nodes) has a quorum (3 out of 5). It can elect a leader and continue to process writes normally.
* Partition B (2 nodes) does not have a quorum. It cannot elect a leader. Any candidate in Partition B will send RequestVote RPCs but will never receive enough votes to win. The partition becomes unavailable for writes until the network heals.
This is a deliberate trade-off: the system chooses consistency over availability for the minority partition. This maps directly to the CAP theorem — under a network partition (P), you must choose between Consistency (C) and Availability (A). Raft and Paxos choose C: the minority partition refuses to serve writes rather than risk inconsistency.
Interviewers frequently ask: "What happens to a Raft cluster during a network partition?" Walk through the quorum math explicitly: for a 5-node cluster, quorum is 3. The majority partition (3+ nodes) continues operating. The minority partition (2 nodes) cannot commit writes — candidates time out repeatedly without winning elections. When the partition heals, the minority nodes' logs are brought up to date by the new leader via the standard AppendEntries mechanism. No manual intervention is required.