The Network Partition
A network partition is when a failure in the network infrastructure — a failed switch, a misconfigured firewall, a severed cable — causes a cluster to split into two or more groups of nodes that can communicate internally but not with each other. From each group's perspective, the other group has simply gone silent.
Consider a 5-node cluster where a switch fails, dividing the cluster into two isolated groups:
- Group A: 2 nodes, including the current leader (call it Leader-Old).
- Group B: 3 nodes, none of which is the current leader.
The Sequence of Doom
- Group B's nodes stop receiving heartbeats from Leader-Old. After a timeout, they declare it dead.
- Group B holds an election. Because it has 3 of 5 nodes, it can form a majority. It elects a New Leader from within its group.
- Leader-Old, isolated in Group A, never receives any signal that it has been deposed. It continues operating normally, accepting client writes.
- Clients connected to Group A continue writing to Leader-Old. Clients connected to Group B write to the New Leader.
- When the network heals, the cluster has two divergent write histories. There is no automatic, lossless way to merge them. Data is corrupted.
The Quorum Solution — and Its Limits
The standard first-line defense against split-brain is the quorum: a rule that a leader can only be elected or remain in power if it has the acknowledgment of a strict majority (more than half) of the cluster.
In the example above:
- Group A has 2 of 5 nodes — a minority. It cannot form a quorum. Leader-Old should step down.
- Group B has 3 of 5 nodes — a majority. It can form a quorum and elect a valid new leader.
This works in theory. In practice, there is a race condition. Leader-Old might accept a write milliseconds before it detects it has lost quorum. Or it might be stuck in a GC pause and not detect the loss at all. Quorums prevent a minority from electing a new leader, but they do not prevent a deposed leader from acting on stale authority. That requires a second line of defense.