The Zombie Leader Problem
A Zombie Leader is a node that was legitimately deposed but has not yet realized it. The most common cause is a Stop-the-World Garbage Collection (GC) pause — a period where the JVM (or other runtime) freezes all application threads to reclaim memory. During this pause, the node sends no heartbeats, holds no locks, and participates in no elections. From the cluster's perspective, it is dead. From its own perspective, no time has passed at all.
When the GC pause ends, the zombie wakes up mid-execution. It might be holding a reference to a database connection, halfway through writing a record, or about to send a "delete this data" command. It has no idea that 15 seconds elapsed, that its lease expired, and that a new leader has already taken over and written new data to the same location.
Quorums cannot stop this. The zombie is not trying to win an election — it is simply executing code it already started. We need a mechanism that operates at the storage layer, not the election layer.
How Fencing Tokens Work
A Fencing Token (also called an Epoch Number or Generation Number) is a strictly monotonically increasing integer issued by the coordination service every time a new leader is elected. The key insight is that the storage resource — not the leader — is responsible for enforcing the token.
The protocol, step by step:
- Token issuance: The coordination service (ZooKeeper, etcd, or similar) maintains a counter. Every time a leader is elected, the counter increments and the new leader receives the current value as its fencing token.
- Leader-Old was elected with Token
10. -
After Leader-Old is deposed, the New Leader is elected with Token
11. -
Token propagation: Every request the leader sends to a shared resource (a database, a distributed lock, a file system) must include its fencing token as a parameter.
-
Storage-side enforcement: The storage resource maintains a single value: the highest fencing token it has ever accepted. Before processing any write, it checks:
- If
request_token >= stored_max_token: accept the write, updatestored_max_token. -
If
request_token < stored_max_token: reject the write and return an error. -
The zombie is blocked: When Zombie Leader (Token
10) wakes up and tries to write, the storage has already seen Token11. The check10 < 11fails. The write is rejected. The zombie's delayed command does nothing.
Why the Storage Layer Must Enforce This
A common mistake is to implement fencing checks only in the leader's own code — for example, having the leader check "am I still the leader?" before writing. This does not work. The zombie leader's code believes it is still the leader. The check will pass. The write will proceed. The fence must be enforced by an external party that has accurate, up-to-date information: the storage resource itself.
This is a general principle in distributed systems: do not trust a node to correctly assess its own authority. Authority must be validated by the resource being accessed, not by the accessor.
A common interview question is: "Why can't you just check if I am still the leader before writing?" The answer is that this check and the subsequent write are not atomic. Between the check and the write, the node could be paused (GC, OS scheduling), the network could hiccup, or the lease could expire. By the time the write executes, the check result is stale. Fencing tokens move the authority check to the storage layer and make it part of the write operation itself, eliminating this race condition.