What Is a Lease?
A lease is a time-bounded grant of authority. Instead of saying "you are the leader until someone deposes you," a lease says "you are the leader for the next N seconds." The leader must continuously renew its lease before it expires. If it fails to renew — because it crashed, paused, or lost network connectivity — the lease expires and another node can safely acquire it.
Leases are used in systems like Kubernetes (which uses etcd leases for leader election in the control plane) and Google Chubby (which pioneered the concept for distributed locking).
The basic lease protocol:
1. A candidate writes its identity to a coordination service with a TTL (time-to-live), e.g., SET leader "node-A" EX 10.
2. The leader runs a background thread that renews the lease every few seconds (e.g., every 3 seconds for a 10-second lease).
3. If the leader crashes or pauses, the renewal stops. After 10 seconds, the lease expires.
4. Another candidate can now acquire the lease and become the new leader.
The Clock Skew Problem
Leases depend on time, and time is not reliable in distributed systems. Clock skew is the difference in the current time as measured by two different machines. Even with NTP (Network Time Protocol) synchronization, clocks on different machines can drift by tens or hundreds of milliseconds. In pathological cases (VM live migration, NTP misconfiguration), skew can reach seconds.
This creates a subtle but serious problem. Suppose the leader's lease expires at time T=100 according to the coordination service's clock. The leader's local clock reads T=98 — it thinks it has 2 more seconds. It proceeds to act as leader. Meanwhile, the coordination service's clock reads T=101 — the lease has expired. Another node acquires the lease and is elected. For a brief window, both nodes believe they are the leader.
Mitigations:
- Conservative lease renewal: Renew the lease well before it expires (e.g., at 30% of the TTL remaining), giving a buffer for clock skew and network latency.
- Lease safety margin: When a node's lease expires, it should wait an additional safety margin (equal to the maximum expected clock skew) before acting on its authority, to ensure no other node has already acquired the lease.
- Fencing tokens as the backstop: Even if clock skew causes a brief dual-leader window, fencing tokens at the storage layer ensure that only the most recently elected leader's writes are accepted.