Locks That Expire
On a single machine, if a thread crashes while holding a mutex, the operating system reclaims it. The OS knows which thread owns which lock and cleans up on process death. In a distributed system, no such authority exists. If a worker acquires a lock and then its machine loses power, the lock entry in Redis or the database just sits there — forever. No other worker can ever acquire it. The system is permanently stuck.
The solution is to make locks self-destruct. A lease is a lock with a built-in expiry: a Time-To-Live (TTL). When a worker acquires the lock, it says: "I am claiming resource X for the next 30 seconds." The lock system sets a timer. If the worker finishes early, it explicitly releases the lock. If the worker crashes and never releases it, the timer fires and the lock disappears automatically, allowing another worker to take over.
How it looks in Redis: acquiring means setting the lock key to a unique owner ID, conditioned on the key not already existing, with a millisecond expiry attached — the command returns success if acquired, or nothing if another worker already holds it. Releasing means running a small Lua script that checks the stored value still matches your own ID before deleting the key, so the check-then-delete happens atomically.
The unique ID (worker-abc-uuid-1234) is critical. Without it, Worker A could accidentally release a lock held by Worker B. The release script checks ownership before deleting — and the Lua script makes that check-then-delete atomic.
The Lease Trade-off: What If Work Takes Longer Than the TTL?
Leases introduce a fundamental tension. Set the TTL too short, and a slow-but-healthy worker loses its lock mid-task. Set it too long, and a crashed worker's lock blocks others for a long time before expiring.
A common mitigation is a watchdog thread: a background thread that periodically renews the lease while the main thread is still working. Redisson (the Java Redis client) implements this automatically — it extends the lock TTL every 10 seconds as long as the owning thread is alive.
Worker acquires lock (TTL = 30s)
├── Main thread: doing work...
└── Watchdog thread: every 10s, call EXPIRE job:daily-email 30
├── If main thread is alive → renew
└── If main thread crashes → watchdog also dies → TTL expires naturally
The watchdog pattern works well in practice, but it has a critical failure mode: a Stop-the-World Garbage Collection (GC) pause freezes the entire JVM process — including the watchdog. If the GC pause outlasts the TTL, the lock expires while the main thread is still mid-task — and the thread has no way to know it lost the lock until it tries to act and gets rejected.