"Take a Message for Me"
Imagine you are using an AP system configured with N=3, W=2. A write comes in. Node A accepts it. Node B accepts it. Node C is temporarily offline due to a quick reboot. Since W=2 and two nodes acknowledged the write, the coordinator tells the client "Success." But Node C is now missing that data — it is inconsistent with Nodes A and B.
How does Node C catch up when it comes back online? Running a full database repair (which compares every piece of data across all nodes) is extremely expensive. The solution for short outages is Hinted Handoff.
When the write coordinator detects that Node C is unreachable, it stores a hint locally on one of the healthy nodes (say, Node A). This hint is a small record that says: "I have a write that belongs to Node C. Here is the data and a timestamp."
Node A periodically checks whether Node C has come back online. Once Node C is reachable again, Node A hands off the buffered write, and Node C applies it. Node C is now consistent again — without requiring a full repair scan.
Limits and Failure Modes
Hinted handoff is designed for short, temporary outages — typically minutes to a few hours. Most systems (including Cassandra) have a configurable hint window (e.g., 3 hours by default). If Node C is offline longer than the hint window, the hints are discarded to prevent unbounded disk usage on the hint-storing node.
If hints are discarded before Node C recovers, Node C will have a permanent data gap. The only way to fix this is a full anti-entropy repair — an expensive background process that uses Merkle trees to compare and synchronize data across nodes. This is why long node outages in AP systems require explicit operational intervention, not just waiting for the node to come back.
Timeline of a Hinted Handoff:
T=0: Write "X=5" arrives at coordinator
T=0: Node A writes X=5 ✓
T=0: Node B writes X=5 ✓
T=0: Node C is DOWN — coordinator stores hint on Node A
T=0: Client receives "Success" (W=2 satisfied)
T=5m: Node C comes back online
T=5m: Node A detects Node C is alive
T=5m: Node A sends hint to Node C: "Write X=5"
T=5m: Node C applies the write ✓
T=5m: All three nodes now have X=5 — consistency restored