SWIM Refinement: The Suspicion Mechanism

The Flapping Problem

Even with indirect pings, pure SWIM has a weakness: if a node is temporarily overloaded (a GC pause in a JVM application, a CPU spike, a slow disk causing the process to stall), it may fail to respond to pings for several seconds. SWIM would mark it FAILED, trigger cluster rebalancing (data redistribution, leader re-election), and then the node recovers and tries to rejoin — causing another round of rebalancing. This oscillation is called flapping, and it is expensive.

The Suspicion State

Modern SWIM implementations add an intermediate state between ALIVE and FAILED: SUSPECT.

The state machine becomes:

Here is how it works:

1. Node A's indirect pings to B all fail. Instead of broadcasting B=FAILED, A broadcasts B=SUSPECT.
2. The SUSPECT message propagates through the cluster via gossip.
3. The critical defense mechanism: Node B is still running. It eventually receives a gossip message saying "Hey, the cluster thinks you are suspect." Node B immediately broadcasts a high-priority B=ALIVE (incarnation=2) message. The incarnation number is a monotonically increasing counter that B increments each time it needs to refute a suspicion.
4. Because the ALIVE message carries a higher incarnation number than the SUSPECT message, every node that receives it overwrites the suspect state with alive. The false positive is cancelled.
5. If B does not refute within a configurable timeout (because it is genuinely dead and cannot respond), the cluster promotes B from SUSPECT to FAILED.

Why Incarnation Numbers Matter

The incarnation number is what prevents a stale ALIVE message from overwriting a legitimate FAILED state. Each message carries (node_id, state, incarnation). The rule is simple: a message only overwrites the current state if its incarnation number is greater than or equal to the current incarnation. A node can only increment its own incarnation number (only B can speak for B), which prevents other nodes from falsely "reviving" a dead node.