From Paper to Production
SWIM is an academic protocol. Memberlist is HashiCorp's open-source Go library that implements SWIM with the Suspicion refinement, plus several production hardening additions. It is the membership layer inside both Consul (service discovery) and Nomad (workload orchestration). Understanding Memberlist means understanding how real-world service meshes maintain cluster state.
What Memberlist Gives You
Before libraries like Memberlist, managing cluster membership required either hardcoded IP lists (fragile), a central database (single point of failure), or a heavyweight consensus system like ZooKeeper (operationally complex). Memberlist replaces all of that with a self-organizing gossip layer.
When a new node starts up, you give it the address of any single existing node in the cluster. Memberlist then:
1. Joins the gossip pool by contacting that seed node.
2. Receives the full membership list (all known nodes and their states) within seconds.
3. Begins participating in failure detection — pinging random peers, responding to pings.
4. Automatically propagates its own presence to the rest of the cluster.
If a node crashes, the cluster detects it within seconds (configurable) and removes it from the membership list — no operator intervention required.
Memberlist's Transport Layers
Memberlist uses two transport mechanisms simultaneously:
| Transport | Used For | Why |
|---|---|---|
| UDP | PING, ACK, INDIRECT-PING, gossip payloads |
Low latency, low overhead, acceptable to lose occasionally |
| TCP | Full state sync when a new node joins | Reliable delivery needed for the initial complete membership snapshot |
UDP is used for the steady-state gossip because losing an occasional ping is acceptable — the next gossip round will carry the same information. TCP is used for the initial join because a new node needs a complete, reliable snapshot of the current membership state.
Configurable Parameters
Memberlist exposes several tuning knobs that directly map to the SWIM parameters: GossipInterval (how often to gossip), GossipNodes (fan-out — peers contacted per round), ProbeInterval (how often to probe a random node), ProbeTimeout (how long to wait before falling back to an indirect ping), SuspicionMult (the suspicion timeout multiplier), and RetransmitMult (how many times to retransmit a piece of gossip).
The SuspicionMult and RetransmitMult values scale with $\log N$ (cluster size) automatically, so the protocol self-tunes as the cluster grows.
Key Point: Memberlist is not a consensus system. It does not use Raft or Paxos. It makes no guarantees about the order in which nodes learn about events, and two nodes may briefly disagree about whether a third node is alive or suspect. This is the "weakly consistent" part of SWIM. For use cases that require strong consistency (like leader election or distributed locks), you need a separate consensus layer — which is exactly why Consul uses Memberlist for membership but Raft for its key-value store.