Simulating Reality, Not Randomness
The Nemesis is not a random chaos monkey. It is a targeted fault injector designed to trigger the specific failure modes that distributed consensus algorithms are supposed to handle. The faults it injects are things that happen in real data centers—not theoretical disasters.
Common Nemesis attacks include:
1. Network Partitioning
Uses Linux iptables rules to drop packets between specific nodes. For example, isolating the Leader from all Followers, or splitting the cluster into two groups of unequal size. This directly tests whether the cluster correctly refuses to elect two leaders simultaneously (the "split-brain" problem). Under the hood, this is just a pair of iptables DROP rules blocking inbound and outbound traffic to the target node's address.
2. Clock Skew
Forces the system clock on a server to jump forward or backward by seconds or minutes using ntpdate or direct manipulation. Many distributed systems use time-based leases to determine who holds a lock. A clock jump can cause a node to believe its lease has expired (and step down) or, worse, believe it still holds a lease when it does not.
3. Process Pausing
Sends SIGSTOP to freeze a database process for 10–30 seconds, then SIGCONT to resume it. This simulates a long Stop-the-World Garbage Collection pause in JVM-based databases (Cassandra, Elasticsearch, HBase). A paused leader may believe it is still the leader when it resumes, even though the cluster has elected a new one.
4. Process Killing
Sends kill -9 to abruptly terminate a node, simulating a server crash. Unlike a graceful shutdown, kill -9 gives the process no time to flush buffers or release locks.
5. Disk Corruption / Latency
Newer Jepsen tests inject disk I/O errors or artificial latency using tools like tc (traffic control) to simulate slow disks or saturated networks.
The Nemesis operates on a schedule: inject a fault, wait, heal the fault, wait, repeat. This gives the cluster time to react to each fault and attempt recovery—which is often when the most interesting bugs surface.
Key Point: The most dangerous bugs are not triggered by the fault itself, but by the recovery. When a partitioned node rejoins the cluster, does it correctly reconcile its diverged state? Does it discard writes it accepted in isolation? Does it replay them? The Nemesis tests the recovery path, not just the failure path.