A Tour of Famous Failures
Understanding what Jepsen found in real systems makes the abstract concepts concrete. These are not hypothetical bugs—they were in production software used by thousands of companies.
MongoDB (2013) — Acknowledged Writes That Were Lost
MongoDB's default write concern at the time was "fire and forget." Jepsen showed that even with w:1 (wait for the primary to acknowledge), writes could be lost during a primary failover. The primary would acknowledge the write, then crash before replicating it. The new primary had no record of the write. MongoDB has since improved its default write concern and replication protocol, but this finding changed how the industry thought about write acknowledgment semantics.
Elasticsearch (2014) — Deleted Documents Resurrected
Elasticsearch was found to resurrect documents that had been deleted. During a network partition, a node that missed the delete operation would rejoin the cluster and its stale copy of the document would win the version conflict resolution. Deleted data came back from the dead.
Cassandra (2013) — Lightweight Transactions Broken
Cassandra advertised "Lightweight Transactions" (LWT) using Paxos for compare-and-swap operations. Jepsen found that under certain partition scenarios, two clients could both successfully execute a CAS operation on the same key with the same precondition—meaning the Paxos implementation had a bug that allowed two "winners" simultaneously.
ZooKeeper (2013) — Surprisingly Solid
ZooKeeper was one of the few systems that passed Jepsen testing with its guarantees intact. This is notable because ZooKeeper explicitly trades availability for consistency (it refuses to serve reads/writes during a partition if it cannot maintain quorum). The lesson: systems that make conservative, explicit trade-offs tend to hold up better under adversarial testing than systems that try to have it all.
The Pattern
Across all findings, a pattern emerges: bugs almost always appear at the intersection of two failure modes. A partition alone is handled correctly. A leader election alone is handled correctly. But a partition during a leader election, or a clock skew during a lease renewal—that's where the bugs hide. Jepsen's nemesis is designed to trigger exactly these intersections.
Key Point: The Jepsen findings did not just fix individual bugs—they changed how the industry writes distributed systems documentation. Vendors now publish explicit consistency guarantees with specific caveats, rather than blanket claims of "strong consistency." This is a direct result of Jepsen's public analyses.