When Bugs Become Emergent Behavior
Everything discussed so far — Bohrbugs, Heisenbugs, Mandelbugs, software rot — becomes dramatically harder in distributed systems. When your application runs on a single machine, the failure modes are bounded. When it runs across dozens of services, networks, and data stores, entirely new categories of anomalies emerge that don't exist in single-process software.
The Eight Fallacies of Distributed Computing (formulated by L. Peter Deutsch and others at Sun Microsystems) describe the false assumptions that cause these anomalies:
- The network is reliable
- Latency is zero
- Bandwidth is infinite
- The network is secure
- Topology doesn't change
- There is one administrator
- Transport cost is zero
- The network is homogeneous
Every one of these assumptions, when violated, produces bugs that look like Heisenbugs or Mandelbugs — intermittent, environment-dependent, and resistant to local debugging.
Common distributed system anomalies:
- Split brain: A network partition causes two nodes to both believe they are the leader, leading to conflicting writes. Neither node is "buggy" in isolation — the bug is the interaction.
- Cascading failures: Service A slows down, causing Service B's connection pool to fill up, causing Service C to timeout, causing the load balancer to route all traffic to Service D, which then also fails. Each individual failure is a consequence of the previous one.
- Clock skew: Two servers disagree on what time it is. Events that happened in order A→B appear as B→A in the logs, making debugging nearly impossible. Distributed tracing with logical clocks (Lamport timestamps) solves this.
- Retry storms: A failed request triggers retries from the client. Each retry also fails and triggers more retries. The system drowns in retry traffic, making the original failure worse. Exponential backoff with jitter is the standard mitigation.
Clock skew is easy to underestimate until you see it in practice: two servers, both confident their clock is correct, can drift a couple of seconds apart — enough to break distributed consensus or make event logs show effects arriving before their causes. Keeping servers synchronized to a shared time source is table stakes for any distributed system that reasons about ordering.
Why this matters for the bug taxonomy:
In distributed systems, the line between a "bug" and "expected behavior under failure conditions" blurs. A timeout isn't a bug — it's a design decision. But a missing timeout handler is a bug. A network partition isn't a bug — it's physics. But a system that corrupts data during a partition is a bug. Senior engineers think about failure modes as part of the design, not as surprises to be debugged after deployment.
Key Point: In distributed systems, the most important anomalies are not bugs in any single component — they are emergent behaviors that arise from the interaction between components under adverse conditions. You cannot unit-test your way out of these. You need integration tests, chaos engineering (deliberately injecting failures to observe system behavior), and game days that simulate real failure scenarios. The goal is to discover these emergent behaviors in a controlled environment before they discover you in production.