Correlation Does Not Imply Causation

The Spurious Connection

Modern infrastructure generates millions of data points per minute. CPU usage, memory consumption, request rates, error counts, disk I/O, network throughput, garbage collection pauses, queue depths—the list is enormous. With that many metrics, if you look hard enough, you will always find two graphs that move together. This is a mathematical certainty, not a sign of a real relationship.

• Correlation: Variable A moves up when Variable B moves up (or down when down). They have a statistical relationship. Example: Latency increases when Disk I/O increases.
• Causation: Variable A directly forces Variable B to change through a specific mechanism. Example: Disk I/O increases → database reads slow down → query latency increases.
• Confounding Variable: A hidden third factor (C) causes both A and B to change, creating the illusion that A and B are related to each other. Example: A network switch failure causes both increased retries (which looks like latency) and dropped packets (which looks like errors). Latency and errors don't cause each other—they're both symptoms of the switch failure.

The distinction matters because your remediation strategy depends entirely on which relationship is real. If A causes B, fix A. If C causes both A and B, fixing A does nothing—you must find and fix C.

SRE Example: The Hidden Third Variable

You're on-call and see "Error Rate" and "Latency" spike at the same time on your dashboard. Your brain immediately constructs a narrative:

• Theory A (Direct Causation): High latency is causing timeouts, which are counted as errors. Fix: Optimize the slow service.
• Theory B (Reverse Causation): Errors are causing retries, which are increasing latency. Fix: Fix the error source.
• Theory C (Confounding): A network switch failure (hidden variable) is dropping packets, causing both retries (which manifest as latency) and connection failures (which manifest as errors). Fix: Escalate to the network team.

All three theories explain the same dashboard. Only one is correct. If you pick Theory A and spend an hour optimizing query performance, but the real cause is Theory C, you've wasted an hour while users continue to experience failures. The only way to distinguish between these theories is to look for additional evidence: network metrics, packet loss data, traceroute results, and error message content.

This is why the first question during any incident should not be "what do I fix?" but "what additional data do I need to distinguish between my competing hypotheses?"

The Three Causal Relationships

Understanding these three causal relationships is fundamental to debugging any system:

1. Direct Causation: A → B
   Example: Database connection pool exhaustion → HTTP 503 errors. The mechanism is clear: application requests a connection, pool is empty, request fails, 503 returned to client.

2. Reverse Causation: B → A
   Example: You think "slow queries cause high CPU," but actually "high CPU (from a runaway log-processing job) causes slow queries" because the database process can't get enough CPU cycles. The arrow points the opposite direction from your initial assumption. Treating this as direct causation leads you to optimize queries that are actually fine—the moment the log job stops, query performance returns to normal.

3. Confounding: C → A and C → B
   Example: A DNS resolver failure (C) causes both "increased API latency" (A, because DNS lookups are timing out and falling back to secondary resolvers) AND "increased error rate" (B, because some DNS lookups fail entirely). Latency and errors don't cause each other—they're both symptoms of the DNS problem. Fixing latency won't reduce errors, and fixing errors won't reduce latency.

How to Distinguish Causation from Correlation

In a controlled experiment (like A/B testing), you can isolate variables. During a live incident, you rarely have that luxury. But you can apply these heuristics:

1. Temporal ordering: Does A consistently happen before B? If they always happen simultaneously, suspect a confounding variable.
2. Dose-response: When A gets worse, does B get proportionally worse? If A doubles but B stays the same, the link is weak.
3. Mechanism plausibility: Can you describe the physical chain from A to B? No mechanism, no confidence.
4. Intervention test: If you fix A, does B improve? This is the strongest test but also the most expensive during an incident.
5. Isolation test: Can you find cases where A happens without B, or B without A? If so, the causal link is weaker than it appears.

These heuristics are ordered by cost. Start with temporal ordering and mechanism plausibility—they're free. Reserve the intervention test for when you have high confidence, because taking action during an incident has real consequences.

Key Point: In distributed systems, confounding variables are the norm, not the exception. Shared infrastructure—network, DNS, load balancers, cloud provider control planes, service meshes—creates hidden dependencies that produce correlated symptoms across unrelated services. When multiple seemingly unrelated things break simultaneously, your first instinct should be to look for a shared dependency, not to debug each service independently. The blast radius of a failure is one of the most informative signals you have: narrow blast radius suggests a service-specific bug; wide blast radius across independent services almost always points to shared infrastructure.