The Post Hoc Fallacy
There is a Latin phrase: post hoc ergo propter hoc ("after this, therefore because of this"). It describes one of the oldest logical fallacies in human reasoning: assuming that because Event B followed Event A, Event A must have caused Event B. In DevOps, this manifests as the Last Changed Thing Bias—the reflexive assumption that whatever was most recently modified must be responsible for the current problem.
- The Scenario: You deploy a minor CSS change to the frontend at 2:55 PM. At 3:00 PM, the database CPU spikes to 100%.
- The Pattern Match: "Roll back the frontend! It broke the DB!"
- The Reality: The frontend change and the DB spike are correlated in time, but not causally linked. The real cause is a scheduled backup job configured to run at 3:00 PM daily, combined with a table that grew past a size threshold this week.
This happens constantly. A team spends 45 minutes rolling back a harmless change, then another 30 minutes re-deploying it after realizing the rollback didn't help, then finally discovers the actual cause—having wasted over an hour of incident response time and extended the outage for every user.
The Mechanism Test
Before you rollback, ask one question: "What is the mechanism?" Can a CSS file physically cause a database CPU spike? If you cannot describe the mechanism of failure—the specific chain of events where change A produces symptom B—you are guessing, not debugging.
A mechanism is not a vague narrative ("the deploy stressed the system"). It is a specific technical chain: "The deploy introduced a new API call on every page load → that API call executes a full table scan → the table scan consumes CPU." Every link in the chain must be technically plausible and, ideally, verifiable.
Here's a framework to apply before taking any remediation action:
| Question | Purpose | Example |
|---|---|---|
| What changed? | List ALL recent changes (deploys, config, infra, cron jobs, traffic patterns) | Frontend deploy, DB backup schedule, marketing campaign launch |
| What's the mechanism? | How could change X physically cause symptom Y? | "CSS change → browser renders differently → no DB interaction. No mechanism." |
| What evidence supports this? | Logs, metrics, traces that prove the causal link | "DB slow query log shows backup queries starting at 3:00 PM" |
| What evidence contradicts this? | Data that breaks the hypothesis | "Frontend deploy touched zero backend code. No new API calls." |
| What would I expect to see if my hypothesis were true? | Predictions that can be verified | "If the deploy caused it, rollback should fix it within seconds" |
The last row is critical. It turns your hypothesis into a testable prediction. If you roll back and the problem persists, you've falsified the hypothesis—now stop pursuing it and move on. Continuing to investigate a falsified hypothesis is one of the most common ways incidents drag on for hours longer than necessary.
Why This Matters at Scale
In a microservices architecture with 50 services, something is always deploying. If your organization deploys 200 times per day across all services, and you experience 3 incidents per week, the probability that an incident happens within 15 minutes of some deployment is extremely high—purely by chance. If you blame the last deploy every time, you'll be wrong far more often than you're right.
Companies with mature incident management track the percentage of rollbacks that actually fixed the issue versus those that were coincidental. Many find that fewer than half of "emergency rollbacks" during incidents address the actual root cause. The rest are expensive distractions that extend mean time to recovery (MTTR).
There is also a cultural cost. Teams that reflexively blame deployments create a chilling effect on deployment frequency. Engineers start batching changes to reduce "rollback risk," which ironically makes each deployment larger and riskier. The Last Changed Thing Bias, left unchecked, can reverse an organization's progress toward continuous delivery.
Key Point: The mechanism test is your strongest defense against the Last Changed Thing Bias. If you cannot articulate a plausible physical chain from cause to effect—"this code change causes this function to execute this query which exhausts this resource"—then your hypothesis is untested speculation, regardless of how compelling the timeline looks. A compelling timeline is evidence of correlation. Only a verified mechanism is evidence of causation.