The Mystery Outage
Imagine your system is running fine. Suddenly, a small database hiccup occurs. It lasts only 2 seconds.
However, your system crashes. You fix the database. You reboot the servers. But the system stays crashed. The load is normal, the code is correct, and the hardware is healthy. Yet, the system refuses to recover.
This is a Metastable Failure.
The Definition
A system is Metastable if it works fine under normal conditions, but if pushed hard enough by a trigger — even a brief, transient one — it falls into a failed state and stays there even after the trigger is gone.
What makes this different from an ordinary outage is the Positive Feedback Loop. The act of failing creates more work, which causes more failure. The system enters a death spiral where it is technically stable — stable in its brokenness — and cannot self-recover. It requires drastic human intervention, like cutting off all traffic, to escape.
The term comes from chemistry: a metastable state is one that appears stable but is not at its lowest energy level. A small perturbation can cause a sudden, irreversible transition to a different state. Supercooled water is a classic example — liquid below 0°C, but one tap on the glass and it instantly freezes solid.
Interviewers often ask: "What's the difference between a cascading failure and a metastable failure?" A cascading failure is a chain reaction where one component's failure causes another's. A metastable failure is specifically characterized by self-sustaining persistence — the system stays broken after the root cause is resolved because the failure mode itself generates the load that perpetuates the failure. The key diagnostic question is: "Would the system recover if we just waited?" If no, it's likely metastable.