Energy States
Think of your system as a ball rolling on a hilly landscape.
- Stable State: The ball sits at the bottom of a valley. If you nudge it (a load spike), it rolls back down. The system self-corrects.
- Vulnerable State: As you increase system utilization — say, running at 80–90% CPU — the walls of the valley get lower. A small nudge might now push the ball over the edge.
- Metastable Failure State: The ball has rolled over the edge into a different, deeper hole. It is stuck. Removing the original nudge does nothing — it doesn't magically levitate the ball back over the hill.
The Sustaining Effect
The force keeping the ball in the hole is called the Sustaining Effect. This is almost always Work Amplification: the system is doing more work to fail than it normally does to succeed.
A healthy system processing 1,000 requests/second might consume 40% CPU. The same system in a metastable failure state might process only 200 useful requests/second while consuming 100% CPU — because it is spending the rest of its resources on retries, error handling, connection teardowns, and logging failures. The overhead of failing is itself a load generator.
Why Utilization Is the Hidden Risk Factor
A system running at 40% utilization has deep valley walls. A system running at 90% utilization has shallow ones. This is why metastable failures are so common during peak traffic periods or after a gradual traffic growth that nobody noticed. The system was fine for months — until the valley walls got low enough that a routine deployment or a brief network blip was enough to push it over.