Defining "Normal"
Before you can break something, you need to know what it looks like when it's working correctly. This is where your SLIs and SLOs (covered in earlier topics) become directly useful. Your Steady State is the measurable, baseline behavior of your system when everything is healthy.
The critical insight here is that steady state should be defined in terms of user-facing outcomes, not internal system metrics. CPU usage being low doesn't tell you if users are happy. A successful checkout does.
- Good Steady State: "Our checkout success rate is >99% and our p95 latency is <500ms, as measured by our APM tool."
- Bad Steady State: "CPU usage is low and no error logs are appearing." (Internal metrics that don't directly reflect user experience.)
Forming the Hypothesis
The hypothesis is a precise, falsifiable prediction. It follows the structure:
"We hypothesize that if [specific failure condition], then [specific system behavior] will occur, and the steady state metric [X] will remain within [Y threshold]."
Example:
"We hypothesize that if we kill one of the three payment service pods, Kubernetes will reschedule it within 30 seconds, the remaining two pods will handle the load via the load balancer, and the checkout success rate will remain above 99%."
This hypothesis is good because it is specific, measurable, and falsifiable. You can run the experiment and get a clear yes/no answer.
Key Point: A hypothesis you can't falsify is not a hypothesis — it's a wish. "The system will be fine" is not a hypothesis. "The p95 latency will remain below 500ms" is.