Chaos Engineering is not random; it is a carefully designed scientific experiment. It follows a strict process to ensure both safety and learning. Think of it exactly like a chemistry lab experiment: you don't just pour random chemicals together — you form a hypothesis, control your variables, and observe the outcome.
The four-step process is:
- Define the "Steady State" (Normal behavior): Establish measurable proof that the system is healthy right now, before you touch anything.
- Form a Hypothesis: Predict what will happen when you introduce a specific failure. Be precise.
- Run the Experiment (Inject the Fault): Introduce the failure in a controlled way, starting with the smallest possible blast radius.
- Verify and Learn: Compare the actual outcome to your hypothesis. Did the system behave as expected? If not, you found a bug.
The Blast Radius is a critical concept: it refers to the scope of potential impact from your experiment. You always start small — one pod, one availability zone, 1% of traffic — and expand only after you've confirmed the system handles it. A well-run chaos experiment should be invisible to end users.
If at any point during the experiment the system becomes unhealthy beyond your defined abort threshold, the experiment must be stopped immediately and the fault reversed. Safety is not optional.