Chaos Engineering Is a Process, Not a Button
Running FIS without a structured process is just breaking things randomly. The value of chaos engineering comes from the scientific method: form a hypothesis, run a controlled experiment, measure the outcome, and act on the results. AWS and the broader chaos engineering community call structured chaos sessions GameDays.
The Five-Step GameDay Workflow
Step 1 — Define the Steady State
Before injecting any fault, establish what "normal" looks like. Identify the metrics that prove your system is healthy:
- Request success rate > 99.5%
- P99 latency < 500ms
- Auto Scaling Group desired capacity = actual capacity
These become your baseline and your stop condition thresholds.
Step 2 — Form a Hypothesis
Write a specific, falsifiable statement: "If we terminate 20% of the web tier instances, the Auto Scaling Group will replace them within 90 seconds and the error rate will not exceed 1%."
A vague hypothesis ("the system will be fine") produces no learning. A specific hypothesis produces either confirmation or a bug report.
Step 3 — Minimize the Blast Radius
Start small. Run the experiment in staging first. When moving to production, use PERCENT(10) before PERCENT(50). Schedule during low-traffic windows. Define stop conditions before you start.
Step 4 — Run the Experiment and Observe
Start the FIS experiment. Watch your dashboards in real time — CloudWatch metrics, application logs, APM traces. Do not walk away. The goal is to observe the system's actual behavior, not just whether it survived.
Step 5 — Analyze and Fix
Compare observed behavior to your hypothesis. If the ASG took 4 minutes instead of 90 seconds, investigate why. Common findings:
- Health check grace periods are too long.
- Connection draining timeouts are misconfigured.
- The application does not retry on connection failure.
- A downstream dependency has no circuit breaker.
Document findings and fix them. Then re-run the experiment to verify the fix. This is the feedback loop that actually improves resilience.
Experiment Logging and Audit
Every FIS experiment produces a detailed log: which actions ran, which targets were affected, when stop conditions triggered, and the final state. These logs are stored in CloudWatch Logs and can be exported to S3. For compliance-sensitive environments, this audit trail proves that chaos experiments were controlled and bounded — important for SOC 2 and ISO 27001 audits.