Every FIS experiment starts with an Experiment Template — a reusable blueprint that defines exactly what will break, what will be broken, and what will stop the experiment if things go too far. Think of it as a controlled demolition plan: you specify the charges, the target wall, and the safety perimeter before anyone lights a fuse.
1. Actions — The "What"
Actions define the fault to inject. FIS provides two categories:
AWS API Actions — native AWS service calls that FIS executes directly:
| Action | Effect |
|---|---|
aws:ec2:stop-instances |
Simulates an instance crash (graceful stop) |
aws:ec2:terminate-instances |
Simulates an unrecoverable instance failure |
aws:rds:reboot-db-instance |
Simulates a database failover (with forceFailover: true) |
aws:eks:terminate-nodegroup-instances |
Terminates Kubernetes worker nodes |
aws:ec2:inject-api-internal-error |
Makes EC2 API calls return errors for targeted accounts |
aws:network:disrupt-connectivity |
Blocks network traffic between resources |
SSM Actions — OS-level faults delivered via AWS Systems Manager:
| Action | Effect |
|---|---|
aws:ssm:send-command with AWSFIS-Run-CPU-Stress |
Spikes CPU to 100% using stress-ng |
aws:ssm:send-command with AWSFIS-Run-Memory-Stress |
Exhausts available RAM |
aws:ssm:send-command with AWSFIS-Run-Disk-Fill |
Fills the filesystem |
aws:ssm:send-command with AWSFIS-Run-Network-Latency |
Adds artificial latency to outbound traffic using tc (traffic control) |
aws:ssm:send-command with AWSFIS-Run-Kill-Process |
Kills a named process (e.g., nginx, java) |
Actions can be chained with duration and start-after dependencies. For example: "Run CPU stress for 5 minutes, then after that completes, terminate 10% of instances." This lets you simulate cascading failures, not just single-point faults.
2. Targets — The "Who"
Targets define which resources receive the fault. FIS supports three selection strategies:
- Resource IDs: Explicit targeting.
i-0abc123def456gets terminated. No ambiguity, maximum control. - Tags and Filters: Dynamic targeting. "All EC2 instances with
Tag:Role=Workerinus-east-1a." This is powerful for fleet-wide experiments. - Selection Mode: Controls the blast radius within the matched set:
ALL— every matched resource is affected.COUNT(N)— exactly N resources are affected (e.g., terminate 3 instances).PERCENT(N)— N% of matched resources are affected (e.g., terminate 10% of the ASG).
The PERCENT mode is especially useful for production experiments. Terminating 10% of a 50-instance fleet tests your auto-healing without risking a full outage.
3. Stop Conditions — The Safety Brake
Stop conditions are the most important part of any FIS experiment, and the most commonly skipped by teams new to chaos engineering. Skipping them is like doing a controlled burn without a firebreak.
A stop condition connects FIS to a CloudWatch Alarm. You define the alarm independently (e.g., "5XX error rate > 5% for 1 minute"), then reference it in the experiment template. During the experiment, FIS polls the alarm state. If the alarm transitions to ALARM, FIS:
- Immediately stops all in-progress actions.
- Attempts to roll back reversible actions (e.g., restart stopped instances).
- Records the stop reason in the experiment log.
This mechanism allows you to run experiments in production with a defined safety envelope. The alarm represents your "this is too much damage" threshold. If the experiment crosses it, FIS stops automatically — no human needs to be watching.
Best practice: Define stop conditions that are more sensitive than your SLO breach threshold. If your SLO is 99.9% availability, set the stop condition alarm at 99.5% — giving you a buffer to stop the experiment before you actually breach your SLO.