The "Happy Path" Fallacy
Imagine rehearsing a fire drill only when the building is empty, the sprinklers are tested, and the exits are propped open. That rehearsal tells you almost nothing about how people will behave in a real fire. Most software testing works the same way: it assumes the network is up, the database is fast, and every dependency responds in milliseconds. This is called the "happy path," and it is a dangerous fiction.
In production, Availability Zones fail, disks fill up, APIs throttle, and processes crash. The only way to know your system can survive these events is to cause them deliberately — in a controlled way — before they happen accidentally.
AWS Fault Injection Service (FIS) is a fully managed service for Chaos Engineering: the practice of intentionally introducing failures into a system to discover weaknesses before they become outages. The core idea is simple — if you can break your system on purpose and it recovers, you can be confident it will recover when something breaks by accident.
What separates FIS from a manual script that terminates instances is precision and integration. FIS is deeply integrated with AWS APIs and IAM, which means you can express targets like "terminate 20% of EC2 instances in the prod VPC that have the tag Version=2.0 and are in us-east-1a" — and FIS will execute that exactly, with a safety brake that stops the experiment automatically if things go wrong.
Key terms defined:
- Chaos Engineering: A discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions.
- Fault injection: Deliberately introducing errors, latency, or resource exhaustion into a running system.
- Blast radius: The scope of impact an experiment can have — how many resources, users, or services are affected.
Interviewers often ask: "How do you test resilience?" A weak answer is "we have unit tests and staging." A strong answer references chaos engineering principles: "We use AWS FIS to run controlled fault injection experiments in production during low-traffic windows, with CloudWatch Alarm stop conditions to automatically halt experiments if error rates exceed thresholds. This gives us empirical evidence of recovery behavior, not just theoretical confidence."