You cannot prepare for a Black Swan event by reading about it. You prepare by practicing failure — deliberately, in a controlled way, before the real event forces you to.
What is a Game Day?
A Game Day (also called a Chaos Engineering exercise or DiRT — Disaster Recovery Test) is a scheduled event where your team intentionally breaks something in production (or a production-like environment) to observe how the system and the team respond.
The goal is not to cause an outage. The goal is to discover your assumptions before reality does.
The Game Day Structure
1. Define the Hypothesis
Before breaking anything, write down what you expect to happen:
"If we terminate all instances in us-east-1a, the load balancer will route traffic to us-east-1b and us-east-1c within 30 seconds, and error rates will not exceed 0.1%."
2. Define the Blast Radius
What is the maximum impact you are willing to accept? Who needs to be notified? What is the abort condition?
"If error rates exceed 5% for more than 60 seconds, we abort by re-launching instances manually."
3. Execute the Experiment
Inject the failure. Common injections:
- Terminate EC2 instances or Kubernetes pods
- Block network traffic between services using iptables or a network policy
- Introduce artificial latency using tc (traffic control) or a service mesh
- Apply IAM deny policies to simulate a cloud provider Control Plane outage
- Fill a disk to 100%
4. Observe and Measure
Watch your dashboards. Did the system behave as hypothesized? Where did it deviate?
5. Document Findings
Write a Game Day report: what you tested, what happened, what surprised you, and what you're fixing.
The Psychological Safety Requirement
Game Days only work if engineers feel safe to surface bad news. If the finding is "our disaster recovery plan doesn't work," that is a success — you found it before a real disaster. If engineers fear blame for surfacing failures, they will hide findings, and the Game Day becomes theater.
Blameless culture is not optional for Black Swan Engineering. It is a prerequisite.
When asked "How do you test your disaster recovery plan?", the weak answer is "we have runbooks and we review them quarterly." The strong answer describes a structured Game Day program: hypothesis-driven experiments, defined blast radius, abort conditions, blameless post-experiment reviews, and a tracking system for findings. Mentioning that you've personally run or participated in a Game Day — and what you discovered — is highly differentiating at the senior level.