The Human Component
Think about a fire drill. The building's sprinkler system might be perfectly functional, but if nobody knows where the exits are, people still get hurt. Resilience testing for software systems works the same way — you can have redundant servers, automated failover, and perfect monitoring, but if the humans in the loop don't know what to do, the system still fails.
We spend thousands of hours testing our software (unit tests, integration tests, end-to-end tests). We spend millions of dollars testing our infrastructure (redundancy, chaos engineering, load testing). But we rarely test the most fragile part of the system: the humans and the processes they follow.
Resilience testing in this context is about answering questions like:
- "Does the new on-call engineer know how to find the runbook at 3 AM?"
- "What happens if the only person who knows the database password is on a plane?"
- "Does the CEO know who to call if we receive a ransomware demand?"
- "Does Legal know they need to be notified within 72 hours of a data breach under GDPR?"
We test these human processes using two primary methods: GameDays and Tabletop Exercises (TTX). Both are forms of rehearsal — the same reason surgeons practice on simulators and pilots train in flight simulators before facing real emergencies.
Key Point: The term "wetware" is borrowed from computer science slang — hardware is the physical machine, software is the code, and wetware is the human brain. Wetware has unique failure modes: panic, fatigue, knowledge silos, and communication breakdowns. These failure modes require their own testing discipline.