Once you have a hypothesis, you test it by injecting a specific type of failure. In chaos tools, these are called "experiments" or "attacks." The type of fault you inject should directly correspond to the failure mode described in your hypothesis.
1. Resource Attacks (Starvation)
These simulate "noisy neighbor" problems or runaway processes — situations where a process consumes more than its fair share of system resources.
- CPU Burn: Consume 100% of a container's CPU allocation. Does the horizontal pod autoscaler (HPA) detect the spike and scale out fast enough? Does the application degrade gracefully or crash?
- Memory Fill: Gradually consume available RAM. Does the OOM (Out of Memory) killer terminate the right process? Does the application restart cleanly?
- Disk Fill: Fill the filesystem to 100%. Does the application log a clear error, or does it silently corrupt data?
2. State Attacks (The Hammer)
These simulate hardware failure, process crashes, or infrastructure-level events.
- Pod/Process Kill: Randomly terminate a container or process. Does Kubernetes restart it within the expected time? Does the application reconnect to the database automatically, or does it hold stale connections?
- Node Drain: Evict all pods from a Kubernetes node, simulating a node failure. Does the cluster reschedule workloads correctly? Are there any pods with no replicas that become unavailable?
3. Network Attacks (The Slowdown)
These are often the most revealing attacks because real-world failures are rarely binary (up/down). Networks degrade gradually, and most applications are not designed to handle partial failures gracefully.
- Latency Injection: Add 500ms of artificial delay to all network calls to the database. Does the application handle timeouts correctly, or do threads pile up waiting for responses and eventually exhaust the thread pool, crashing the entire service?
- Packet Loss: Drop 5% of network packets. How does the TCP retry logic handle this? Does the application surface meaningful errors to users, or does it silently hang?
- Network Partition: Completely block traffic between two services. Does the circuit breaker open and return a fallback response, or does the calling service cascade-fail?
Key Point: Network attacks expose a class of bugs called cascading failures — where one slow dependency causes a thread pool to fill up, which causes the calling service to slow down, which causes its callers to slow down, until the entire system is degraded. This is one of the most common causes of large-scale outages, and it is almost never caught in unit or integration tests.