Establishing Order in Chaos
A critical production outage, a project that has visibly gone off the rails, a security breach — these situations share a common dynamic: everyone is scared, information is incomplete, and multiple people are trying to act simultaneously, often in conflicting directions. Panic is contagious. So is calm. Your first job as a leader in a crisis is to be the calmest person in the room.
Calm does not mean passive. It means you are thinking clearly while others are reacting emotionally. That clarity is what allows you to impose structure on chaos.
The Incident Command Framework:
This framework was adapted from emergency services (firefighting, disaster response) and applied to software operations. It works because it solves the two biggest failure modes in a crisis: too many decision-makers and too many communication channels.
1. Designate a Single Incident Commander (IC)
One person makes decisions. Everyone else provides information and executes. This eliminates the "too many cooks" problem where three senior engineers are simultaneously trying three different fixes and making the situation worse. The IC does not have to be the most technically expert person — they need to be the best at maintaining situational awareness and making decisions under uncertainty.
2. Separate "Fixing" from "Communicating"
This is the most commonly violated rule in incident response. Engineers who are deep in debugging get interrupted every five minutes by a Slack message from a VP asking "What's the status?" Each interruption costs them 10–15 minutes of context reconstruction. The fix is to assign a dedicated Communications Lead whose only job is to update stakeholders on a fixed cadence (every 30 minutes is standard). Engineers go heads-down. The Comms Lead handles all external communication. These two roles must not be the same person.
3. Mitigate First, Diagnose Second
The instinct in a crisis is to understand why something broke before fixing it. Resist this. Your first priority is to stop the bleeding — restore service, roll back the deployment, fail over to the backup region. Root cause analysis (RCA) happens after the incident is resolved, in a calm post-mortem, not during the crisis when everyone is under pressure and making mistakes.
4. Maintain a Running Incident Log
Designate someone to keep a timestamped log of every action taken and every hypothesis tested. This serves two purposes: it prevents duplicate work ("Did we already try restarting the cache?") and it provides the raw material for the post-mortem.
The Post-Mortem:
After the incident is resolved, conduct a blameless post-mortem within 48 hours while memory is fresh. The goal is not to find who made a mistake — it is to find what system or process allowed the mistake to have the impact it did. Every post-mortem should produce at least one concrete action item that makes the system more resilient.
Key Point: Effective crisis leadership requires structural clarity: one decision-maker, separated communication and engineering tracks, and a bias toward mitigation over diagnosis. The post-mortem — not the crisis itself — is where you improve the system.