The Missing Bullet Holes
During World War II, the U.S. military examined bombers returning from missions. They mapped where the planes had been hit by enemy fire: wings, fuselage, tail. The logical conclusion seemed clear — reinforce those areas. Mathematician Abraham Wald at the Statistical Research Group pointed out the critical flaw: they were only looking at planes that survived. The planes hit in the engines and cockpit never made it back. The bullet hole map showed where planes could take damage and survive, not where they were most vulnerable. Wald recommended reinforcing the areas with no bullet holes — the engines and cockpit.
This is survivorship bias: drawing conclusions from an incomplete dataset because the failures are invisible. You only see the survivors, so you over-attribute their characteristics to success.
In Technology Architecture
"Netflix uses microservices. Google uses microservices. Amazon uses microservices. Therefore, microservices are the path to success."
What you don't see:
- The thousands of startups that adopted microservices prematurely, drowned in operational complexity, burned through their runway, and shut down. They didn't write blog posts. They didn't give conference talks. They're invisible.
- Netflix, Google, and Amazon adopted microservices after achieving massive scale with monoliths. Their microservices architecture may be a result of their success (they had the engineering resources to manage the complexity), not the cause of it.
More Engineering Examples
- "Dropouts succeed — look at Bill Gates and Mark Zuckerberg!" You don't see the millions of dropouts who didn't become billionaires. Gates and Zuckerberg are the planes that came back.
- "We should use [Hot New Framework] — look at all these successful companies using it!" You're reading case studies from companies that succeeded. Companies that adopted the framework and failed didn't publish case studies.
- "Our deploy process works fine — we've never had a major outage." Maybe. Or maybe you've been lucky, and the deploys that would have caused outages happened to coincide with low traffic. You're surviving, not succeeding.
- Stack Overflow answers: You find a solution on Stack Overflow with 200 upvotes. You don't see the hundreds of people who tried that solution, had it fail silently in their specific context, and moved on without commenting.
The Fix — Process Patches
- Seek the Failures: When evaluating a technology, search for "[technology] failure," "[technology] problems at scale," and "[technology] migration away from" with the same energy you search for success stories. Read the post-mortems, not just the launch announcements.
- Base Rate Awareness: Before concluding "X leads to success," ask: "What percentage of teams that tried X actually succeeded?" If 10,000 teams tried microservices and 50 succeeded spectacularly, the success rate is 0.5% — not a ringing endorsement. The visible 50 are not representative of the invisible 9,950.
- Context Matching: When studying a successful company's architecture, ask: "Do we share their constraints?" Netflix has thousands of engineers, a dedicated platform team, and custom tooling built over years. If you have 12 engineers, their architecture is not evidence that it will work for you — it's evidence that it worked for them, in their context, with their resources.
Key Point: Survivorship bias is particularly insidious in technology because the industry has a strong publication bias toward success. Conference talks, blog posts, and case studies overwhelmingly feature things that worked. The failures rot silently. Senior engineers develop a habit of asking "What am I not seeing?" and "Who tried this and failed?" before making architectural decisions based on visible success stories. The absence of failure stories is not evidence of absence of failures.