Getting Everyone on the Same Page
Imagine a group of five friends trying to decide where to go for dinner via text message. Messages get delayed, some friends' phones die, and two people might suggest different restaurants at the exact same time. How does the group manage to agree on a single, final choice? Now scale that problem to thousands of servers spread across multiple data centers, where any server can crash at any moment and network messages can be delayed, duplicated, or lost entirely. That is the consensus problem.
Consensus is the challenge of getting multiple computers (nodes) to agree on a single value or an ordered sequence of values, even when some nodes fail or messages are lost. The value being agreed upon could be anything critical to the system's operation:
- Who is the current "leader" of the cluster?
- What is the next transaction to commit to the database?
- Is a specific node currently online or offline?
- What is the correct configuration for the entire cluster?
In a perfect world with reliable networks and computers that never crash, this would be trivial — just broadcast the value and everyone accepts it. In the real world, networks partition, servers reboot unexpectedly, and clocks drift. Consensus algorithms are the rigorously proven protocols that allow a system to reach agreement despite these failures.
Without consensus, you cannot build a reliable distributed database, a fault-tolerant coordination service, or any system that requires multiple nodes to share a consistent view of the world. Consensus is the foundational primitive on which distributed systems are built — everything else depends on it working correctly.
Interviewers often ask: "Why can't you just have all nodes write to a shared database and use locks?" The answer is that a single shared database is itself a single point of failure. Consensus algorithms solve the problem of making the coordination layer itself fault-tolerant. There is no "shared truth" to appeal to — the algorithm creates the shared truth from scratch.