The "Split Brain" Risk
In a small or poorly configured cluster, a single node can wear two hats: acting as the Master (managing cluster state — tracking which shards live on which nodes, which indexes exist, and which nodes are alive) and as a Data Node (storing actual documents and processing search queries).
This dual role creates a dangerous failure mode. When your data ingestion spikes, CPU hits 100%. The node becomes unresponsive. If that node was also the Master, the entire cluster enters a "Red" state — it cannot determine which nodes are alive, cannot route queries, and cannot accept writes. Your search engine is effectively down.
Even worse is the Split Brain scenario: if you have two master-eligible nodes and the network between them is interrupted, each node believes the other is dead and promotes itself to Master. Now you have two clusters, each accepting writes, diverging silently. When the network heals, you have a data conflict with no clean resolution.
The Solution: Dedicated Master Nodes
For any production workload, you must use Dedicated Master Nodes — nodes whose sole job is cluster management.
| Property | Detail |
|---|---|
| Role | Cluster state management only |
| What they do NOT do | Store data, process search queries, or index documents |
| Recommended size | Small instances (e.g., c5.large.search) — they do no heavy lifting |
| Required count | Always 3 (or 5 for very large clusters). Never 1 (single point of failure). Never 2 (Split Brain risk). |
Why exactly 3? The cluster uses a quorum-based election system. A quorum is a simple majority: (N/2) + 1. With 3 master nodes, the quorum is 2. If one node fails, the remaining two can hold an election and reach agreement (2 out of 3 is a majority). With only 2 master nodes, if one fails, the survivor (1 out of 2) does not have a majority and the cluster locks itself to prevent data corruption — which is the correct behavior, but it means your cluster is down.
Sizing note: Dedicated master nodes manage metadata, not data. A c5.large.search (2 vCPU, 4 GB RAM) is sufficient for clusters with up to 30 data nodes. For clusters with 30–75 data nodes, step up to c5.xlarge.search. The key metric to watch is JVM heap pressure on master nodes — if it exceeds 85%, the master is struggling with cluster state size, usually caused by too many shards.
Interviewers frequently ask: "What is Split Brain and how does OpenSearch prevent it?" The answer has two parts: (1) Split Brain occurs when a network partition causes two nodes to simultaneously believe they are the elected master, leading to divergent cluster states. (2) OpenSearch prevents it by requiring a quorum of (N/2) + 1 master-eligible nodes to agree before any node can be elected. With 3 dedicated master nodes, one failure is tolerated. With 2, no failure is tolerated — making 2 worse than 1 in some failure scenarios because it creates false confidence.