The Old Way: ZooKeeper Dependency
Historically, Kafka required a separate cluster called ZooKeeper to answer two critical questions at all times:
- Who is the leader? For each partition, one broker is the leader (handles all reads and writes). ZooKeeper tracked which broker held each leadership role.
- Which brokers are alive? Brokers registered themselves with ZooKeeper. If a broker's heartbeat stopped, ZooKeeper triggered a leader election for that broker's partitions.
This created real problems at scale:
- Partition Limit: ZooKeeper stored metadata for every partition in memory. At roughly 200,000 partitions, ZooKeeper's write latency degraded noticeably. Large organizations hit this ceiling and had to split workloads across multiple Kafka clusters.
- Operational Overhead: ZooKeeper is a separate distributed system with its own quorum requirements (you need at least 3 nodes for fault tolerance), its own security model, and its own failure modes.
- Slow Controller Failover: When the Kafka controller (the broker responsible for managing partition leadership) failed, it had to reload all partition metadata from ZooKeeper — a process that could take 30+ seconds on large clusters.
The New Way: KRaft (Kafka Raft)
KRaft (pronounced "kraft") is Kafka's built-in metadata management system, introduced in Kafka 2.8 and production-ready in Kafka 3.3. New MSK clusters can be created in KRaft mode.
How it works: Instead of an external ZooKeeper cluster, a subset of Kafka brokers are designated as controllers. These controllers form a Raft consensus group and maintain the cluster metadata log — a special internal Kafka topic that records every metadata change (broker joins, partition reassignments, leader elections). The active controller is the Raft leader. If it fails, the remaining controllers elect a new leader in milliseconds using the Raft protocol, without any external coordination.
Key benefits:
- Scalability: Because metadata is stored in a Kafka log (not ZooKeeper's in-memory tree), MSK KRaft clusters can support millions of partitions per cluster. The metadata log is replicated and compacted efficiently.
- Faster Failover: Controller failover drops from tens of seconds to under a second, because the new controller already has the full metadata log replicated locally.
- Simplified Operations: No separate ZooKeeper nodes to provision, monitor, or patch. The cluster is a single system.
- Unified Security: You configure authentication and authorization once for Kafka, not separately for Kafka and ZooKeeper.
The migration constraint: You cannot upgrade an existing ZooKeeper-based MSK cluster to KRaft in-place. The metadata formats are incompatible. You must provision a new KRaft cluster and migrate your data using MirrorMaker 2.0. This is a planned migration, not a rolling upgrade.