Deep Dive into Kafka KRaft Mode: ZooKeeper vs KRaft, Benefits & Real-World Use Case

*New to Kafka? Start with my guide on Kafka Basic Terminologies.*

What is KRaft (Kafka Raft Metadata mode)?

• KRaft is Kafka’s internal consensus protocol introduced to eliminate the dependency on Apache ZooKeeper for metadata management.
• It consolidates the responsibility for metadata into Kafka itself, simplifying its architecture and operations.

Differences between ZooKeeper-based and KRaft mode

• ZooKeeper-based: Utilizes an external service (ZooKeeper) for managing metadata, leading to complexity in deployment and management.
• KRaft mode: Kafka manages its own metadata internally, reducing the need for external dependencies and simplifying the architecture.

Zookeeper vs KRaft

Benefits of removing ZooKeeper

• Simplified architecture: Eliminates the need for a separate ZooKeeper service.
• Improved scalability: Allows Kafka to scale more efficiently without the limitations imposed by ZooKeeper.
• Faster failover: Enhances the speed of leader election and metadata propagation.

How Raft consensus works (in brief)

• Leader election: A leader is elected among the controller nodes using the Raft consensus algorithm.
• Log replication: The leader appends metadata changes to its log and replicates them to follower controllers.
• Commitment: Once a majority of followers acknowledge the log entries, they are considered committed.

Controller quorum vs broker quorum

Controller Quorum:

• A set of controller nodes responsible for managing metadata (topics, partitions, configurations) and ensuring consistency.
• In KRaft mode, these controllers use the Raft protocol to replicate metadata logs.

Broker Quorum:

• A set of broker nodes that store and serve actual data (messages).
• In KRaft mode, brokers also participate in the metadata quorum because the controllers are co-located with brokers.

In KRaft mode, the metadata quorum is essentially the controller quorum.

• The term metadata quorum refers to the Raft group that replicates metadata.
• This group is formed by controller nodes, so controller quorum = metadata quorum.

Real-world example:

In an e-commerce platform, KRaft mode ensures that product inventory updates are consistently propagated across all services without relying on an external system like ZooKeeper.

Company use case:

LinkedIn transitioned to KRaft mode to streamline its Kafka architecture, reducing operational complexity and improving scalability.

Kafka: Zookeeper vs KRaft

Want to understand the algorithm behind KRaft? Read my blog on *Raft Consensus Algorithm Explained.*

Glossary

• Raft: Internal protocol Kafka uses to replicate metadata
• Leader: The controller that writes metadata
• Followers: Backup controllers that replicate metadata
• Quorum: Majority of nodes that must agree for a change
• Failover: Automatic switch to a new leader when the current fails