Kafka Raft Consensus Algorithm Explained: A Simple Guide to Leader Election and Log Replication

In distributed systems, consistency and fault tolerance are key. Apache Kafka, in its KRaft mode, replaces ZooKeeper with the Raft consensus algorithm to manage metadata reliably across controller nodes.

For a complete understanding, start with Kafka Basic Terminologies and then Deep Dive into Kafka KRaft Mode.

The Raft consensus algorithm is a protocol used to manage a replicated log of changes across multiple servers (nodes) so that all nodes agree on the same data, even if some fail.

Why Kafka Uses Raft in KRaft Mode

Kafka’s metadata (like topic creation, partition assignments, and configuration updates) needs to be consistent across all controller nodes. Raft helps Kafka achieve this by:

• Electing a leader controller
• Replicating metadata changes
• Ensuring consistency across controller nodes

Think of Raft as Kafka’s internal “voting system” that keeps all controllers synchronized — even during failures.

Goals of Raft in Kafka

• Keep all controller nodes in sync
• Ensure only one leader makes metadata changes
• Enable safe failover when a controller crash

How Raft Works in Kafka (Simplified)

1. Leader Election

• One node is elected as the leader.
• Others become followers.
• If the leader fails, a new one is elected automatically.

Leader Election Steps FlowChart

In Kafka, this ensures that only one controller writes metadata, avoiding conflicts.

2. Log Replication

• The leader logs every metadata change (e.g., “create topic X”).
• These changes are replicated to followers.
• Once a majority (quorum) acknowledges, the change is committed.

This guarantees that metadata like topic info and partition assignments are safely stored and replicated.

3. Safety & Consistency

• Followers only apply committed changes.
• If a node fails and recovers, it catches up from the leader.

This ensures Kafka’s metadata remains reliable and fault-tolerant, even during outages.

Real-World Analogy: Project Managers and a Shared Logbook

Imagine a team of project managers:

• One is chosen as the lead.
• Every decision (e.g., “launch feature X”) is written in a shared logbook.
• A decision is final only when most managers agree.
• If the lead is absent, a new one is elected, and work continues.

That’s exactly how Raft works — simple, democratic, and resilient.

Kafka + Raft in Action: LinkedIn’s Use Case

LinkedIn runs Kafka in KRaft mode:

• Raft ensures high availability of metadata (like user feed topics).
• If a controller fails, another takes over automatically.
• No ZooKeeper needed — operations are simplified.

Conclusion

Raft brings clarity, safety, and resilience to Kafka’s metadata management. By understanding its core steps — leader election, log replication, and consistency guarantee — you’ll appreciate how Kafka stays robust in real-world deployments.

Whether you’re a beginner or an architect, Raft in Kafka is a concept worth mastering.