A Deep Dive into Raft Leader Election

This post covers leader election in depth — how a leader is elected from a cold start, how it maintains authority through heartbeats, what happens when it fails, and how a new election restores the cluster. We will also walk through a working Spring Boot and Docker implementation to see all of this in action. Log replication and the broader safety guarantees will be covered in later posts.

Part 1 — The Problem Space

What is distributed consensus

Distributed consensus is the problem of getting a group of nodes to agree on a single value — or a sequence of values — even when some of them can fail or become temporarily unreachable. If you have multiple machines running the same service, they each hold a copy of the data. The challenge is keeping them in sync, especially when writes are arriving continuously and machines can go down at any point. This is harder than it looks. Network partitions happen, nodes crash without warning, messages arrive out of order, and nodes rejoin after missing updates. A lot of scenarios have to be handled correctly before you can say your algorithm actually works.

When there is no reliable mechanism for nodes to agree on shared state, you run into some very real problems:

Split Brain: If a leader node goes down and the rest of the cluster elects a new one, but the original leader comes back online still thinking it is in charge. You now have two nodes simultaneously accepting writes. Both produce diverging logs, and there is no way to determine which one is correct. The cluster has effectively split into two inconsistent halves.

Data Loss/Inconsistency: If a follower goes down and misses writes that happened while it was offline, it rejoins with a stale log. If the leader continues sending new writes without reconciling the gap, that node ends up with a permanently different state from the rest of the cluster. The entire point of replication is defeated if nodes can silently diverge.

No single source of truth: Without a leader enforcing the order of operations, different nodes could apply the same commands in different sequences. Clients reading from different nodes would see different results for the same key at the same time.

To solve this, several consensus algorithms have been proposed over the years. One of the most widely used is multi-Paxos, which is mathematically proven correct and is used in systems like Google’s Chubby and Amazon’s DynamoDB. The problem with Paxos is that it is notoriously difficult to understand and even harder to implement correctly. Raft was designed with understandability as a primary goal, while still providing strong safety guarantees for replicated state machines. [We will not be covering Paxos or how it works internally in this article]

What is Raft?

Raft is a distributed consensus algorithm for a replicated log. The core idea is simple: elect a single leader and give it full responsibility for maintaining the log. The leader accepts writes from clients, appends them to its log, and replicates them to all followers. It waits for a majority acknowledgement before marking an entry as committed. It also sends periodic heartbeats to all followers to signal it is still alive. If the leader goes down, which is detected by followers through the absence of heartbeats, a new election starts and a new leader takes over.

Raft breaks the consensus problem into three sub-problems:

1. Leader election: Elect a new leader when the current one is unreachable.
2. Log replication: Propagate writes from the leader to all followers in a consistent, ordered way.
3. Safety: Ensure only a candidate with the most up-to-date log can become leader, so committed entries are never lost, and ensure at most a single leader is chosen at any given time.

Part 2 — Building Blocks

Server Roles

At any point in time, each node in a Raft cluster is in one of three roles: FOLLOWER, CANDIDATE, or LEADER.

Follower: A follower’s job is purely reactive — it responds to requests from the leader or a candidate. It accepts log entries from the leader via AppendEntries RPCs and appends them to its own log. It does not serve client requests directly, even if a client contacts a follower, it is redirected to the current leader. It also responds to heartbeats from the leader, which are just AppendEntries RPCs with an empty log. If the follower’s election timeout elapses without receiving a valid heartbeat, it concludes the leader is gone and starts an election.

Candidate: When a follower’s election timeout fires, it promotes itself to a candidate and kicks off an election. It increments its term, votes for itself, and sends a RequestVote RPC to all other nodes. If it collects votes from a majority, including its own self-vote, it becomes the leader for the current “term”. If no majority is reached before its election timeout fires again, it starts a new election with an incremented term. If it receives a valid AppendEntries RPC from a legitimate leader during the election, it immediately steps back down to follower.

Leader: The leader handles all client requests and is responsible for maintaining the replicated log across the cluster. It propagates writes to followers via AppendEntries RPCs and commits an entry once a majority of nodes have stored it. If a follower’s log falls out of sync with the leader’s, the leader detects the inconsistency and forces the follower to overwrite the conflicting entries with its own log. The leader also sends periodic heartbeats to every follower to prevent them from timing out and triggering unnecessary elections.

State Transition: Every node starts as a follower. A follower becomes a candidate when its election timeout fires without receiving a heartbeat. A candidate becomes a leader on winning a majority vote, or drops back to follower if another node wins the election or if it receives any message carrying a higher term. A leader operates until it crashes or receives a message with a higher term, at which point it immediately steps down to follower.

State transitions of Raft Nodes

The Concept of Term

Term is how Raft measures time, in other words, it is the algorithm’s logical clock. Every new election starts a new term. Terms are monotonically increasing integers, and a node’s term advances whenever it starts an election or receives a message from a node with a higher term. If a leader is elected, it operates for the rest of that term. If no leader is elected, for ex. because of a split vote — the term ends without a leader, and the next election begins in the following term.

Every node persists a currentTerm variable. On every incoming message, whether a RequestVote or an AppendEntries, the receiver compares the term in the message to its own. If the incoming message carries a higher term, the node updates currentTerm to match. If the message carries a lower term, the node rejects it outright — no exceptions. Even a leader will immediately step down to follower if it receives a message from a node with a higher term.

This makes terms the primary defence against stale leaders. If a network partition isolates a leader, the rest of the cluster elects a new one with a higher term. When the old leader reconnects, every message it sends will be rejected because its term is out of date. It will eventually receive a heartbeat from the new leader, see the higher term, and quietly convert itself to a follower.

Election Relevant States & Heartbeat mechanism

Each node maintains the following states at all times:

• currentTerm: The latest term this node has seen. Initialized to 0, increases monotonically.
• votedFor: The candidate this node voted for in the current term.
• log[]: The log entries received from the leader, representing commands to be applied to the state machine.

Heartbeat mechanism: The leader sends periodic heartbeats to all followers. A heartbeat is simply an AppendEntries RPC with an empty log — no new entries, just the leader’s term and leaderId. When a follower receives one, it recognizes the leader is still alive, resets its election timeout, and responds with its currentTerm and a success flag.

Note: The full AppendEntries RPC contains additional fields used for log replication, ex. log index checks, commit index, and more. Those are intentionally left out here since the focus is on leader election. The complete RPC schema and all state variable definitions are in Figure 2 of the original Raft paper.

A condensed summary of the Raft consensus algorithm

Part 3 — Leader Election Process

Now that we have covered how nodes operate and what state they track, let us walk through how an election actually happens.

An election is triggered by exactly one thing: a follower’s election timeout fires without receiving a valid heartbeat (ie empty log AppendEntries rpc) from a leader. Consider a cluster of 5 nodes.

Phase 1 — Becoming a candidate

All nodes cold-start in the follower state with currentTerm = 0. There is no leader yet. Each node independently starts an election timeout timer set to a randomized value. The randomization is critical — if all nodes used the same timeout, they would all fire at the same moment, all vote for themselves, and no one would reach majority. By randomizing, one node almost always fires before the others.

If, say node-1’s election timeout fires first, it promotes itself to candidate, increments currentTerm to 1, votes for itself, and broadcasts a RequestVote RPC to all other nodes.

Phase 2 — Broadcasting VoteRequest rpc

The other nodes (say node-2) may still be waiting on their own election timeout, which has not yet expired. When node-2 receives the RequestVote, it evaluates the request and decides whether to grant the vote.

The RequestVote RPC contains four fields:

• term: the candidate’s current term
• candidateId: who is requesting the vote
• lastLogIndex: the index of the last entry in the candidate’s log
• lastLogTerm: the term of that last log entry

The last two fields are used for a log freshness check.. They enforce the Leader Completeness Property — only a candidate whose log is at least as up-to-date as a majority of the cluster can win. This prevents a node with a stale log from winning and overwriting entries that were already committed.

Phase 3 — Vote granting decision on receivers end

When a node receives a RequestVote RPC, it runs through a set of checks to decide whether to grant the vote:

The log currency check deserves a closer look. A candidate’s log is considered “at least as up-to-date” if:

• Its lastLogTerm is higher than the voter’s — a higher term on the last entry means it has seen more recent leadership, so the voter defers to it.
• Its lastLogTerm equals the voter’s AND its lastLogIndex is greater than or equal to the voter’s — same term, but the candidate’s log is at least as long.

If either condition holds, and the voter has not already voted in this term, the vote is granted. This rule, combined with majority voting, gives Raft its core safety guarantee: any entry committed on a majority is guaranteed to be present on every future leader’s log. Followers may lag, but the leader never will.

Phase 4 — Collection votes & winning

Node-1 collects VoteResponse messages as they arrive. The moment it receives votes from a strict majority of the cluster, including its own self-vote, it immediately becomes the leader for that term. It does not wait for the remaining responses. In Raft, leadership is decided the moment quorum is reached.

At the same time, the candidate is running its own election timeout. If the timeout fires before a majority responds, because the network is slow or because votes were split, it increments currentTerm again and reruns the whole process as a new election in the new term.

Phase 5 — Split vote?

Sometimes two nodes time out at nearly the same time and both become candidates simultaneously. Each collects some votes but neither reaches majority. Both candidates then wait for their election timeout to fire again. Since the delays are randomized, they will usually pick different timeout values in the next round, so one candidate times out first, starts a fresh election earlier, and wins cleanly. This process repeats until a winner emerges. It typically resolves within a few rounds due to randomization in election timeouts.

Phase 6 — After Winning: Establishing Authority

On winning, the new leader does three things immediately:

• Initializes nextIndex[i] for every follower to lastLogIndex + 1, optimistically assuming they are fully caught up.
• Initializes matchIndex[i] to 0 for every follower, conservatively assuming nothing has been confirmed yet.
• Appends a no-op entry to its log in the current term and sends AppendEntries (heartbeat) to all followers. This does two things: it lets the leader quickly discover each follower’s actual log position, and it creates a committed entry in the current term, which is required before the leader can serve linearizable reads or approve membership changes. This detail matters more for log replication, commitment behavior, and certain safety/read guarantees, so we’re only mentioning it briefly here and will cover it properly in a later post.

The moment a follower receives and accepts the new leader’s AppendEntries, it resets its election timeout and settles into follower state for this term.

Raft website provides a clear simulation of the election timeout process.

Why randomized timeouts

This is one of Raft’s most elegant design choices. If every node used the same election timeout, they would all fire at the same moment, all become candidates simultaneously, all vote for themselves, and no one would get a majority. The cluster would loop forever without electing a leader. By randomizing the election timeout (ex. 150–300ms) one node almost always fires before the others, starts the election before they even become candidates, and wins without a contest. It is a simple idea that solves a hard coordination problem without any additional communication.

Leader Failure and Re-election

Elections are always triggered by an election timeout on a follower. When the leader crashes, heartbeats stop. The first follower whose election timeout fires becomes a candidate and starts an election, going through all the same phases described above.

If the old leader comes back online after being replaced, it will find that every other node rejects its messages — its term is lower than the cluster’s current term. It will eventually receive a heartbeat from the new leader carrying the higher term and convert itself to a follower.

Election recap

In one full election cycle, followers wait for heartbeats, one follower times out and becomes a candidate, it increments its term and requests votes, a majority makes it leader, and periodic heartbeats then keep the cluster stable until the leader fails or a higher term appears.

Part 4— Implementation

Note: This implementation is intentionally simplified to focus on leader election mechanics: term changes, vote requests, heartbeats, and re-election after leader failure. It does not fully implement all of Raft’s production guarantees. In particular, the full RequestVote log up-to-date check (based on lastLogIndex and lastLogTerm), durable state persistence, quorum health checks, and log replication details are intentionally left out or simplified for clarity.

Overview and Apis

We will implement leader election in Spring Boot and simulate it using Docker. You may also run it locally without Docker by passing configuration overrides on the command line:

mvn spring-boot:run -Dspring-boot.run.arguments="--raft.node-id=node-1 --server.port=8081"

The goal is to spin up three nodes, let them elect a leader, then kill the leader container and observe the re-election.

Three REST endpoints drive the whole simulation:

• POST /heartbeat — used by the leader to send periodic heartbeats to followers
• POST /vote-request — used by a candidate to request votes from peer nodes
• GET /state — exposes the node’s current term, role, leader, and vote-related state for observation

Project Structure

.
├── Dockerfile
├── docker-compose.yml
├── pom.xml
├── RaftApplication.java
├── client
│   └── RaftClient.java
├── config
│   └── RaftConfig.java
├── controller
│   └── RaftController.java
├── core
│   ├── ElectionService.java
│   ├── NodeRole.java
│   ├── Peer.java
│   └── RaftState.java
├── dto
│   ├── HeartbeatRequest.java
│   ├── HeartbeatResponse.java
│   ├── StateResponse.java
│   ├── VoteRequest.java
│   └── VoteResponse.java
└── resources
    ├── application-docker.properties
    └── application.properties

There are two properties files. application.properties holds the base configuration for local development. application-docker.properties overrides peer addresses with Docker container hostnames instead of localhost ports, it activates when the app runs with the docker Spring profile inside a container.

Configuration & Domain

RaftConfig — reads node-id, the peer list, heartbeat interval, and election timeout range. This is the only place environment configuration is read and everything else injects from here.

Peer — a simple holder for a peer’s nodeId and the URL used to reach it.

NodeRole — an enum with three values enforced for a node: FOLLOWER, CANDIDATE, LEADER.

RaftState — holds the full in-memory state of the node: its id, current role, current term, who it voted for, and the current leader. Exposes state transition methods: becomeLeader(), becomeCandidate(), voteFor(), stepDown().

@Configuration
public class RaftConfig {
 public static final long HEARTBEAT_MS = 3000;
 private String nodeId;
 private Map<String, String> nodes = new LinkedHashMap<>();
 private List<Peer> peers;

 public long getNextElectionTimeout() {
  return 4000 + ThreadLocalRandom.current().nextInt(2000);
 }
}

public record Peer(String peerId, String url) {}

public enum NodeRole {
    LEADER,
    FOLLOWER,
    CANDIDATE
}

public class RaftState {
 private final String nodeId;
 private NodeRole nodeRole;
 private int currentTerm;
 private String votedFor;
 private String currentLeaderId;
 private long lastHeartbeatTime;

 public RaftState(String nodeId) {
  this.nodeId = nodeId;
  this.nodeRole = NodeRole.FOLLOWER;
  this.currentTerm = 0;
  this.lastHeartbeatTime = System.currentTimeMillis();
 }

 public void stepDown(int term) {
  this.nodeRole = NodeRole.FOLLOWER;
  this.currentTerm = term;
  this.votedFor = null;
  this.currentLeaderId = null;
 }

 public void becomeCandidate() {
  this.nodeRole = NodeRole.CANDIDATE;
  this.currentTerm++;
  this.currentLeaderId = null;
  this.votedFor = nodeId;
 }

 public void becomeLeader() {
  this.nodeRole = NodeRole.LEADER;
  this.currentLeaderId = nodeId;
 }

 public void recordHeartBeat(String leaderId, int term) {
  this.currentLeaderId = leaderId;
  this.currentTerm = term;
  this.lastHeartbeatTime = System.currentTimeMillis();
  this.nodeRole = NodeRole.FOLLOWER;
 }

 public void voteFor(int term, String candidateId) {
  this.votedFor = candidateId;
  this.currentTerm = term;
 }
}

DTOs

HeartbeatRequest / HeartbeatResponse — carries currentTerm and leaderId from leader to followers. Followers respond with a success flag.

VoteRequest / VoteResponse — in this demo, VoteRequest carries only the candidate’s term and candidateId, and VoteResponse carries the responder’s term plus whether the vote was granted. In full Raft, RequestVote also includes lastLogIndex and lastLogTerm so that voters can reject candidates with stale logs.

in this demo, VoteRequest carries only the candidate’s term and candidateId, and VoteResponse carries the responder’s term plus whether the vote was granted. In full Raft, RequestVote also includes lastLogIndex and lastLogTerm so that voters can reject candidates with stale logs.

StateResponse — the response shape for GET /state. Includes current term, role, leader id, voted-for, and heartbeat timing info.

public record HeartbeatRequest(int term, String leaderId) {}
public record HeartbeatResponse(int term, boolean success) {}

public record VoteRequest(int term, String candidateId) {}
public record VoteResponse(int term, boolean voteGranted) {}

public record StateResponse(String nodeId, int currentTerm, String role, 
                            String currentLeaderId, String votedFor, 
                            long lastHeartbeatTime,
                            long timeSinceLastHeartbeatMs) {}

HTTP & Core Logic

RaftClient — a thin REST client that makes outbound calls to peer nodes.

RaftController — handles the three endpoints and delegates all state transition logic to ElectionService.

ElectionService — the core of the implementation. Manages the election timeout, runs elections, handles incoming votes, and drives heartbeats when the node is leader. This is covered in detail in the next part.

@Service
public class ElectionService {

    private static final Logger log = LoggerFactory.getLogger(ElectionService.class);
    private final RaftConfig raftConfig;
    private final RaftState raftState;
    private final ScheduledExecutorService heartbeatScheduler;
    private final ScheduledExecutorService electionTimeoutScheduler;
    private ScheduledFuture<?> electionFuture;
    private final RestClient restClient;
}

Core logic of ElectionService

Elections after a randomized timeout

The election timeout is managed by a ScheduledExecutorService. On startup, a timeout task is scheduled with a random delay between 4 and 6 seconds. Each time a heartbeat arrives from the leader, the existing scheduled task is cancelled and a new one is scheduled with a fresh random delay. If the task fires — meaning no heartbeat arrived within the window — startElection() is called.

The 4–6 second range is intentionally longer than the real-world recommendation of 150–300ms. In a local Docker environment, a realistic value would scroll logs too fast to observe anything useful.

When startElection() runs, the node becomes a candidate, increments its term, records its own self-vote, and sends a RequestVote to all peers. If it receives votes from a majority — (totalNodes / 2) + 1, counting itself — it becomes the leader.

On the receiving end, the vote request handler applies the same decision rules from Part 3, Phase 3: term check, already-voted-this-term check, and log currency check. If all conditions pass, it grants the vote, records it in state, and returns true.

@PostConstruct
public void init() {
    resetElectionTimer();
    this.heartbeatScheduler.scheduleAtFixedRate(this::sendHeartBeat, HEARTBEAT_MS, HEARTBEAT_MS, TimeUnit.MILLISECONDS);
}

private void resetElectionTimer() {
    if (electionFuture != null) {
        electionFuture.cancel(false);
    }

    long timeoutMs = raftConfig.getNextElectionTimeout();
    electionFuture = electionTimeoutScheduler.schedule(this::onElectionTimeout, timeoutMs, TimeUnit.MILLISECONDS);
}

private void onElectionTimeout() {
    if (LEADER.equals(raftState.getNodeRole())) {
        return;
    }

    startElection();
    if (!LEADER.equals(raftState.getNodeRole())) {
        resetElectionTimer();
    }
}

private void startElection() {
    raftState.becomeCandidate();
    log.info("[{}] Starting election for term {}", raftState.getNodeId(), raftState.getCurrentTerm());
    int votesReceived = 1; // vote for self

    for (Peer peer: raftConfig.getPeers()) {
        try {
            VoteResponse voteResponse = RaftClient.requestVote(peer, restClient, raftState.getCurrentTerm(), raftConfig.getNodeId());
            if (voteResponse.voteGranted() && voteResponse.term() == raftState.getCurrentTerm()) {
                votesReceived++;
                log.info("[{}] Received vote from {} for term {}", raftState.getNodeId(), peer.peerId(), raftState.getCurrentTerm());
            } else if (voteResponse.term() > raftState.getCurrentTerm()) {
                log.warn("[{}] Discovered higher term {} from {}. Stepping down to follower.", raftState.getNodeId(), voteResponse.term(), peer.peerId());
                stepDown(voteResponse.term());
                return;
            }
        } catch (Exception e) {
            log.error("[{}] Failed to request vote from {}: {}", raftState.getNodeId(), peer.peerId(), e.getMessage());
        }
    }

    int totalNodes = raftConfig.getPeers().size() + 1;
    int majority = (totalNodes / 2) + 1;
    if (votesReceived >= majority) {
        log.info("[{}] Won election for term {} with {}/{} votes", raftState.getNodeId(), raftState.getCurrentTerm(), votesReceived, totalNodes);
        raftState.becomeLeader();
        sendHeartBeat();
    } else {
        log.info("[{}] Lost election for term {} with {}/{} votes", raftState.getNodeId(), raftState.getCurrentTerm(), votesReceived, totalNodes);
    }
}

// Receivers End
public VoteResponse handleVoteRequest(int term, String candidateId) {
    if (term < raftState.getCurrentTerm()) {
        log.info("[{}] Rejected stale vote request from {} (term {})", raftState.getNodeId(), candidateId, term);
        return new VoteResponse(raftState.getCurrentTerm(), false);
    }

    if (term > raftState.getCurrentTerm()) {
        log.info("[{}] Received higher term {} from {}. Stepping down to follower.", raftState.getNodeId(), term, candidateId);
        stepDown(term);
    }

    if (raftState.getVotedFor() != null && !raftState.getVotedFor().equals(candidateId)) {
        log.info("[{}] Already voted for {} in term {}. Rejecting {}", raftState.getNodeId(), raftState.getVotedFor(), term, candidateId);
        return new VoteResponse(term, false);
    }

    log.info("[{}] Granting vote to {} for term {}", raftState.getNodeId(), candidateId, term);
    raftState.voteFor(term, candidateId);
    resetElectionTimer();
    return new VoteResponse(term, true);
}

Periodic heartbeats to followers

Once a node becomes leader, a separate scheduled task fires every 3 seconds and calls sendHeartbeat() on all peers. As with the election timeout, 3 seconds is deliberately slow for demo readability, in production this would be closer to 100ms, well under the election timeout range, ie, heartbeat_timer << election_timeout so that unnecessary election won’t be triggered.

Failed heartbeats to individual peers are silently ignored. A production Raft implementation like etcd tracks quorum — if a leader loses contact with enough nodes that it can no longer reach a majority, it steps down. That check is out of scope for this demo.

private void sendHeartBeat() {
    if (!LEADER.equals(raftState.getNodeRole())) {
        return;
    }

    for (Peer peer: raftConfig.getPeers()) {
        int currentTerm = raftState.getCurrentTerm();
        try {
            HeartbeatResponse heartbeatResponse = RaftClient.sendHeartBeat(peer, restClient, currentTerm, raftState.getNodeId());
            log.info("[{}] Heartbeat response from {}: success={}, term={}", raftState.getNodeId(), peer.peerId(), heartbeatResponse.success(), heartbeatResponse.term());

            if (heartbeatResponse.term() > currentTerm) {
                log.warn("[{}] Discovered higher term {} from {}. Stepping down to follower.", raftState.getNodeId(), heartbeatResponse.term(), peer.peerId());
                stepDown(heartbeatResponse.term());
                return;
            }
        } catch (Exception e) {
            log.error("[{}] Failed to send heartbeat to {}: {}", raftState.getNodeId(), peer.peerId(), e.getMessage());
        }
    }
}

public HeartbeatResponse handleHeartbeat(int term, String leaderId) {
    if (term < raftState.getCurrentTerm()) {
        /*
         * If the term is stale, reject the heartbeat and do NOT reset the election timer.
         * This prevents a follower with an outdated term from disrupting the current leader's authority
         * by sending heartbeats that would reset followers' election timers.
         * This is a critical part of the Raft protocol to ensure stability and prevent split-brain scenarios.
         */
         log.info("[{}] Rejected stale heartbeat from leader {} (term {})", raftState.getNodeId(), leaderId, term);
        return new HeartbeatResponse(raftState.getCurrentTerm(), false);
    }

    if (term > raftState.getCurrentTerm() || !NodeRole.FOLLOWER.equals(raftState.getNodeRole())) {
        raftState.stepDown(term);
    }

    raftState.recordHeartBeat(leaderId, term);
    resetElectionTimer();
    return new HeartbeatResponse(term, true);
}

Part 5— Demo

Now let’s see how our code actually works with new leader election & a leader crash.

Starting a docker container with 3 containers.

Node-1 becomes a leader after receiving votes from node-x and node-x. Logs for heartbeat responses are visible in node-1 container.

Node-1 won the election with 2/3 votes. Node-2 & Node-3 are followers

/state api provides an overview of the state of the nodes.

Leader:

{
   "currentLeaderId" : "node-1",
   "currentTerm" : 2,
   "lastHeartbeatTime" : 1775054375864,
   "nodeId" : "node-1",
   "role" : "LEADER",
   "timeSinceLastHeartbeatMs" : 205395,
   "votedFor" : "node-1"
}

Follower:

{
   "currentLeaderId" : "node-1",
   "currentTerm" : 2,
   "lastHeartbeatTime" : 1775054597869,
   "nodeId" : "node-2",
   "role" : "FOLLOWER",
   "timeSinceLastHeartbeatMs" : 1240,
   "votedFor" : "node-1"
}

Now let’s crash the leader, ie, node-1 (using docker kill).

After an election timeout of 4+ seconds, node-3 start an election first, followed by node-2. Since the candidate now needs 2 votes to reach a quorum, they go back and forth for terms 3 & 4 with no leader elected. Node-2 eventually wins the election in term 5 and becomes the new leader.

Node-2 wins the election after a 2 terms with no leader

This is the dockerfile and docker-compose file.

DockerFile

# Build
FROM maven:3.9.4-eclipse-temurin-17 AS build
WORKDIR /app
COPY . .
RUN mvn clean package -DskipTests

#Runtime
FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
ENTRYPOINT ["java", "-jar", "app.jar"]

docker-compose.yml

services:
  raft-leader-election-1:
    image: raft-leader-election:latest
    container_name: raft-leader-election-node-1
    ports:
      - "8081:8080"
    environment:
      SPRING_PROFILES_ACTIVE: "docker"
      RAFT_NODEID: "node-1"
    networks:
      - raft-network
  raft-leader-election-2:
    image: raft-leader-election:latest
    container_name: raft-leader-election-node-2
    ports:
      - "8082:8080"
    environment:
      SPRING_PROFILES_ACTIVE: "docker"
      RAFT_NODEID: "node-2"
    networks:
      - raft-network
  raft-leader-election-3:
    image: raft-leader-election:latest
    container_name: raft-leader-election-node-3
    ports:
      - "8083:8080"
    environment:
      SPRING_PROFILES_ACTIVE: "docker"
      RAFT_NODEID: "node-3"
    networks:
      - raft-network

networks:
  raft-network:
    driver: bridge

Code used in demo is available on github.

The code is only for demo/simulation purposes and not a production ready code. Concurrency/parallelism, log persistence, log index checks during voting, quorum checks, network timeouts/retries handling etc. is deliberately left out to keep it simple.

References:

• Raft Paper — https://raft.github.io/raft.pdf
• https://raft.github.io/
• MicroRaft (Java implementation)
• Martin Kleppmann — YT lecture