Read-After-Write Consistency Challenge in Read Replicas

During a KV store mock interview on system design, I encountered a question that initially seemed simple but quickly shifted focus toward centralized replication challenges.

The interviewer asked about handling replication in a leader-follower setup, which led to a discussion of read consistency under replication lag — a classic problem in distributed systems.

Through this discussion, I gained a deep understanding of a very typical issue in leader-based distributed architectures: how to ensure fresh reads from follower nodes while still distributed read traffic effectively.

The Problem is in a Leader-Follower (Master-Slave) KV store replication scenario:

The leader handles writes (and sometimes reads), and followers handle read traffic to reduce the leader's load. Due to asynchronous replication, followers may lag → reads on followers can return stale data. This problem is commonly referred to as stale read / read-your-writes violation.

Distributed System Strategies for Handling Stale Read

Synchronous / Semi-Synchronous Replication

Synchronous replications: write must be acknowledged by all or a specific number of followers → ensures any follower reads are up-to-date. The pros of this strategy include strong consistency guarantees, but it also has cons, such as higher write latency and lower throughput.

Synchronous Replication

Semi-synchronous replication (quorum writes) requires W out of N nodes to acknowledge a write; reads can be served from any node, but using a read quorum ensures consistency, and it is often used in Raft- or Paxos-based systems.

Semi-Synchronous Replication

Leader-Aware Read Routing

Leader-aware read routing is for strong-consistency reads, like routing directly to the leader. For eventually consistent reads, the reading request gonna be routed to the followers. This strategy’s optimization is that it can dynamically measure follower replication lag and only route reads to followers with lag below a threshold.

strong consistency read

eventual consistency read

Lag / Version-Based Reads

Lag / Version-Based Reads

• Each follower maintains its latest applied log offset / LSN (Log Sequence Number)
• The client or load balancer routes reads to followers where offsets ≥ last committed write
• This ensures reads from followers are fresh, while still distributing the load

Read-After-Write (Session Stickiness)

For the same session or user: Immediately after a write → route reads to the leader, and after replication → allow reads from followers. This is a common strategy in web systems to ensure session consistency.

Asynchronous Cache + Versioning

In this strategy, we use a caching layer (e.g., Redis, Memcached). The leader updates the cache with version info during writes. Followers read from the cache and verify the version, and TTL or version checks ensure fresh reads.

General Design Approach

Leader handles strong-consistency writes, and followers absorb read traffic. Dynamic read routing based on follower lag, client-side, or the load balancer can monitor follower replication lag and route read requests to low-lag followers; high-lag followers handle only eventual-consistency reads.

Session stickiness for post-write reads: if there is an immediate read after a write, that read request will be routed to the leader node. Once the replication log offset catches up with the leader node’s offset, followers are allowed to accept read requests.

Optional strong-consistency modes: quorum, in which quorum reads and semi-sync replication strategies are adopted, and the router checks the cached metadata of all health nodes’ versions, LSN, and log offsets, and picks the proper ones with log offsets≥ the request’s last committed offset.

Summary

The core problem is how to distribute read traffic to followers while ensuring freshness in the presence of asynchronous replication lag.

A common general solution involves:

• Dynamic monitoring of follower lag
• Routing reads according to lag thresholds
• Using the leader for strongly consistent or immediate post-write reads
• Optionally leveraging versioning or LSN checks for correctness.