Distributed Process Consistency: The Real Problem Isn’t “Can It Run” — It’s “Run Once, On Schedule, and Hand Off Cleanly When It Fails”

Distributed System

Disclosure: I use GPT search to collection facts. The entire article is drafted by me.

Most teams first encounter distributed process consistency as a tooling decision: add leader election, configure a distributed lock, set up a health check, wire a service registry, and declare the system “stable.” The reality arrives later, usually at 2am, when something that was supposed to run exactly once runs three times, or when a task that should have failed over quietly instead leaves two nodes arguing over who owns the work.

The gap between those two experiences is what this article is about.

Distributed process consistency isn’t a component you add. It’s an emergent property of how you answer four questions across every layer of your system:

• Who has the authority to make decisions
• Who is currently holding exclusive access to a critical operation
• Who is still alive
• Who takes over when the answer to any of the above changes unexpectedly

On a single machine, these questions barely exist. The process either runs or it doesn’t. There’s one clock, one scheduler, one process table. In a distributed system, each of these questions requires explicit modeling — and the ways they interact with each other are where the real problems live.

Before Picking Tools: Get the Problem Taxonomy Right

AI Generated Image

The most common mistake in distributed systems work isn’t choosing the wrong tool. It’s applying the right tool to the wrong problem classification.

Split brain is not “the system went down.” It’s “the system kept running, but fractured.” After a network partition, a long GC pause, or a node freeze, you may end up with two nodes that both believe they are the authoritative leader. Both are healthy by their own assessment. Both are processing the same class of tasks. The data corruption doesn’t happen because something broke — it happens because something kept working when it shouldn’t have.

Duplicate execution is related but distinct. A scheduled task is supposed to run once per hour. The leader migrates. The lock that should prevent re-entry expires during task execution. The old leader hasn’t finished. The new leader starts the same task. The overlap window might be two seconds. That’s long enough to double-charge a payment, double-decrement inventory, or send the same notification twice.

State inconsistency is the most insidious form. Individual nodes appear healthy. Service discovery says they’re available. But they’re running with different configuration versions, different task state snapshots, or different interpretations of who currently holds a critical lock. No alarm fires. No process crashes. The inconsistency just quietly accumulates until it surfaces as a data anomaly or a user complaint.

Deadlock and livelock round out the taxonomy. Distributed locks are not inherently safe just because they’re distributed — they can produce exactly the same failure modes as single-machine locks, plus network-induced variants. A node acquires a lock and crashes before releasing it. All other nodes queue indefinitely. Or worse: nodes acquire and release locks in a cycle, consuming resources and generating contention without any work being completed.

The reason taxonomy matters is this: leader election solves problem class one. Distributed locks solve problem class two. Health checks and leases solve problem class three. Applying any of these to the wrong class wastes engineering effort and introduces false confidence. A team that adds a Redis lock to prevent split brain hasn’t solved split brain — they’ve added a coordination mechanism that has no authority over which node claims leadership.

The Inviolable Baseline: Assume Network Partitions and Node Failures Are Normal

Before examining any specific technique, there is one assumption that has to be correct or nothing else will be.

Networks drop packets. Connections time out. Routers reroute. Load spikes delay acknowledgments. This happens in well-operated cloud environments, not just in unreliable edge cases. Nodes disappear too — process crashes, OOM kills, container evictions, disk exhaustion, kernel panics, maintenance restarts. The fact that your system ran stably for three weeks doesn’t mean the fourth week will be the same.

The counterargument worth taking seriously: not every system needs to be designed for every failure mode. An internal analytics pipeline with ten-second SLAs can tolerate eventual consistency in places where a payment processing system cannot. Over-engineering for fault tolerance has real costs — added complexity, added latency from consensus rounds, added operational surface area.

The valid version of this counterargument is: calibrate the tolerance to the business consequence. If your scheduled task runs twice, does that cost you ten cents or ten thousand? If your leader takes thirty seconds to fail over, does that cause a user-visible outage or a minor blip in a background metric?

The invalid version is: skip modeling failure because failures are rare. Production failure modes are not rare — they’re just unevenly distributed across time. When they hit, they hit at inconvenient moments, under load, and often in combination. The system that was never designed for failure gives you the worst version of every incident: no prepared recovery path, no clear authority for who should handle the state, no way to determine whether the in-flight work was committed or not.

Mature systems don’t assume failures won’t happen. They define the boundary within which failures are contained.

Leader Election: The Problem Is Not “Who Gets the Crown” — It’s “When Does the Crown Expire”

Leader election is typically introduced as the solution to uniqueness: only one node should be doing a given thing at a given time. That framing is correct but incomplete. The harder problem is: what happens to the crown when the leader slows down, gets GC-paused, or hits a network partition?

etcd Lease + Election is the strongest production option for cases that require strict ordering and strong consistency. The mechanism deserves a close look because understanding it makes the tradeoffs concrete.

// Leader election using etcd client-go (Kubernetes-native pattern)
import (
    clientv3 "go.etcd.io/etcd/client/v3"
    "go.etcd.io/etcd/client/v3/concurrency"
)

func runLeaderElection(ctx context.Context, client *clientv3.Client, role string) error {
    // Create a session with a TTL - this is the lease
    // If the session expires (heartbeat stops), the lock is released
    session, err := concurrency.NewSession(client, concurrency.WithTTL(15))
    if err != nil {
        return fmt.Errorf("failed to create session: %w", err)
    }
    defer session.Close()
    // Create an election object bound to this session
    election := concurrency.NewElection(session, "/leader/"+role)
    // Campaign - blocks until this node wins the election
    if err := election.Campaign(ctx, nodeID); err != nil {
        return fmt.Errorf("election campaign failed: %w", err)
    }

    log.Printf("Node %s elected leader for role %s", nodeID, role)

    // Perform leader work - run tasks, coordinate, etc.
    // If this context is canceled or the session TTL expires,
    // leadership is automatically relinquished
    return doLeaderWork(ctx, election)
}

What this code illustrates is the binding between session (lease) and election (authority). The leader isn’t just the node that won a vote — it’s the node whose session is currently valid. When the session TTL expires because heartbeats stopped, the election key is deleted by etcd, and other nodes watching for that deletion immediately trigger a new election round. The crown has a timer on it. That timer is the safety mechanism.

The TTL setting matters significantly. Too short, and network jitter causes unnecessary leader migrations. Too long, and a failed leader takes too long to lose authority. The practical starting point is a TTL of 10–30 seconds with heartbeat intervals at roughly one-third of the TTL. This gives three missed heartbeats before expiry — enough buffer for transient delays without creating multi-minute failover windows.

Kubernetes Lease API is the practical choice for applications already running in Kubernetes. The coordination.k8s.io/v1 Lease object is exactly the same conceptual mechanism — a resource with a holder identity and a renewal timestamp. client-go's leaderelection package wraps this into a pattern that most Kubernetes operators use directly. The advantage is that you're not adding an external dependency (etcd is already there), and the coordination machinery is managed by the cluster itself.

The boundary that leader election does not cross: it controls who initiates work, not whether the work itself is idempotent. A correctly-elected leader that executes a non-idempotent task during a failover window can still produce duplicate effects. Election is the entry gate. Idempotency is the correctness guarantee inside the gate.

Distributed Locks: The Dangerous Part Is Not Acquiring — It’s the Expiry Window

Distributed locks are the mechanism for mutual exclusion — ensuring that exactly one process has access to a shared resource or operation at any moment. The implementation looks simple. The failure modes are not.

Redis-based locking is the entry-level pattern, typically using SET key value NX PX milliseconds with a unique per-holder value and a Lua script for atomic release.

import redis
import uuid
import time

class RedisDistributedLock:
    def __init__(self, client: redis.Redis, key: str, ttl_ms: int = 30000):
        self.client = client
        self.key = key
        self.ttl_ms = ttl_ms
        self.holder_id = str(uuid.uuid4())  # unique per lock acquisition

    def acquire(self, timeout_s: float = 10.0) -> bool:
        deadline = time.time() + timeout_s
        while time.time() < deadline:
            # NX: only set if not exists; PX: expire in milliseconds
            result = self.client.set(self.key, self.holder_id, nx=True, px=self.ttl_ms)
            if result:
                return True
            time.sleep(0.1)
        return False

    def release(self) -> bool:
        # Lua script ensures we only release OUR lock, not someone else's
        # This is critical: if our TTL expired and someone else acquired the lock,
        # we must not release their lock
        lua_script = """
        if redis.call("get", KEYS[1]) == ARGV[1] then
            return redis.call("del", KEYS[1])
        else
            return 0
        end
        """
        result = self.client.eval(lua_script, 1, self.key, self.holder_id)
        return bool(result)

    def __enter__(self):
        if not self.acquire():
            raise TimeoutError(f"Could not acquire lock {self.key}")
        return self

    def __exit__(self, *args):
        self.release()

The unique holder ID and the Lua atomic release are not optional refinements — they’re the mechanism that prevents releasing another holder’s lock after expiry. Without them, the scenario is: Node A acquires lock, A’s task takes longer than TTL, lock expires, Node B acquires lock, Node A’s task finishes and calls DEL key, deleting Node B's valid lock. Node C then acquires it. All three nodes now believe they have exclusive access within overlapping windows.

The honest assessment of Redis locking from Martin Kleppmann’s analysis — which remains the clearest public treatment of the problem — is that Redis locks are appropriate for efficiency-class locking (preventing most duplicate execution, reducing thundering herd) but not for correctness-class locking (guaranteeing that exactly one node ever executes a given operation). If your task can tolerate occasional duplicate execution when combined with application-level idempotency checks, Redis locks work. If duplicate execution has direct correctness consequences — double-charging, double-decrement, conflicting state writes — you need stronger guarantees.

etcd mutex locking provides those stronger guarantees through the same session-and-lease mechanism as leader election. The lock is bound to a session with a TTL. If the holder’s process dies or becomes unresponsive, the session expires, the lock is released, and a waiting node acquires it. The Raft consensus underlying etcd means that lock state is replicated across the etcd cluster — there’s no single point of failure in the coordination layer itself.

Redlock — Redis’s multi-node distributed lock algorithm — remains genuinely controversial in the industry. The algorithm acquires locks on a majority of N Redis instances and accepts the lock only when it succeeds on more than N/2 instances within a time window. The intention is to eliminate single-node Redis as a coordination SPOF. The criticism, articulated clearly in Kleppmann’s analysis, is that the algorithm’s correctness assumptions about time and clock drift don’t hold under certain failure scenarios including GC pauses and network delays. The pragmatic position: treat Redlock as a middle-ground option, higher confidence than single-instance Redis but lower than etcd, appropriate for high-availability locking where the occasional edge-case duplicate is tolerable but single-node Redis failure is not.

Heartbeats Are Not “Reporting In” — They Are “Renewing a Lease”

The mental model matters here. A heartbeat is not a status broadcast. It is the mechanism by which a node continuously proves it is still alive and deserves to keep whatever authority it currently holds.

The structural relationship: heartbeats maintain sessions, sessions maintain leases, leases maintain authority. Break any link in that chain and the authority expires. This is intentional. It’s the safety net that catches the failure mode of a leader that is still technically running but has stopped making progress — due to full GC, I/O blocking, CPU saturation, or network isolation.

The calibration parameters:

• Heartbeat interval: typically one-third to one-quarter of the lease TTL. This gives the system two to three missed heartbeats before expiry, providing buffer for transient delays without creating long failover windows.
• Session TTL: long enough to survive normal network jitter and scheduling delays, short enough that a failed node loses authority within an acceptable window. 10–30 seconds covers most production environments.
• Failure detection threshold: the number of consecutive missed heartbeats before declaring a node failed. Three is a common starting point — it reduces false positives from transient hiccups while catching genuine failures within one full TTL period.

The “false death” problem deserves specific attention. A node in a full GC pause looks exactly like a dead node from outside. Its heartbeats stop. The session TTL ticks toward expiry. If the TTL is set too short and the GC pause is too long, the node loses its lease mid-task, a new node picks up the work, and then the GC-paused node wakes up and resumes — with two nodes now executing the same operation.

This is not a theoretical edge case. JVM-based services running on shared cloud infrastructure can see GC pauses exceeding 10 seconds under load. A 5-second TTL combined with a JVM workload is a reliability risk. The calibration has to account for the actual pause budget of the runtime environment, not just the ideal case.

Kubernetes probe design applies the same concepts at the platform layer. livenessProbe answers "is this process still healthy enough to keep running?" readinessProbe answers "is this process ready to handle traffic?" The distinction matters operationally:

• A node that’s alive but overloaded should fail its readiness check (stop receiving new requests) before failing its liveness check (triggering a restart). Restarting a valid process under load often makes things worse, not better.
• A node that’s become permanently stuck — hung thread, corrupted state — should fail its liveness check and be restarted.
• A node that’s temporarily dependent on a downstream service being slow should fail readiness (remove from load balancer rotation) without triggering a liveness restart.

Treating these as interchangeable by using the same endpoint for both is a common misconfiguration. It results in either too many restarts (liveness too aggressive) or traffic being sent to unhealthy nodes (readiness too permissive).

Service Discovery: The Problem Is Not “Finding the Address” — It’s “Finding the Current Authoritative State”

Service discovery is commonly described as “how clients find service endpoints.” That description undersells the actual problem. The client doesn’t just need an address — it needs the address of a node that is currently healthy, currently authoritative, and currently running the version it expects.

AI Generated Image

The tools serve different points on the consistency-availability spectrum:

ZooKeeper provides strong consistency and clear ordering guarantees, making it appropriate for cases where the service registry itself needs to be the authoritative coordination point — not just a discovery mechanism. Its design prioritizes consistency over availability under partition.

etcd is the Kubernetes ecosystem’s standard coordination substrate. It handles service state with the same Raft-based consistency guarantees as its use in leader election and locking. It’s the natural choice for systems already deeply integrated with Kubernetes.

Consul provides a combination of service discovery, health checking, configuration, and optional service mesh that most purpose-built registries don’t. Its health check integration is particularly relevant: a service is only discoverable if its health checks pass. This means the registry’s view of available nodes tracks actual health, not just registration state. The tradeoff is that health check design becomes a critical dependency — poorly designed health checks produce either false availability (routing to unhealthy nodes) or false unavailability (removing healthy nodes under load).

Eureka sits at the more available end of the spectrum, offering eventual consistency in exchange for higher tolerance for network partitions. For systems where brief registration staleness is acceptable and service endpoint churn is low, this tradeoff makes sense. For systems requiring rapid failover or gray-scale routing, the propagation delay becomes a risk.

The gap that service discovery cannot eliminate is client-side cache staleness. Every client-side cache of endpoint data has a TTL. Between a node becoming unhealthy and the client’s cached endpoint list expiring, requests will reach a node that the registry already considers unavailable. The right response to this gap is not trying to eliminate it — it’s designing the application to handle request failures gracefully and sizing the cache TTL relative to your acceptable failover window.

Consistency Isn’t Binary — It’s Layered by Business Risk

A consistent mistake in distributed system design is applying the same consistency model everywhere. Strong consistency has real costs: latency from consensus round trips, reduced availability under partition, operational complexity. Applying those costs to every data type in the system is wasteful. Not applying them where they’re necessary is dangerous.

The productive approach is to segment by business consequence.

Strong consistency layer — leader election state, distributed lock state, critical configuration, authoritative task control state. These data types directly determine system behavior. An inconsistency here means two nodes disagreeing about who should be running, what the current configuration is, or whether a critical lock is held. The appropriate tools are etcd, ZooKeeper, or Consul with Raft paths.

Eventual consistency layer — logs, metrics, monitoring data, analytics events, audit trails. These data types describe what happened, not what should happen next. A slight delay in propagation doesn’t change system behavior — it only delays observability. Kafka, ClickHouse, object storage, and async aggregation systems are appropriate here. Trying to make metrics collection strongly consistent adds latency and complexity with no operational benefit.

Mixed-pattern layer — scheduled task execution. This is the most common source of confusion because it combines uniqueness requirements (only one executor) with high-volume events (many tasks, many trigger points). The productive decomposition: use strong consistency for the control plane (who owns the task slot), use idempotency and versioning for the execution plane (what happened during execution), and use an eventual consistency sink for results (what the task produced). This way, even if a leader failover happens mid-task, the control plane transition is clean, the execution is either committed or re-attempted with idempotent protection, and the results are captured correctly.

Kubernetes Gives You a Foundation — Not a Finished Building

For teams running on Kubernetes, the platform provides significant distributed process management capability out of the box. The mistake is treating that capability as a complete solution rather than as a starting point.

The Operator pattern is the most powerful abstraction for managing stateful, lifecycle-heavy distributed processes. An Operator codifies the operational knowledge that would otherwise live in runbooks, scripts, and engineers’ heads — into a controller that monitors actual state, compares it to desired state, and takes action to close the gap. For systems with complex initialization sequences, leader-dependent operations, or multi-phase upgrade procedures, an Operator is significantly more maintainable than layering these behaviors onto a standard Deployment.

StatefulSets provide stable network identity and ordered startup/shutdown for workloads that require them. The use case is specific: processes where each replica has a distinct identity (different data shard, different role in a consensus group, different configuration based on ordinal). They’re not appropriate as a default for any “stateful” application — the ordered rollout and stable identity come with tradeoffs in scheduling flexibility and cluster bin packing.

PodDisruptionBudgets are the most underused safety mechanism on the list. The failure mode they prevent is a cluster maintenance event — node drain, rolling upgrade, cluster autoscaler action — reducing available replicas below quorum or below a required minimum. Without a PDB, a reasonable-looking cluster maintenance operation can temporarily reduce a three-replica service to one replica, or take all replicas of a singleton down simultaneously. PDBs put a hard floor on simultaneous disruption. For any service where quorum or minimum replica count matters, a PDB is not optional.

client-go leader election provides the K8s-native path for leader election without requiring an external etcd client. For controllers and operators running inside the cluster, this is the standard approach — the Kubernetes API server mediates the Lease object, and the existing cluster infrastructure handles the consistency guarantees.

The platform doesn’t substitute for application-level design. Kubernetes will restart your failed pod, update your endpoint, and evict your unhealthy node. It will not guarantee that the work your pod was doing was idempotent, that your service handles reconnection gracefully, or that your configuration updates are atomic. The platform handles process lifecycle. Application correctness still requires application design.

The Layered Architecture That Actually Works in Production

Rather than advocating for a single stack, the more useful mental model is a responsibility map. Different problems belong to different layers, and each layer should have exactly one clearly-owned solution.

The discipline is in keeping the layers separate. The failure mode is when teams reach for a strong-consistency tool for everything (over-engineering, excessive latency) or when they apply a fast eventual-consistency tool to a problem that requires correctness guarantees (silent bugs under concurrent load).

The Explanation Complete Moment

Here is the structural insight the whole analysis converges toward.

Split brain, duplicate execution, and state inconsistency are not fundamentally technology problems. They’re modeling problems. Every one of them comes from a system that failed to explicitly represent one of the four questions: who has authority, who has exclusion, who is alive, who takes over.

On a single machine, these questions have implicit answers. The OS provides a single scheduler, a single process table, a single clock. Answers are free.

In a distributed system, the answers have to be purchased — from consensus rounds, from lease renewals, from health check evaluations, from election protocols. Each purchase has a cost in latency, complexity, and operational burden. The engineering discipline is knowing which purchases are worth making and which ones you can avoid by designing for idempotency instead.

The hardest part of distributed process consistency is not the algorithms. The algorithms are well-understood and well-implemented in production-grade libraries. The hardest part is correctly scoping which guarantees you actually need, designing verification so you know when those guarantees hold, and specifying failure behavior precisely enough that when the guarantee fails — and it will, eventually — the system knows how to hand off rather than how to silently corrupt.

The goal is not a system that never fails. The goal is a system that, when it fails, still knows who it is, who should continue, and who should stop.

If you’d like to show your appreciation, you can support me through:

✨ **Patreon ✨ [Ko-fi](https://ko-fi.com/jinlowmedium) ✨ [BuyMeACoffee](https://buymeacoffee.com/jinlowmedium)**

Every contribution, big or small, fuels my creativity and means the world to me. Thank you for being a part of this journey!