Software Architecture

The Retry Policy That Took Down the Entire Cluster

Five retries, exponential backoff, jitter — and a cluster outage in under ninety seconds.

Photo by Plato Terentev on Pexels

Dear fellow engineer who just shipped a retry policy,

I know exactly what you did. You pulled in Polly, configured exponential backoff with jitter, maybe even added a WaitAndRetryAsync with five attempts, and felt the quiet satisfaction of a responsible adult adding resilience to a distributed system.

I did the same thing. Several months ago, on a platform serving around 500 concurrent clients (it was one of my side projects), I watched that exact configuration — the textbook-correct one — turn a partial database slowdown into a full cluster outage in under ninety seconds.

The retry policy didn’t save us. The retry policy was the weapon.

The textbook implementation

Here is what you probably wrote, because it is what I wrote:

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(
        retryCount: 5,
        sleepDurationProvider: attempt =>
            TimeSpan.FromSeconds(Math.Pow(2, attempt))
            + TimeSpan.FromMilliseconds(Random.Shared.Next(0, 1000)),
        onRetry: (exception, timeSpan, context) =>
        {
            Log.Warning("Retry after {Delay}ms: {Message}",
                timeSpan.TotalMilliseconds, exception.Message);
        });

The snippet uses the Polly v7 API for readability; v8’s ResiliencePipelineBuilder is the current shape but doesn't change the failure mode this article is about.

Five retries. Exponential backoff. Jitter to decorrelate. This is exactly what the Polly documentation recommends, and exactly what every blog post on resilience tells you to do.

Run the arithmetic before you deploy it.

You have 500 clients. Each sends one request per second. Under normal conditions, your downstream handles 500 requests per second without breaking a sweat. Now the downstream starts returning errors — say, 10% of requests fail because a connection pool is saturated.

Fifty clients get errors. Each one retries. Now your downstream isn’t handling 500 requests per second. It’s handling 550. That extra 10% load pushes the failure rate to 18%. Ninety clients get errors. Each retries. Now you’re at 590 requests per second. Failure rate climbs to 25%. One hundred and twenty-five clients retry. You’re at 625.

By the time the fifth retry wave hits, your 500-client system has generated up to 2,500 additional requests — all aimed at a service that was already struggling at 500.

Five-tile metrics dashboard: Wave 0 at 500 rps with 10% failure (green), Wave 1 at 550 rps with 18% failure (amber), Wave 2 at 590 rps with 25% failure (amber), Wave 3 at 625 rps with ~40% failure (red), final tile shows 2,500 rps of cumulative additional load across 5 waves (red). Color shift traces the cascade compounding.

In thermodynamics, this shape is called a runaway reaction: a small nudge amplifies itself through the very mechanism designed to correct it. Positive feedback loops are easy to see in retrospect and almost impossible to see in advance, because the policy looks like negative feedback. A failed request triggers a corrective action. The corrective action is just more load on the failing thing.

The hotel that ran out of rooms

The hospitality industry understood this dynamic decades before we started building microservices.

Hotels routinely overbook rooms. The math is elegant: historical data shows roughly 15% of guests with reservations don’t show up. So a 100-room hotel sells 115 reservations. On a normal Tuesday, 14 people cancel or no-show, one guest arrives to a perfectly available room, and the hotel runs at 100% occupancy instead of 86%. Everyone wins.

Both systems rely on the assumption that failures are independent and randomly distributed.

Then a major conference comes to town. Suddenly the walk rate drops from 15% to 2–3%. Now that 100-room hotel has 112 guests standing in the lobby, and only 100 rooms. The overbooking model — designed for normal variance — becomes the crisis.

The parallel runs at the structural level. Both systems rely on the assumption that failures are independent and randomly distributed. A few retries here, a few there, spread across time, absorbed by a system with spare capacity. But when the downstream is genuinely unhealthy, failures are correlated. Every client fails at the same time, retries at the same time, and the assumption of independence — the invisible load-bearing wall of the entire model — collapses.

Amazon learned a version of this in December 2012. An engineer running a routine maintenance process accidentally deleted state data the Elastic Load Balancing service depended on — the kind of human-operations slip no unit test would catch. What turned a contained data-loss incident into a many-hour, multi-service outage was the recovery path: as ELB tried to rebuild state, clients hammered it with retries, and the retry load competed with the recovery work itself. The public post-mortem is careful with language, but the shape it describes is the one this article is about — retries amplifying load on a service that was trying to heal.

The pattern has a close cousin in distributed systems folklore: the Thundering Herd. The herd, strictly, is the wake-up problem — every waiting client stampedes the moment a shared resource becomes available. Retry storms run the same shape against recovery. The resource fails, the herd waits, the resource starts to recover, the herd stampedes. The system oscillates between collapse and recovery until something external intervenes.

The negative feedback loop you forgot to install

A different retry policy alone — fewer retries, longer backoff — buys time. It doesn’t change the dynamic. The dynamic changes only when something in the path reduces pressure when the system is failing, instead of adding to it.

In control theory, this is the difference between positive and negative feedback. Positive feedback amplifies deviation from equilibrium. Negative feedback dampens it. A circuit breaker is negative feedback.

var circuitBreaker = Policy
    .Handle<HttpRequestException>()
    .CircuitBreakerAsync(
        exceptionsAllowedBeforeBreaking: 10,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (exception, duration) =>
            Log.Warning("Circuit OPEN for {Duration}s — halting retries",
                duration.TotalSeconds),
        onReset: () =>
            Log.Information("Circuit CLOSED — downstream recovered"));

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(
        retryCount: 3,
        sleepDurationProvider: (attempt, context) =>
        {
            var baseDelay = TimeSpan.FromSeconds(Math.Pow(2, attempt));
            var jitter = TimeSpan.FromMilliseconds(Random.Shared.Next(0, 1000));

            // GetCurrentFailureRate(): sliding-window counter (last 30s of
            // attempts and failures); assumed shared across the HttpClient's
            // policy scope, not per-call.
            var failureMultiplier = GetCurrentFailureRate() > 0.5 ? 4.0 : 1.0;

            return baseDelay * failureMultiplier + jitter;
        });

// The order matters: circuit breaker wraps the retry
var resilientPolicy = Policy.WrapAsync(circuitBreaker, retryPolicy);

Two things changed. The circuit breaker opens after 10 consecutive failures and stays open for 30 seconds. During those 30 seconds, no retries even attempt to reach the downstream. The system gets breathing room to recover. The retry itself is adaptive: when the observed failure rate crosses 50%, the backoff delay multiplier jumps from 1x to 4x, so the retries that do fire spread across a window wide enough for the downstream to breathe.

The assumption hiding in plain sight

The danger of the textbook retry policy is the implicit assumption that your retries are the only ones happening.

When you write retryCount: 5, you're thinking about one client's journey through a failure. Five attempts, increasing delays, reasonable timeout. In isolation, it is perfectly sensible. But you are not the only client. There are 500 of you, and you're all making the same perfectly sensible decision simultaneously.

This is the tragedy of the commons applied to system resources. Each individual retry is reasonable; the aggregate is what destroys the cluster.

The fix that most teams miss is a retry budget — system-wide, not per-client.

public class RetryBudget
{
    private readonly int _windowSeconds;
    private readonly double _maxRetryRatio;
    private long _requestCount;
    private long _retryCount;
    private long _windowStartTicks = DateTime.UtcNow.Ticks;

    public RetryBudget(int windowSeconds = 60, double maxRetryRatio = 0.1)
    {
        _windowSeconds = windowSeconds;
        _maxRetryRatio = maxRetryRatio;
    }

    public void RecordRequest() => Interlocked.Increment(ref _requestCount);

    public bool CanRetry()
    {
        ResetWindowIfNeeded();

        if (_requestCount == 0) return true;

        var currentRatio = (double)_retryCount / _requestCount;
        if (currentRatio >= _maxRetryRatio) return false;

        Interlocked.Increment(ref _retryCount);
        return true;
    }

    private void ResetWindowIfNeeded()
    {
        var now = DateTime.UtcNow.Ticks;
        var start = Interlocked.Read(ref _windowStartTicks);
        if (new TimeSpan(now - start).TotalSeconds <= _windowSeconds) return;

        // Only the thread that successfully advances the window resets the counters.
        if (Interlocked.CompareExchange(ref _windowStartTicks, now, start) == start)
        {
            Interlocked.Exchange(ref _requestCount, 0);
            Interlocked.Exchange(ref _retryCount, 0);
        }
    }
}

A retry budget caps the total percentage of traffic that can be retries. Set maxRetryRatio to 0.1, and no more than 10% of your requests in any rolling window can be retry attempts. Once the budget is exhausted, clients fail fast instead of piling on.

This is the negative feedback the hotel industry adopted. When the conference-weekend scenario triggers, you stop selling rooms. You don’t wait until 112 guests are standing in the lobby. You cut off bookings the moment occupancy crosses a threshold.

I want to be honest about how these three mechanisms fail, because none of them is free.

Putting the circuit breaker outside the retry — wrapping it around, not inside — is the single non-cosmetic ordering choice. With retry on the outside, the breaker only sees individual attempts and can never halt the cascade. With the breaker outside, it stops the whole stack the moment downstream health turns red. The cost shows up at the user-facing edge: a breaker that opens during a recoverable blip turns slow failures into fast ones while the downstream is mid-heal. Fast failures are still failures, just less ambiguous ones.

The adaptive backoff multiplier reads clean in code and depends, in production, on telemetry that can stall under exactly the conditions you wrote it for. I have watched this fail. When the metrics pipeline gets backpressured by the same surge that is overwhelming the downstream, the failure-rate signal reads zero, the multiplier never engages, and you are back where you started — possibly worse, because you trusted a control loop that wasn’t running.

The retry budget is harder to ship than to design. Per-process, it works. A fleet of fifty instances can each be running 9.9% utilized — comfortably under their local caps — and collectively drown the downstream anyway. The shared counter you would need to coordinate the fleet is itself a distributed system, with its own latency, its own consistency model, and its own failure modes when the downstream you’re trying to protect is the one mediating the shared state. Saving the cluster sometimes costs one user a bad afternoon, and saving the cluster coordination layer sometimes costs you a worse one.

Before you ship the next retry policy, run this calculation:

var concurrentClients = 500;
var retriesPerClient = 5;
var totalRetryRequests = concurrentClients * retriesPerClient;

Console.WriteLine(
    $"If every client retries, your failing service " +
    $"receives {totalRetryRequests:N0} additional requests.");
// Output: If every client retries, your failing service
//         receives 2,500 additional requests.

The number that prints is the extra load your retry policy will place on a system that is already on fire.

[embed]List: Software Architecture | Curated by Anto Semeraro | Medium Software Architecture · Exploring established and proven solutions for designing software systems: from microservices…medium.com