Software Architecture

Designing Retry Strategies That Don’t Bring Down Your Own System

Tree rings, retry storms, and what happens when your safety mechanism becomes the outage

Photo by Piat Van Zyl on Pexels

The first thing that went red was the payment gateway. Then the booking confirmation service. Then the notification pipeline. Within four minutes, every row on the operations dashboard had shifted from steady green to a pulsing, arterial red, and the on-call engineer’s Slack channel was scrolling faster than anyone could read.

The original failure was minor. A downstream scheduling service had gone unresponsive during a routine deployment — ninety seconds of downtime, the kind of hiatus that should have been invisible to users. Instead, it became a forty-five-minute cascading outage across a large-scale European transport platform, affecting tens of thousands of active sessions.

The root cause was not the deployment. It was not the scheduling service. It was the retry logic we had built to protect against exactly this scenario.

Every upstream service had been configured with three retries on failure. Reasonable in isolation. But when the scheduling service came back online after ninety seconds, it didn’t face its normal traffic. It faced a wall of accumulated retries from ten different services, each of which had been faithfully multiplying its own request rate by three. The recovering service buckled under thirty times its normal load, failed again, and triggered a second wave of retries across the entire mesh.

We had built a safety mechanism. The safety mechanism had become the outage.

Retry storm cascade: 10 services × 3 retries = 30x load on recovering service

I spent the next two weeks redesigning the retry architecture for that platform, and along the way I started reading about how biological systems handle repeated stress. I wasn’t looking for metaphors — I genuinely wanted to understand whether there was a principled way to think about recovery that wasn’t just “add more configuration knobs.”

What I learned reshaped how I think about resilience — not as a set of configuration values, but as an ecological system where every protective mechanism interacts with every other one, and where the wrong combination of individually sensible defaults can produce catastrophic emergent behavior.

Why retries are the most dangerous safety mechanism

The arithmetic is deceptively simple. If ten services each retry a failed request three times, a single failure generates thirty requests against the recovering service instead of ten. If those retries are immediate — no delay, no coordination — those thirty requests arrive in approximately the same instant, creating a load spike that looks identical to a denial-of-service attack. Except the attacker is your own infrastructure.

This is retry amplification, and the formula scales worse than it reads. In a service mesh with fan-out — where service A calls B, B calls C, C calls D — retries compound across each layer. Three retries at three layers means each original request can generate up to twenty-seven attempts at the deepest service. The multiplication is exponential, not additive. Your retry strategy doesn’t just add load. It multiplies it at precisely the moment your system can least afford it.

The term for this is thundering herd: a mass of clients simultaneously retrying against a recovering service, preventing recovery from ever completing. The service comes up, gets hammered, goes down, comes up, gets hammered again. I’ve watched this cycle repeat for thirty minutes in production before someone manually killed the retry logic upstream.

There’s an analogy I keep returning to from an unexpected field. Dendrochronology — the science of reading tree rings — reveals something counterintuitive about how organisms survive stress. When a tree encounters drought, it doesn’t keep trying to grow at full speed. It narrows its rings. Growth slows dramatically, sometimes to near zero. The tree survives not by pushing harder, but by pulling back.

A distributed system under stress needs the same instinct. The problem with naive retries is that they encode the opposite behavior: when the environment signals distress, the system pushes harder. More requests. More load. More pressure on the exact component that is already failing.

The rest of this article is about designing retry strategies that behave like the tree — strategies that sense stress and adapt their growth rings accordingly.

Exponential backoff is necessary but not sufficient

The first correction to naive retries is exponential backoff: instead of retrying immediately, each subsequent attempt waits exponentially longer. The formula is straightforward — the delay for attempt n is calculated as the base delay multiplied by two raised to the power of n, plus a random jitter component. If your base delay is one second, your retries fire at roughly one second, two seconds, four seconds, eight seconds — each pause wider than the last, giving the downstream service progressively more breathing room.

In Polly v8 — the current resilience library for .NET — this looks clean:

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 4,
        BackoffType = DelayBackoffType.Exponential,
        Delay = TimeSpan.FromSeconds(1),
        UseJitter = true,
        MaxDelay = TimeSpan.FromSeconds(30),
        ShouldHandle = new PredicateBuilder()
            .Handle<HttpRequestException>()
            .HandleResult<HttpResponseMessage>(r =>
                r.StatusCode == System.Net.HttpStatusCode.ServiceUnavailable ||
                r.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
    })
    .Build();

A few things to notice in this configuration.

**UseJitter = true is not optional in practice.** Without jitter, ten clients using the same exponential backoff will synchronize their retry attempts — all backing off to one second, then all retrying simultaneously, then all backing off to two seconds, then all retrying simultaneously again. You replace one thundering herd with a periodic thundering herd. Jitter decorrelates the retry timing across clients so that retries spread across the delay window rather than clustering at the boundaries.

Exponential backoff with jitter formula

**MaxDelay prevents absurdity.* Without a ceiling, exponential growth means your tenth retry waits over seventeen minutes. At that point, the original user has closed the app, called support, or found a competitor. Thirty seconds is a reasonable upper bound for most interactive systems. The cap says: if the service hasn't recovered in thirty seconds between retries, retrying is no longer a recovery strategy — it's denial.*

**ShouldHandle narrows the retry surface.** Not every failure is worth retrying. A 404 will still be a 404 after three retries. A 401 won't become a 200 with persistence. Retry only transient failures: 503 (service unavailable), 429 (rate limited), network timeouts, connection resets. Retrying a validation error wastes time and muddies your error metrics.

But here’s the trade-off. Exponential backoff with jitter is necessary, and it’s the right foundation. It is not sufficient on its own. Each service still makes its own local retry decisions with no awareness of what the rest of the system is doing. Ten services each backing off independently will still, collectively, produce significant load on a recovering downstream. The tree is narrowing its rings, yes — but every tree in the forest is still growing, and the aquifer is still draining.

Circuit breakers — the retry coordinator

If retries are the individual decisions of each tree in the forest, the circuit breaker is the drought signal that stops growth entirely when conditions are severe enough. It sits between your calling code and the downstream service, monitoring failure rates and making a binary decision: is it safe to send requests, or should we stop trying?

The circuit breaker has three states. Closed is normal operation — requests flow through, failures are counted. Open means the failure threshold has been exceeded — all requests are rejected immediately without ever reaching the downstream service. No retries, no connections, no load. Half-Open is the probe state — after a timeout, the breaker allows a single test request through. If it succeeds, the breaker closes and traffic resumes. If it fails, the breaker opens again.

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage>
    {
        FailureRatio = 0.5,
        SamplingDuration = TimeSpan.FromSeconds(30),
        MinimumThroughput = 20,
        BreakDuration = TimeSpan.FromSeconds(15),
        ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
            .Handle<HttpRequestException>()
            .HandleResult(r =>
                r.StatusCode == System.Net.HttpStatusCode.ServiceUnavailable)
    })
    .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        Delay = TimeSpan.FromSeconds(1),
        UseJitter = true
    })
    .Build();

A circuit breaker doesn’t make individual requests smarter. It makes the system collectively aware that the downstream service needs time to heal.

The order matters in that pipeline. The circuit breaker wraps the retry — meaning the breaker sees each retry attempt as a separate data point. If the first attempt fails and retries also fail, the breaker counts all of those failures against its threshold. This means retries accelerate the circuit breaker’s trip, which is exactly the behavior you want: if the service is genuinely down, the breaker should open fast, not wait for every caller to exhaust its full retry budget.

The dendrochronology parallel here is precise. In severe drought, a tree doesn’t just narrow its rings — it sometimes produces what’s called a false ring: a partial growth layer where growth started, stopped entirely mid-season, then resumed. The circuit breaker is the false ring. It’s a complete pause in growth — not a slower pace, but a full stop — followed by a cautious probe to see if conditions have improved.

The MinimumThroughput = 20 parameter is the detail that separates production circuit breakers from textbook examples. Without it, a single failed request on a low-traffic endpoint could trip the breaker because one failure out of two requests is a 50% failure rate. The minimum throughput ensures the breaker only evaluates failure ratios when it has enough data to make a statistically meaningful judgment. Twenty requests in thirty seconds is a reasonable minimum. Fewer than that, and the sample is too small to distinguish a real outage from normal variance.

When I built the retry architecture for an enterprise financial services platform, the circuit breaker was the component that took the longest to tune. We had it tripping too aggressively at first — opening on a handful of slow responses during database maintenance windows, which turned brief latency spikes into full service blackouts. The failure ratio went from 0.25 to 0.5, the sampling duration from ten seconds to thirty, and the break duration from thirty seconds down to fifteen. Every number was earned through an incident.

The retry budget pattern

There’s a subtler problem that neither exponential backoff nor circuit breakers fully solve. Both operate at the individual caller level — each service making local decisions about its own retry behavior. But in a system with dozens of services, the aggregate retry load is what matters. Even with well-tuned backoff and breakers, if enough services are simultaneously retrying, the cumulative pressure on a recovering service can still prevent recovery.

The retry budget pattern shifts the perspective from per-request retry counts to system-wide retry ratios. The idea: track the ratio of retry requests to total requests across a service. If retries exceed a threshold — say, 10% of total traffic — stop allowing new retries entirely until the ratio drops.

public class RetryBudget
{
    private long _totalRequests;
    private long _retryRequests;
    private readonly double _maxRetryRatio;
    private readonly TimeSpan _window;
    private DateTime _windowStart;

    public RetryBudget(double maxRetryRatio = 0.1, int windowSeconds = 60)
    {
        _maxRetryRatio = maxRetryRatio;
        _window = TimeSpan.FromSeconds(windowSeconds);
        _windowStart = DateTime.UtcNow;
    }

    public bool CanRetry()
    {
        ResetWindowIfExpired();
        if (_totalRequests < 10) return true;
        return (double)_retryRequests / _totalRequests < _maxRetryRatio;
    }

    public void RecordRequest(bool isRetry)
    {
        ResetWindowIfExpired();
        Interlocked.Increment(ref _totalRequests);
        if (isRetry) Interlocked.Increment(ref _retryRequests);
    }

    private void ResetWindowIfExpired()
    {
        if (DateTime.UtcNow - _windowStart <= _window) return;
        Interlocked.Exchange(ref _totalRequests, 0);
        Interlocked.Exchange(ref _retryRequests, 0);
        _windowStart = DateTime.UtcNow;
    }
}

This pattern comes from Google’s approach to gRPC retry policy, documented in their SRE practices. The reasoning is sound: if a significant fraction of your traffic is already retries, adding more retries is not helping recovery — it’s prolonging the failure. The budget acts as a system-level governor that individual retry policies cannot provide.

In practice, a 10% retry budget means that if 100 requests per second flow through a service and 10 are already retries, the eleventh retry request is rejected immediately. The calling code receives a failure and must handle it through a fallback path — degraded response, cached data, or a graceful error — rather than adding to the retry storm.

The implementation above is simplified for clarity — a production version needs thread-safe sliding windows, not hard resets, and should expose metrics for monitoring. But the core idea is sound: measure the system’s retry health in aggregate and refuse to add more retries when the system is already under retry pressure.

The trade-off is real. A retry budget can occasionally prevent legitimate retries that would have succeeded. During brief, localized blips, the budget might throttle retries that would have resolved quickly. But in practice, this cost is vastly outweighed by the protection it provides during actual outages. A retry that’s prevented from firing during a storm costs you one failed request. A retry that fires into a storm costs you continued system degradation.

Idempotency: the prerequisite nobody mentions first

Every retry strategy discussion I’ve been part of eventually reaches this question: “What happens when the first request actually succeeded but the response was lost?” Your client thinks the request failed. It retries. The server processes it a second time. Depending on what the request does, you now have a duplicate payment, a double booking, or a phantom message in someone’s inbox.

Retries without idempotency are a data integrity hazard. This is the unsexy prerequisite that makes everything else safe.

The pattern is straightforward. The client generates a unique idempotency key before the first attempt and sends it with every retry. The server checks if it has already processed that key and, if so, returns the cached response instead of re-executing the operation.

public class IdempotencyMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IDistributedCache _cache;

    public IdempotencyMiddleware(RequestDelegate next, IDistributedCache cache)
    {
        _next = next;
        _cache = cache;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        if (!context.Request.Headers.TryGetValue("Idempotency-Key", out var key))
        {
            await _next(context);
            return;
        }

        var cached = await _cache.GetStringAsync($"idempotent:{key}");
        if (cached != null)
        {
            context.Response.StatusCode = 200;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsync(cached);
            return;
        }

        var originalBody = context.Response.Body;
        using var buffer = new MemoryStream();
        context.Response.Body = buffer;

        await _next(context);

        buffer.Seek(0, SeekOrigin.Begin);
        var responseBody = await new StreamReader(buffer).ReadToEndAsync();

        if (context.Response.StatusCode is >= 200 and < 300)
        {
            await _cache.SetStringAsync(
                $"idempotent:{key}",
                responseBody,
                new DistributedCacheEntryOptions
                {
                    AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(24)
                });
        }

        buffer.Seek(0, SeekOrigin.Begin);
        await buffer.CopyToAsync(originalBody);
    }
}

The cache TTL is a judgment call. Twenty-four hours covers most retry windows generously. Longer than that, and you’re caching responses that are likely stale. Shorter, and you risk a retry arriving after the idempotency record has expired.

What catches teams off guard is the scope. Idempotency isn’t just for payment endpoints. Any operation with side effects — sending notifications, updating state, enqueuing work — needs to be idempotent if it sits behind retry logic. I learned this the hard way at that transport platform, where a retry on a message dispatch endpoint sent the same delay notification to forty thousand passengers twice. The retry succeeded. The customer support queue also succeeded — at handling the complaints.

Dead letter queues — the final safety net

When retries exhaust, the circuit breaker is open, and the retry budget is depleted, you have a failed request. The question becomes: what do you do with it?

The wrong answer is to drop it silently. A swallowed failure is invisible failure. It doesn’t trigger alerts, doesn’t generate evidence, and doesn’t appear in post-incident analysis. Three months later, someone notices a discrepancy in the data and nobody can trace it back to the original failure.

The right answer, in most message-driven systems, is a dead letter queue (DLQ). When a message has exhausted all retry attempts, it is parked in a dedicated queue — not discarded, not re-enqueued, just held. An operator or an automated process examines the dead letters, identifies patterns, and either replays them after the downstream recovers or investigates why they failed permanently.

At the transport platform, we configured RabbitMQ with a dead letter exchange bound to a dedicated queue per service. Every message that exceeded its retry count landed there with its full headers intact — the original timestamp, the failure reason, the number of attempts, the routing key. Within a week of enabling it, the DLQ exposed a serialization bug that had been silently dropping messages for months. Nobody had noticed because the messages were just vanishing. The DLQ made the invisible visible.

But the DLQ is more than a parking lot. It is evidence. A dead letter queue that fills up after a deployment tells you something specific about what that deployment broke. A DLQ that accumulates messages of a particular type tells you which contract changed. A DLQ that stays empty for months and then spikes tells you that a dependency you hadn’t been monitoring has degraded.

The dendrochronology connection here is perhaps the most direct. A tree’s growth rings are not just a record of growth. They are a dead letter queue of environmental stress. Every narrow ring records a drought survived. Every frost ring records a temperature shock absorbed. Every missing ring records a year so severe that growth stopped entirely. Dendrochronologists reconstruct centuries of climate history by reading these patterns — not from the good years, which all look alike, but from the bad years, which each carry unique signatures.

Your DLQ is the same. The messages that made it through your retry strategy, your circuit breakers, your retry budgets — those are the normal rings, the good-growth years. The messages that ended up in the dead letter queue are the narrow rings, the frost lines, the historical record of every stress your system survived. If you’re not reading them, you’re ignoring the most valuable diagnostic data you have.

Naive retry vs resilient retry architecture

Resilience stack health dashboard

I started this article with a dashboard turning red. Not from a service failure, but from the retries meant to protect against service failures. That incident taught me something that years of architecture design had not: resilience is not a set of patterns you apply. It is a system you design.

Exponential backoff without a circuit breaker just slows down the thundering herd. A circuit breaker without a retry budget lets aggregate load overwhelm a recovering service. Retries without idempotency create data corruption. Any of these mechanisms in isolation is dangerous precisely because it creates confidence without completeness.

The full stack — backoff with jitter, circuit breakers with proper thresholds, a system-wide retry budget, idempotency for side effects, dead letter queues for exhausted retries — works as an ecology, not a checklist. Each component restrains the others. The retry budget limits what backoff allows. The circuit breaker overrides what the retry budget permits. The DLQ catches what everything else misses. And idempotency ensures that the retries which do fire don’t create worse problems than the ones they’re solving.

Like the tree that survives drought not by pushing harder but by narrowing its rings, pausing growth, and recording the stress in its structure — a well-designed retry architecture survives failure by sensing it, adapting to it, and preserving evidence of it.

But there’s a question I still sit with after that transport platform incident, and after every retry architecture I’ve built since.

What happens when the thing that’s supposed to recover IS the thing that’s failing? When the circuit breaker itself has a bug? When the retry budget calculation drifts? When the dead letter queue fills up and the alerting system that monitors it is the same system that’s degraded?

We build resilience on the assumption that at least the resilience layer is reliable. We build safety mechanisms on the assumption that the safety mechanisms don’t need safety mechanisms.

I’m not sure that assumption survives contact with production.