Why .NET AI Gateways Melt Down on 429s: The Retry Storm Nobody Plans For

The easiest way to make a healthy AI feature look broken is to wrap it in a naive retry policy.

Most teams see429 Too Many Requestsfrom a model provider and think, "No problem, we already use retries." That logic works for some flaky HTTP dependencies. It fails badly when dozens of .NET instances hammer the same LLM endpoint on the same cadence. Instead of smoothing traffic, the retry layer synchronizes failure.

The result is familiar: queue depth climbs, token spend spikes, p95 latency spreads into endpoints that were never rate-limited, and on-call starts blaming the model vendor. In practice, the model is often doing exactly what it should do. Your gateway is the thing turning backpressure into an outage.

The Production Scenario

Imagine a .NET 9 API serving chat and document summarization for multiple tenants:

• 40 application instances behind a load balancer
• a shared queue for long-running prompt work
• one LLM provider with tenant-level rate limits
• a fallback model that is slower and more expensive

At 09:00, one large tenant sends a burst of requests after a scheduled import job. The provider starts returning429. Every app instance retries after the same fixed delay. The queue now contains original work plus duplicate retries. The fallback model gets activated too early, so cost rises exactly when throughput drops.

Architecture view: the outage is not just the provider returning

What Actually Breaks

Architecture view: the outage is not just the provider returning429. The real blast radius comes from the gateway replaying work into a shared queue on the same cadence, which multiplies duplicate execution, latency, and token cost.

The hidden problem is coordination. Fixed-delay retries line up across workers. Backoff without jitter causes burst waves. Queue consumers, HTTP callers, and fallback logic all amplify each other.

The Safer Approach

Treat the LLM as a constrained dependency and make admission control explicit.

Key rules:

1. Budget requests per tenant before they hit the provider.
2. Retry with jitter, not fixed intervals.
3. Deduplicate equivalent prompt work when the same request is already in flight.
4. Open a circuit when rate limiting crosses a threshold, then shed or defer work.
5. Move fallback routing behind cost-aware rules instead of panic switching.

Here is a practical pattern in ASP.NET Core:

public sealed class LlmRequestGate
{
 private readonly ConcurrentDictionary<string, SemaphoreSlim> _tenantLocks = new();
 private readonly ConcurrentDictionary<string, DateTimeOffset> _circuitUntil = new();
 private readonly TimeSpan _circuitWindow = TimeSpan.FromSeconds(30);

 public async Task<T> ExecuteAsync<T>(
 string tenantId,
 string requestHash,
 Func<CancellationToken, Task<T>> operation,
 CancellationToken cancellationToken)
 {
 if (_circuitUntil.TryGetValue(tenantId, out var until) && until > DateTimeOffset.UtcNow)
 {
 throw new InvalidOperationException("LLM circuit is open for this tenant.");
 }

 var gate = _tenantLocks.GetOrAdd(tenantId, _ => new SemaphoreSlim(4, 4));
 await gate.WaitAsync(cancellationToken);

 try
 {
 for (var attempt = 0; attempt < 3; attempt++)
 {
 try
 {
 return await operation(cancellationToken);
 }
 catch (HttpRequestException ex) when (IsRateLimited(ex))
 {
 if (attempt == 2)
 {
 _circuitUntil[tenantId] = DateTimeOffset.UtcNow.Add(_circuitWindow);
 throw;
 }

 var jitterMs = Random.Shared.Next(150, 900);
 var delay = TimeSpan.FromMilliseconds((attempt + 1) * 500 + jitterMs);
 await Task.Delay(delay, cancellationToken);
 }
 }

 throw new UnreachableException();
 }
 finally
 {
 gate.Release();
 }
 }

 private static bool IsRateLimited(HttpRequestException ex) =>
 ex.StatusCode == System.Net.HttpStatusCode.TooManyRequests;
}

This code does three useful things:

• limits concurrency per tenant instead of globally starving everyone
• adds jitter so retries do not synchronize
• opens a short tenant-level circuit after repeated rate limits

It is not enough on its own, but it stops the most common retry storm pattern.

Common Mistakes

• Using Polly retries with the same delay across all workers
• Triggering fallback models immediately on the first429
• Ignoring deduplication for repeated prompts from the same upstream event
• Measuring only provider errors, not queue growth and duplicate execution

Trade-offs

ApproachWhen it helpsWhen it hurtsFixed retriesSmall internal tools with low concurrencySynchronizes failure under burst loadJittered retries + tenant gateMulti-tenant AI APIs with shared provider limitsAdds implementation complexityImmediate fallback modelCritical low-volume flowsExplodes cost during traffic spikesQueue deferral with backpressureBatch or async workloadsUsers may wait longer for completion

Final Take

The model provider is not always the unstable part of your AI system. Very often, your retry behavior is.

When a provider says429, it is telling you the system needs less pressure, not more enthusiasm. Teams that treat that signal as an architecture problem build calmer AI gateways, cheaper fallbacks, and fewer midnight incidents.

If you are building AI features in .NET, start by reviewing your retry policy before you tune another prompt.