When Your Mobile App Accidentally DDoS-es Your Own Backend

How a missing backoff turned a 400 error into 823,000 redundant API calls — and what we did about it

It was a regular Wednesday morning when our Dynatrace dashboards lit up. API response times for one of our core services were climbing steadily, and Kibana logs showed an unusual pattern — a single endpoint was being hammered with requests at a rate that didn’t match any known user behavior.

We weren’t under attack. We were attacking ourselves.

Over the course of four days, a single flow in our driver-facing mobile app had generated approximately 823,000 redundant API calls. Not 823. Not 8,230. Eight hundred and twenty-three thousand. And the number was still climbing.

This is the story of how we found it, why it happened, and the defensive patterns we’ve since adopted to make sure it never happens again.

The Alert That Didn’t Make Sense

Our SRE team monitors 5XX error rates and response time thresholds across all services. When alerts started firing for elevated latency, the initial assumption was a downstream dependency issue — maybe a database slow query or a third-party API timeout.

But the pattern was wrong. The load wasn’t distributed across endpoints the way organic traffic would be. It was concentrated — laser-focused on a single API path, with requests arriving in tight, repetitive bursts.

We pulled the Kibana logs and filtered by client fingerprint. That’s when the picture became clear: a relatively small number of mobile clients were calling the same endpoint thousands of times each, in rapid succession, all receiving the same 400 Bad Request response.

The clients weren’t giving up. They were looping.

Anatomy of an Accidental Retry Storm

Here’s what was happening under the hood:

1. The mobile app initiated a specific workflow — in our case, a driver status update flow.
2. Due to a subtle data inconsistency on the client side, the request payload was malformed.
3. The backend correctly rejected it with a 400 Bad Request.
4. The mobile app’s error-handling logic treated this as a transient failure — something that might succeed if retried.
5. There was no exponential backoff. No retry limit. No circuit breaker.
6. The app immediately retried. Got another 400. Retried again. And again. And again.

This is the software equivalent of repeatedly pushing a “Pull” door harder and harder, expecting it to eventually open.

The critical mistake wasn’t the retry itself — retries are a legitimate resilience pattern. The mistake was treating a deterministic client error (400) the same as a transient server error (503). A 400 means “your request is wrong.” Sending the exact same wrong request again will never produce a different result. It’s not bad luck. It’s bad data.

Why This Was Hard to Catch

You might wonder: how does something like this ship? The answer is that it works perfectly in every happy path and most sad paths. The loop only triggers when a very specific combination of conditions align:

• The client has stale or inconsistent local state — which happens rarely, but at scale “rarely” means “constantly somewhere.”
• The specific error code isn’t classified correctly — the retry logic lumped all non-success responses together.
• The user doesn’t kill the app — if the driver switches apps or the OS reclaims memory, the loop stops. But many drivers leave the app running in the foreground while driving.

In testing, the payload is always valid. In staging, the data is clean. It’s only in production, with thousands of drivers across varying network conditions and app states, that the edge case materializes — and when it does, it materializes hard.

The Detection Playbook

Here’s how we actually surfaced and confirmed the issue, step by step. If you’re an SRE or backend engineer, this is the part worth bookmarking.

Step 1: Anomaly Detection via APM (Dynatrace)

Our Dynatrace service-level dashboards flagged elevated response times. The key signal wasn’t the absolute latency — it was the rate of change. Response times were climbing linearly, which is characteristic of resource exhaustion from a sustained load increase, not a sudden spike.

Step 2: Traffic Pattern Analysis (Kibana)

We filtered access logs by the affected endpoint and looked at request frequency per unique client identifier. Normal traffic for this endpoint was ~2–5 requests per client per day. The anomalous clients were making hundreds of requests per minute.

The query that cracked it open:

GET /driver/status
| stats count by client_id
| where count > 1000
| sort -count

Step 3: Response Code Correlation

Every single request from the looping clients was returning 400. Zero 200s. This immediately ruled out legitimate retry behavior (where you'd expect some successes mixed in) and pointed to a deterministic failure being retried indefinitely.

Step 4: Payload Inspection

We sampled the request bodies from the top offenders. They were all identical — the same malformed payload, sent thousands of times. The client wasn’t modifying the request between retries. It was pure, mechanical repetition.

The Fix: Defense in Depth

We didn’t just fix the bug. We built layers of protection so that any similar bug in the future gets caught and contained before it can cause damage.

Layer 1: Client-Side — Classify Your Errors

The most important fix was the simplest: don’t retry 4xx errors.

function updateDriverStatus(payload) {
    response = api.post("/driver/status", payload)
    if (response.success) {
        return response
    }
    if (response.statusCode >= 400 && response.statusCode < 500) {
        // Client error. Our request is wrong.
        // Retrying won't help. Log it and stop.
        logError("Client error, not retrying", response)
        return response
    }
    if (response.statusCode >= 500) {
        // Server error. Might be transient. Retry with backoff.
        return retryWithBackoff(() => 
            api.post("/driver/status", payload),
            maxRetries: 3,
            baseDelay: 1000  // milliseconds
        )
    }
}

This single change — distinguishing between “your request is broken” and “our server is struggling” — eliminates the entire class of infinite retry loops caused by client-side data issues.

Layer 2: Client-Side — Exponential Backoff with Jitter

For the cases where retries are appropriate (5xx errors, network timeouts), we implemented exponential backoff with jitter:

function retryWithBackoff(fn, maxRetries, baseDelay) {
    for (attempt = 0; attempt < maxRetries; attempt++) {
        result = fn()
        if (result.success) return result
        delay = baseDelay * (2 ** attempt)  // 1s, 2s, 4s
        jitter = random(0, delay * 0.5)     // Add randomness
        sleep(delay + jitter)
    }
    return { error: "Max retries exceeded" }
}

The jitter is crucial. Without it, if 1,000 clients all fail at the same time, they’ll all retry at the same time — creating a thundering herd that can turn a minor blip into a full outage.

Layer 3: Server-Side — Rate Limiting per Client

We added per-client rate limiting on the affected endpoint. If a single client exceeds a reasonable request threshold (say, 10 requests per minute for an endpoint that should be called once or twice), they get a 429 Too Many Requests with a Retry-After header.

This is your safety net. Even if a future client bug introduces another retry loop, the server caps the damage.

Layer 4: Observability — Anomaly Alerts

We added a specific Dynatrace alert rule: if any single client ID exceeds N requests to the same endpoint within a 5-minute window, page the on-call SRE. This turns a potential multi-day silent degradation into a same-hour detection.

The Human Side

Here’s the part that doesn’t fit neatly into a technical blog post but matters more than any of the code above: this bug existed because two teams didn’t share observability.

The mobile team had no visibility into server-side error rates broken down by client. The backend team had no visibility into client-side retry behavior. Each team’s monitoring was correct within its own scope — the mobile app was “handling errors” (by retrying), and the backend was “rejecting bad requests” (by returning 400s). Both were doing their jobs. Neither could see the emergent behavior that arose from the interaction.

After this incident, we made three organizational changes:

1. Shared dashboards — Mobile and backend teams now have a single Dynatrace dashboard showing request volume, error rates, and retry patterns across the full client-server boundary.
2. Client behavior contracts — We documented expected retry behavior for every error code, and both teams review changes to error handling logic.
3. Chaos testing for error paths — We now deliberately inject 400s and 500s in staging to verify that client-side error handling behaves correctly under sustained failure conditions.

Key Takeaways

If you take nothing else from this post, remember these five things:

1. 4xx errors are not retryable. A 400 means your request is wrong. Sending it again won’t make it right.
2. Always cap your retries. An unbounded retry loop is an infinite loop with extra steps.
3. Exponential backoff with jitter is non-negotiable for any retry logic that targets a shared resource.
4. Server-side rate limiting is your safety net. You cannot trust every client to behave correctly. Protect your backend from your own frontend.
5. Shared observability across the client-server boundary is the only way to catch emergent failure modes that neither team can see in isolation.

The Numbers

For the curious, here’s the final accounting:

• 823,000+ redundant API calls over 4 days
• ~2,400 requests per minute at peak from looping clients
• 0 data loss or customer-facing outage (we caught it before it cascaded)
• 4 hours from first alert to root cause identification
• 2 days to ship the client-side fix to production
• 0 recurrences since implementing the defense-in-depth layers

The most dangerous bugs aren’t the ones that crash your app. They’re the ones that silently multiply until your own infrastructure buckles under the weight of your own traffic. Build your defenses before you need them.

Nikhil Ninawe is a Site Reliability Engineer who spends his days making sure backends stay standing — even when the frontends are trying their best to knock them down.