How a Single Cache Miss Took Down Our Entire Production System

One missing cache key. Four hours of total outage.

A war story about thundering herds, cascading failures, and the 15 seconds that broke everything.

It was a Wednesday afternoon when the alerts started firing. Our production API — a Ruby on Rails application serving thousands of clients — had gone completely unresponsive. Every single endpoint was returning errors. Not just the heavy ones. Even /robots.txt was timing out.

What followed was a 4-hour outage, two failed recovery attempts, and a deep dive into a cascading failure that started with something deceptively simple: an empty cache key in Redis.

The Architecture

Our system is a relatively straightforward Rails stack:

• Unicorn as the application server (process-based, pre-forking)
• Nginx as the reverse proxy
• PostgreSQL for the database
• Redis (AWS ElastiCache) for caching
• AWS ALB as the load balancer
• Sidekiq for background jobs

One particular API endpoint — let’s call it /catalog — was responsible for returning a large JSON payload containing the entire dependency graph of our platform. Think of it like a package registry's resolver endpoint: every package, every version, every dependency constraint, all in one response.

This endpoint accounted for 70% of all traffic to our application.

The Design

The /catalog endpoint was backed by a raw SQL query joining three tables — packages, versions, and dependencies — across the entire dataset. The original developers had chosen raw SQL over ActiveRecord for performance (roughly 20x faster), and the result was cached in Redis with no TTL.

class CatalogCache
  def fetch(&block)
    Rails.cache.fetch(cache_key, &block)
  end

  def self.flush
    Rails.cache.delete("http-catalog")
    Rails.cache.delete("https-catalog")
  end
end

The cache was invalidated (flushed) every time a package was uploaded, deleted, or a version was removed. Simple, correct, and — as we discovered — catastrophically fragile at scale.

The 15-Second Death Sentence

Our Unicorn configuration had worker_timeout set to 15 seconds. This is a watchdog: if a worker takes longer than 15 seconds to respond, the master process kills it and spawns a replacement. It's a safety mechanism to prevent hung workers from consuming resources permanently.

Here’s the thing: when our dataset was small, the /catalog query completed in 3-5 seconds. Plenty of headroom. But datasets grow. Slowly, invisibly, over months and years. Nobody noticed when the query crept past 10 seconds. Nobody noticed when it hit 12.

Until the day it crossed 15.

The Cascade

Here’s what happened, reconstructed from logs, metrics, and a lot of caffeine:

T+0: The cache key disappeared from Redis. (We never confirmed whether this was triggered by a package upload calling flush, or by ElastiCache evicting the key under memory pressure.)

T+1 second: Multiple concurrent requests hit /catalog, all found an empty cache, and all began executing the expensive SQL query simultaneously. This is the classic thundering herd problem — Rails.cache.fetch offers no lock mechanism to prevent multiple processes from regenerating the same key.

T+15 seconds: Unicorn’s watchdog killed every worker that was running the query. The cache remained empty. The query never completed.

T+16 seconds: Fresh workers spawned. New /catalog requests immediately arrived. The same query started again. Killed again at 15 seconds. An infinite loop.

T+30 seconds: Each killed-and-respawned worker opened a new connection to Redis. Connection count began climbing.

T+5 minutes: Redis connections reached ~400. Under this connection load, Redis read latency degraded from sub-millisecond to 15–20 seconds. Now even endpoints that did have cached data couldn’t read it within the worker timeout.

T+10 minutes: The entire application was unresponsive. Every endpoint. The ALB’s 60-second idle timeout was closing connections, and Nginx logged them as HTTP 499 (client closed connection before server responded).

T+10 minutes onward: Our API clients have a built-in retry policy — 5 retries with a 5-second interval. This amplified request volume by ~1.6x. The ALB’s active connection count spiked from a baseline of 14K to 55K.

The system could not self-recover.

The Failed Fix

Our ops team, seeing the connection spike and specific high-traffic IPs, suspected a DDoS attack. They:

1. Blocked the top 2 IP addresses at the WAF
2. Restarted the application

It didn’t work. The issue recurred within minutes.

Why? Because the traffic was legitimate. Those “attacking” IPs were just our own API clients retrying failed requests. Blocking them reduced load marginally, but the remaining clients were more than sufficient to trigger the same cascade. And the restart didn’t help because Redis was already empty and still overloaded with stale connections.

The Actual Fix

What ultimately worked was a scorched-earth reset of the entire request path:

1. Deregister all instances from the ALB — stop all incoming traffic
2. Stop the application on all instances — kill all Unicorn workers and their Redis connections
3. Reboot ElastiCache — clear the ~400 accumulated connections and reset Redis
4. Restart the application — fresh workers, no request backlog
5. Re-register instances to the ALB — traffic resumes; the first /catalog request regenerates the cache without contention

The critical insight: you must isolate from traffic before restarting. A restart under load is not a restart — it’s just spawning fresh victims.

The Five Failures

Looking back, this outage wasn’t caused by one thing. It was the intersection of five design decisions that were each individually reasonable:

1. No Cache Lock

Rails.cache.fetch with a block doesn't prevent concurrent processes from all entering the block simultaneously when the key is missing. With a single expensive key accessed by hundreds of concurrent workers, this is a ticking time bomb.

2. An Unbounded Query

The SQL query had no pagination, no limit, no timeout. It scanned the entire dataset every time. As the data grew, execution time grew — silently, invisibly.

3. A Tight Worker Timeout

15 seconds is a good default for web requests. But it created a hard ceiling that the /catalog query could never exceed, even if it was "almost done" at 14.9 seconds. The query was killed every time, the cache could never warm, and the workers died trying.

4. No Connection Pooling for Redis

Each Unicorn worker maintained its own Redis connection. When workers were rapidly killed and respawned, each new process opened a fresh connection without the old one being cleanly closed. This caused connection accumulation that degraded Redis itself.

5. Aggressive Cache Invalidation

Flushing the entire cached response on every single package upload meant that even routine operations — a user publishing a new version — could trigger a cascading failure during peak traffic.

What We’re Doing About It

ProblemSolutionThundering herdImplement single-flight cache regeneration — only one worker rebuilds, others waitSilent query growthMonitor query execution time; alert when approaching worker timeoutWorker timeout ceilingMove cache regeneration to a Sidekiq background job not subject to web timeoutsRedis connection stormsAdd connection pooling; limit max connections per instanceAggressive invalidationMove to background cache warming on flush instead of synchronous regenerationNo backpressure signalReturn 503 with Retry-After header so clients back off gracefully

Lessons for Your Systems

1. Your cache is a single point of failure.

If your most expensive operation is cached, ask yourself: what happens when that cache disappears? If the answer is “every worker tries to rebuild it simultaneously,” you have a thundering herd problem waiting to happen.

2. Timeouts can create irrecoverable states.

A worker timeout is meant to protect your system. But if the timeout is shorter than the time needed to regenerate critical cached data, it creates an irrecoverable loop where the cache can never be populated through normal request flow.

3. Connection management is infrastructure.

We tend to think of Redis as “just a cache” — fast, reliable, invisible. But connections are a resource. Uncontrolled connection growth turns a compute problem into an infrastructure problem that takes down your cache layer entirely.

4. Retry policies amplify failures.

Every client retry during an outage is additional load on an already-saturated system. Without backpressure signals (503, Retry-After), well-intentioned retry logic becomes a force multiplier for the outage.

5. Never restart under load.

If your application is drowning in traffic and you restart it, the new processes will immediately start drowning too. Isolate from traffic first, stabilize, then reconnect. A restart is not a recovery strategy — it’s just a fresh start for the same failure mode.

The Humbling Part

None of this was exotic. No novel attack vector, no hardware failure, no cosmic ray flipping a bit. It was a cache key disappearing from Redis — something that happens routinely — combined with a dataset that grew 0.1% larger every week until it crossed an invisible threshold.

The system worked perfectly for years. Right up until it didn’t.

If you’ve dealt with similar cascading failures in your systems, I’d love to hear your war stories. And if you haven’t — go check your most expensive cached endpoint. What happens when that key disappears?