💥 The Cache Miss That Took Down Our DB

💥 The Cache Miss That Took Down Our DB

(Spring Boot Production Reality)

“Cache is supposed to reduce load.” So why did one cache miss create the worst database incident of our year?

Because caching doesn’t just hide latency.

It also hides pressure.

And when the cache fails… the database pays for everything at once.

😌 The day everything looked “fine”

We had:

• Spring Boot services
• PostgreSQL
• Redis
• @Cacheable
• Normal traffic
• No code changes

And then suddenly:

• DB CPU hit 95%
• Connection pool exhausted
• P95 latency exploded
• Timeouts everywhere
• Retries kicked in
• Kafka lag started climbing

And the trigger?

A cache miss.

Not cache down. Not Redis outage. Just a miss.

🧩 The cache we thought was saving us

We had this:

@Cacheable(cacheNames = "product", key = "#id")
public Product getProduct(String id) {
    return productRepository.findById(id)
            .orElseThrow();
}

Redis held millions of keys.

The DB was calm.

Everything felt safe.

💥 What actually happened

A product import job ran.

It updated a huge portion of product records.

Which triggered this:

@CacheEvict(cacheNames = "product", key = "#id")
public void updateProduct(String id, ProductUpdate update) {
    ...
}

We evicted a lot of keys.

And that’s where the trap started.

☠️ The killer effect: cache stampede

The moment those keys were evicted:

• Thousands of requests hit the service
• All of them missed cache
• All of them hit the DB
• At the same time

This is called a:

Cache Stampede

And it’s basically a synchronized DDoS… from your own users.

🧠 Why it took down the database

Because the DB wasn’t sized for “real traffic”.

It was sized for:

traffic AFTER caching

Meaning:

• Cache reduced DB reads by 80–95%
• DB ran cool and happy
• DB capacity was never tested at full load

So when cache missed:

The DB suddenly saw the real world.

And it couldn’t handle it.

🔥 The second punch: connection pool exhaustion

This is what happened next:

• Requests hit DB
• DB slowed down
• Threads waited longer
• More threads piled up
• Hikari pool filled

Now even simple queries couldn’t run.

Everything stalled.

🧨 The third punch: retries

Clients started retrying:

RetryTemplate.builder()
    .maxAttempts(3)
    .fixedBackoff(100)
    .build();

So one slow request became three.

That turned the stampede into a traffic storm.

😵‍💫 Why the cache miss was worse than cache being down

Here’s the brutal truth:

If Redis is DOWN:

• You notice instantly
• Alerts fire
• Failover triggers
• Teams respond fast

If Redis is UP but missing keys:

• Health checks stay green
• Monitoring looks normal
• The DB slowly dies

Cache misses are stealth outages.

🧠 The most dangerous caching illusion

Caching makes your system look like it scales.

But often it just means:

Your database is under-tested.

✅ How we fixed it (Spring Boot + Redis + production-safe)

1️⃣ Use “cache stampede protection” (locking)

You want:

• Only 1 request loads from DB
• Others wait briefly
• Or return stale data

Example using Redis lock:

public Product getProductSafe(String id) {
    String lockKey = "lock:product:" + id;
    boolean locked = redisTemplate.opsForValue()
            .setIfAbsent(lockKey, "1", Duration.ofSeconds(3));
if (Boolean.TRUE.equals(locked)) {
        try {
            Product p = loadFromDb(id);
            cachePut(id, p);
            return p;
        } finally {
            redisTemplate.delete(lockKey);
        }
    }
    // Another request is loading it
    Product cached = cacheGet(id);
    if (cached != null) return cached;
    // last fallback: wait briefly or fail fast
    throw new IllegalStateException("Product warming, retry later");
}

This prevents 1,000 misses from becoming 1,000 DB queries.

2️⃣ Use TTL + jitter (avoid synchronized expiration)

Bad TTL:

All keys expire at the same second

Good TTL:

Keys expire gradually over time

Example:

Duration ttl = Duration.ofMinutes(10)
        .plusSeconds(ThreadLocalRandom.current().nextInt(30));

That random 0–30 seconds prevents mass expiration.

3️⃣ Never mass-evict without rate limiting

If you do this:

@CacheEvict(cacheNames = "product", allEntries = true)
public void refreshAll() {}

You are basically saying:

“Dear database, good luck.”

Instead:

• Evict gradually
• Refresh asynchronously
• Use background warming

4️⃣ Add a DB safety fuse (bulkhead)

If DB is overloaded:

• stop accepting more load
• fail fast
• protect the DB

Example with a semaphore:

@Component
public class DbBulkhead {
private final Semaphore semaphore = new Semaphore(50);
    public <T> T execute(Supplier<T> supplier) {
        boolean acquired = semaphore.tryAcquire();
        if (!acquired) {
            throw new RuntimeException("DB overloaded, try later");
        }
        try {
            return supplier.get();
        } finally {
            semaphore.release();
        }
    }
}

Then wrap DB calls:

public Product loadFromDb(String id) {
    return dbBulkhead.execute(() ->
            productRepository.findById(id).orElseThrow()
    );
}

This keeps your DB alive.

5️⃣ Prefer “stale-while-revalidate”

For read-heavy systems, the best strategy is:

Serve stale data for a short window while refreshing in background.

Because:

• Stale is better than down
• Most data isn’t life-or-death accurate

🧾 The rule we now live by

A cache is not a performance feature. It’s a load-bearing component.

Treat it like production infrastructure.

Because when it fails:

• Your DB becomes the fallback
• Your DB becomes the victim
• Your system becomes unstable

❤️ Final truth

The biggest risk in caching is not wrong data.

It’s sudden truth.

When the cache stops hiding load, your database sees reality — all at once.