💥 The Day a Single YAML Change Took Down Our Spring Boot Cluster

💥 The Day a Single YAML Change Took Down Our Spring Boot Cluster

“It’s just a small config tweak. What’s the worst that could happen?” Famous last words.

This is the story of how one innocent line in application.yml brought down our entire Spring Boot cluster in under 7 minutes.

No code changes. No bad deploy. Just YAML.

🧾 The system

We were running:

• ~20 Spring Boot microservices
• Kubernetes (EKS)
• Java 21 + virtual threads
• PostgreSQL + HikariCP
• Kafka for async
• ~25k RPS peak traffic

Solid. Stable. Or so we thought.

✏️ The change that started it all

A teammate wanted to “optimize” DB usage.

PR diff:

spring:
  datasource:
    hikari:
-     maximum-pool-size: 20
+     maximum-pool-size: 100

That’s it.

The idea:

“More connections = more throughput.”

The PR sailed through. No alarms. No one expected trouble.

⏱️ Minute 0–2: Looks fine

Pods rolled out. Health checks green. Dashboards normal.

We even congratulated ourselves:

“Nice, DB waits are gone!”

⏱️ Minute 3–5: Latency creeps in

p95 latency: 📈 200 ms → 800 ms → 2 s

DB CPU: 📈 40% → 85%

Connections: 📈 climbing fast.

But autoscaling hid it. More pods spun up.

We thought:

“Just traffic.”

💥 Minute 6–7: Everything collapsed

Suddenly:

❌ DB connections maxed out ❌ Queries timing out ❌ Pods stuck waiting for connections ❌ Kafka consumers lagging ❌ API gateway returning 503

Then: ☠️ Read replicas fell behind ☠️ Primary started rejecting new conns ☠️ Health checks failed → pods restarted

In less than a minute:

The whole cluster was down.

All from one line.

😱 What actually happened?

Before:

• 20 pods × pool size 20 = 400 max connections

After:

• 20 pods × pool size 100 = 2000 connections

Our Postgres max: ➡️ 500 connections.

So we accidentally told:

20 services to DDOS our own database.

Each pod:

• Opened more connections
• Held them in transactions
• Starved others
• Caused timeouts
• Triggered retries
• Multiplied load

➡️ A self-inflicted denial of service.

🔄 The retry death spiral

To make it worse:

• Timeouts triggered retries
• Retries opened more connections
• More connections slowed DB further
• Slower DB caused more timeouts

A perfect feedback loop.

YAML + retries = outage amplifier.

🛑 The fix (and the panic)

At minute 9:

• We rolled back
• Restarted pods
• Manually killed idle connections
• Paused consumers
• Prayed

By minute 20: 🟢 Services slowly recovered.

Total impact:

• ~15 minutes downtime
• Thousands of failed requests
• A very quiet Slack channel

🧠 The lessons that hurt the most

1️⃣ Config scales multiplicatively

Per-pod limits × number of pods = real impact.

Never change:

• DB pools
• Thread pools
• HTTP pools

Without calculating:

new_limit × max_pods

2️⃣ Databases hate “more”

DBs prefer: ✔️ fewer, well-used connections ❌ not thousands of idle ones

More connections ≠ more throughput. Often it means more context switching and locks.

3️⃣ YAML is production code

That one line had:

• No tests
• No canary
• No guardrails

Yet it had:

More impact than most code changes.

Treat config as: ➡️ code with blast radius.

4️⃣ Autoscaling can hide disasters

HPA saw latency: 📈 → added pods.

Each new pod: ➡️ added 100 more DB connections.

Autoscaling didn’t save us. It made it worse.

5️⃣ We had no connection budget

We never defined:

“How many DB connections can this system afford?”

So YAML guessed for us.

Badly.

🛡️ What we changed after

Now, every service has:

spring.datasource.hikari.maximum-pool-size: 15

And cluster-wide:

• DB connection budget documented
• Per-service quotas
• Alerts on:
• active connections
• wait time
• Canary deploys for config
• PR checklist for “blast radius”

We also added:

spring.datasource.hikari.connection-timeout: 2s

Fail fast > fail cluster.

📏 A simple safe formula

If DB max connections = 500 And worst-case pods = 20

Then:

pool_size ≤ 500 / 20 = 25

And keep headroom.

Never let YAML do this math for you.

❤️ Final thought

The most dangerous changes in production aren’t complex refactors. They’re “tiny safe tweaks.”

Because no one is scared enough to double-check them.

That day taught us:

A single YAML line can be more powerful than a thousand lines of Java.

Respect it.

If you enjoyed this story, share it with your team — especially the one who says “It’s just config.” 😄