⚠️ The Kubernetes Setting That Made Deploys Risky

⚠️ The Kubernetes Setting That Made Deploys Risky

(Spring Boot Production Reality)

“Nothing changed in the code. Deploy still caused errors.”

We didn’t break production. Kubernetes did — with a default.

🧩 The innocent-looking setting

Here it is. The silent troublemaker:

maxUnavailable: 25%

It lives inside:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 25%
    maxSurge: 25%

Most teams:

• Never touch it
• Assume it’s safe
• Deploy on Friday anyway

We did too.

😌 What teams think it means

“Kubernetes will take down a few pods at a time. Traffic will shift smoothly. No downtime.”

That assumption is what makes this dangerous.

💥 What actually happened

We had:

• 8 Spring Boot pods
• Moderate traffic
• Healthy dashboards
• Confident deploy

Kubernetes calculated:

25% of 8 = 2 pods

So during deploy:

• 2 pods terminated immediately
• Before new ones were ready

Capacity dropped by 25% instantly.

🧨 The cascading failure

Here’s the chain reaction:

1️⃣ Two pods receive SIGTERM 2️⃣ Traffic redistributes to remaining pods 3️⃣ Latency increases 4️⃣ Threads block 5️⃣ Timeouts occur 6️⃣ Clients retry 7️⃣ Load spikes 8️⃣ Remaining pods overload 9️⃣ Readiness fails 🔁 More pods removed

Deploy → traffic storm → outage

🧠 Why this is worse for Spring Boot

Spring Boot apps often have:

• Fixed thread pools
• Fixed DB pools
• Blocking I/O
• Retry logic

They do not degrade gracefully under sudden load loss.

A 25% capacity drop ≠ 25% latency increase It’s often exponential.

🔍 The killer detail nobody notices

Kubernetes does not wait for:

• New pods to be ready
• Traffic to stabilize
• GC warmup
• Connection pools to fill

It only respects:

maxUnavailable

If that number allows pod removal — it removes them.

🧠 The math that fooled us

With 4 pods:

25% = 1 pod

With 3 pods:

25% rounds up → still 1 pod

With 2 pods:

25% = 0.5 → rounds up → 1 pod

Meaning:

Kubernetes can always remove at least one pod.

Even when you can’t afford it.

💣 Why this hits during peak traffic

Deploys often happen:

• During business hours
• With active users
• With background jobs running

Capacity loss + peak traffic = instant saturation.

🚨 What made it worse

• Readiness probes took time to fail
• Load balancer kept routing traffic
• Retry storms amplified load
• Autoscaling lagged behind reality

All because one pod disappeared too early.

✅ The safer deployment strategy

1️⃣ Set maxUnavailable to zero

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

This guarantees:

• No pod is removed
• Until a new one is ready

2️⃣ Accept slower deploys

Yes, deploys take longer.

That’s the cost of:

• Stability
• Predictability
• Sleeping at night

Fast deploys are useless if they hurt users.

3️⃣ Combine with fast readiness failure

@EventListener(ContextClosedEvent.class)
public void onShutdown() {
    shuttingDown.set(true);
}

Stop traffic before pod removal.

4️⃣ Test deploys under load

Most load tests:

• Never include deploys
• Never simulate capacity loss

That’s where reality lives.

🧾 The rule we now follow

Capacity must increase before it decreases. Never the other way around.

If Kubernetes is allowed to remove pods early, it eventually will — at the worst moment.

❤️ Final truth

Most deploy outages aren’t caused by bugs. They’re caused by orchestration math.

maxUnavailable isn’t a tuning knob. It’s a blast radius setting.