The 50/50 Trap: What I Learned by Purposefully Breaking My Blue-Green Deployment

I’ve coded the Blue-Green split. I’ve run the terraform apply. It’s 50/50. If one side dies, the other side just takes over... right?

That was my confident assumption last week while staring at my terminal. I had just finished architecting a simple cafe website — Karthik’s Cafe. I was proud of the code. I had a Blue environment (Version 1.0) and a Green environment Karthick’s Cafe & Snacks(Version 2.0) defined as distinct Target Groups behind an Application Load Balancer (ALB).

I was practicing Blue-Green Deployment using Infrastructure as Code (IaC) — the holy grail of modern, zero-downtime releases. But as I’ve learned, there is a massive difference between a successful deployment and a resilient architecture.

When you manage traffic through code, you realize very quickly that the cloud doesn’t have common sense— it only follow your configurations..

The Setup: The Perfect Deployment

I wanted to test Weighted Routing. The idea is simple:

1. Start with 100% traffic on Blue (The stable old version).
2. Shift 50% to Green (The shiny new version).
3. If Green looks good, go to 100%. If not, just roll back to blue.

I updated my ALB listener rules, hit refresh on my browser, and it was beautiful.

Refresh: Karthik’s Cafe V1. Refresh: Karthik’s Cafe V2. It worked like a charm. But then, the curious engineer in me took over. I wondered: “What happens if the Green environment completely crashes while I’m at a 50/50 split?”

The Experiment: Killing the Green Room

I went into my EC2 dashboard and stopped the web server in the Green environment. In my head, I imagined the ALB seeing the failure and heroically redirecting all traffic to the healthy Blue environment.

I was wrong.

The “Aha!” Moment (The Reality Check)

When I checked the site after killing the Green environment, I didn’t see a seamless failover. Instead, every second refresh gave me the dreaded 502 Bad Gateway.

It reminded me of that legendary Vadivelu share auto comedy. You know the one — where the angry guy is blindly punching into the auto, convinced Vadivelu is inside, while Vadivelu has already casually stepped out and is standing right behind him?

My ALB trying to hit Target Group eventhough server becomes unhealthy

Why the “Loyal Soldier” fails you:

• The ALB is a Loyal Soldier: If your routing rule says “Send 50% to Green,” the ALB will try its hardest to do exactly that. It doesn’t automatically give that weight back to Blue just because Green is struggling.
• The “Fail-Open” Trap: When all targets in a target group become unhealthy, the Application Load Balancer does not automatically fail over to other healthy groups. Instead, it continues to honor the configured routing weights and may still forward traffic to the unhealthy target group.
• The Result: 50% of my customers were getting served a broken page, even though a perfectly healthy Blue environment was sitting right next to it.

Response of my Server inside Blue Target Group

Response of my Server inside Green Target Group

The Reality: Zero Downtime vs. Automatic Failover

After seeing those 502 errors, I realized something critical about my setup.

I did achieve Zero Downtime… but only because I was sitting there, manually watching the console like a hawk. Once I saw the Green environment was unhealthy, I manually edited the listener rule to shift the weight back to 100% Blue. Boom — the errors stopped.

But Manual isn’t Cloud-Native

In a real production environment at 3:00 AM, you aren’t going to be there to move the slider. This is where most beginners (including myself) get confused.

• Weighted Routing gives you the infrastructure for zero downtime.
• Health Checks give you the status of your targets.
• Automation/Orchestration is the brain that connects the two.

It honors the weight first and checks health second.

The Lessons Learned:

Weighted routing controls how traffic is distributed across target groups, while health checks determine the availability of individual targets within a group.

These two mechanisms are independent. The load balancer does not automatically shift all traffic to a healthy environment just because you are using weighted routing.

1. How to actually fix it

If you want Automatic Failover — where the ALB sees the Green Target group is empty and immediately shifts 100% of the traffic back to Blue without you lifting a finger — you need an extra layer.

Achieving Automatic failover:

To achieve a true automatic failover, you can’t rely on ALB weights alone. You need additional mechanisms:

• AWS CodeDeploy: For managed blue-green deployments that monitor health and trigger an automatic rollback, shifting the weight back to Blue before the user notices.
• Amazon Route 53: Using health check–based routing at the DNS level.
• Event-Driven Automation: Using a Lambda function triggered by CloudWatch Alarms to automatically update your Terraform/CloudFormation stack or ALB weights when 5XX errors spike

ALB without Route 53, CodeDeploy, Monitoring, Lambda triggers

2. Observability is King

If I hadn’t been looking at the DNS in a loop, I might have missed the 502 errors.

Always monitor your HTTPCode_Target_5XX_Count in CloudWatch.

Deployment is only half the job; monitoring is the other half.

Final Thoughts

My experiment taught me that Zero Downtime is a journey, not a checkbox. I started with a manual weight shift and ended with a deep understanding of why automation is non-negotiable. If your fail-safe system requires you to be awake and clicking buttons, it’s not a fail-safe — it’s a high-stakes video game.

So, the next time you’re building a fail-safe system, do one favor: Break it. Watch it fail. Automate the fix. You’ll be a much better architect for it.