Designing for failure in AWS: EKS + serverless resilience in practice

On a recent workload, we rebuilt a critical service using AWS EKS for core processing and serverless (Lambda + EventBridge) for buffering, retries, and brittle downstream integrations. Goal: survive node failures, AZ degradation, and regional issues with minimal impact.

Architecture at a glance

• EKS cluster with node groups across 3 AZs.
• Pods using anti-affinity and topology spread to avoid co-locating replicas.
• HPA + Cluster Autoscaler + Karpenter for fast scaling and node replacement.
• Multi-AZ ALB with health checks.
• Lambda for bursty/failure-prone endpoints.
• EventBridge queues with retries and DLQs.
• Circuit breakers in Lambda to stop calling degraded services.

Failure patterns we designed for

• Node failure: pods rescheduled automatically; topology spread avoids single- AZ/node risk.
• AZ degradation: multi-AZ ALB + pod spread keeps traffic flowing.
• Downstream failures: retries, backoff, DLQs, and circuit breakers limit blast radius.
• Regional issues: cross-region EKS hot standby + Route 53 failover + Aurora Global DB.

Measurable impact

• Uptime: 99.99% (up from ~99.5%).
• MTTR for node failures: < 2 minutes, fully automated.
• Zero customer-visible outages during AZ degradation.
• Downstream-induced errors down ~60% after adding circuit breakers and DLQs.

Key takeaways

• Assume failures will happen: nodes, AZs, and downstream services.
• Spread risk with multi-AZ, topology constraints, and autoscaling.
• Use serverless as a resilience layer: buffering, retries, circuit breakers.
• Automate recovery so most failures are self-healing.
• Measure uptime, MTTR, and error sources.

Resilience isn’t preventing failures; it’s designing systems that keep working when they happen.