AWS Outage: Root Cause Analysis

October 19–20, 2025 | US-EAST-1 Region | Duration: 14h 32m

Executive Summary

On October 19–20, 2025, AWS US-EAST-1 experienced a 14+ hour service disruption, affecting thousands of customers worldwide. The outage originated from a subtle DNS race condition in DynamoDB’s internal management system, which cascaded across core services such as EC2, Lambda, NLB, ECS/EKS, and dozens of downstream services. During the peak, Lambda invocation failures spiked significantly, and EC2 control-plane requests were delayed, illustrating how even cloud giants face systemic fragility.

This RCA examines the technical root cause, cascading failure patterns, and lessons learned for building resilient distributed systems.

Before You Read: Technical Context

If you’re not deeply familiar with AWS’s internal orchestration layers, here’s a quick primer to help make sense of what follows.

• DWFM (Distributed Workflow Manager): AWS’s internal orchestration system that coordinates configuration updates across regions. You can think of it like a massively scaled version of Step Functions or Airflow, used to automate tasks that keep AWS services in sync.
• DNS Management Layer: The control plane that handles updates to DNS records for internal AWS services. It ensures that all availability zones have consistent mappings between service names and endpoints.
• Planner and Enactor Workers: Two cooperating worker types within DWFM. The Planner decides what changes need to happen, and the Enactor applies them. Synchronization between the two is critical; a version mismatch here triggered the outage.
• Version Control Mechanism: Each configuration update carries a version number to prevent stale writes. In this outage, a subtle race condition caused an older version to overwrite a newer one, leading to DNS inconsistencies.
• Change Propagation Pipeline: Updates are rolled out gradually (region → AZ → service) to minimize blast radius. The race condition disrupted this controlled rollout.
• Health Check and Rollback Logic: DWFM verifies each update before marking it healthy. When mismatched versions failed validation, automated rollbacks reintroduced the stale state, worsening the impact.
• Dependency Graph: Higher-level AWS services like EC2, Lambda, and DynamoDB depend on this

🎓 Grab Your Free AWS Certification Roadmap! Get curated tips, exam strategies, and structured learning paths in the AWS Certification Handbook. (https://www.awscertificationhandbook.com/)

Timeline of Events

Events Timeline

Cascading Domino Effect

Analogy: The Domino Room

Imagine a control room where two engineers, a Planner and an Enactor who manage a massive wall of switches representing DNS entries.

Planner decides which switches to flip, while Enactor physically flips them. One day, both move at the same time.

Planner sends “turn on switch A,” but Enactor, reading outdated instructions, turns off switch A instead.

Lights across the building flicker, and some go dark and others stutter.

That’s what happened inside DynamoDB’s DNS engine. A brief coordination glitch between Planner and Enactor caused system-wide confusion, deleting valid DNS records and disrupting everything that relied on them.

A small timing mismatch became a building-wide blackout.

The Root Cause

The disruption stemmed from a race condition in DynamoDB’s DNS management layer. The system uses two main components:

• DNS Planner — Generates endpoint update plans.
• DNS Enactor — Applies these plans independently across three Availability Zones (AZs).

The bug appeared when one enactor lagged behind another. The older process overwrote a newer configuration and triggered cleanup automation that deleted the valid DNS entry, resulting in an empty record for the regional endpoint.

Essentially, a stale check allowed outdated data to win a write race — a subtle but catastrophic condition in distributed coordination.

How the Bug Happened (Simple Version)

1. Worker #1 starts a slow task Worker #1 picks up “Version 100” and starts updating 1,000 servers. It’s slow — hours instead of minutes. → (Still processing Version 100…)

2. Worker #2 finishes a newer version quickly Meanwhile, the system generates “Version 102.” Worker #2 completes it fast. → (Version 102 complete!)

3. Worker #2 cleans up old versions Worker #2 removes “old” versions like 100 and 101. → (Version 100 deleted.)

4. Worker #1 finally finishes — DISASTER Worker #1, unaware of cleanup, finishes and writes “Version 100” back. → (Version 100 now points to nothing = empty DNS = outage.)

This sequence mirrors a stale state overwrite, a common but hard-to-detect bug in parallel coordination systems.

Key Technical Insight

The stale check problem: The DNS Enactor verified plan freshness once at the start of processing, but by the time it finished updating endpoints (after unusual delays), that check was stale. This allowed an old plan to overwrite a newer one, which was then deleted by cleanup automation.

AWS Response & Mitigation

Immediate Actions

• Manual DNS restoration completed within 2h 37m
• DWFM (Distributed Workload Flow Manager) selective restarts
• NLB health check failover temporarily disabled
• Lambda ESM throttled to reduce operational load

Recovery Strategy

• Cleared DWFM queues via controlled restarts
• Throttled and gradually restored incoming workload rates
• Prioritized backlog processing for dependent services

Full Recovery Time

• DynamoDB: 2h 52m
• EC2: 14h 2m
• NLB: 8h 39m
• Lambda: 14h 24m

AWS Remediation Plan

Completed

✓ DynamoDB DNS automation disabled worldwide

✓ Post-mortem and technical analysis completed

In Progress

• Fix DNS race condition scenario
• Add protections against incorrect DNS plans
• NLB velocity controls for AZ failover
• EC2 DWFM recovery workflow testing
• Network Manager queue-based rate limiting

📘Prefer Hands-On Learning? Check out these interactive courses tailored to your goals:

AWS Certified Cloud Practitioner — Beginner (https://www.educative.io/courses/aws-certified-cloud-practitioner-exam) AWS Solutions Architect Associate — Intermediate (https://www.educative.io/courses/aws-solutions-architect-associate)

Lessons Learned

• Race Conditions: Even multi-AZ systems can fail due to stale checks under specific load patterns.
• Congestive Collapse: DWFM retries can spiral without queue limits and circuit breakers.
• Cascading Dependencies: One failure (DynamoDB DNS) can propagate across EC2, Lambda, NLB, and more.
• Regional Dependencies: Redshift’s IAM API had a hard dependency on US-EAST-1, triggering a worldwide effect. Cross-region dependencies should be minimized, with explicit fallbacks or failovers where unavoidable.
• Manual Intervention: Automation alone may fail; systems need auto-remediation or fail-safe mechanisms.

Why This Matters for Engineers

For Platform Engineers: This incident demonstrates that DNS, often treated as “solved,” remains a critical failure domain. Even with multi-AZ redundancy, automation race conditions can create single points of failure.

For SREs: Congestive collapse is not theoretical. When recovery mechanisms can’t complete before timing out, systems enter death spirals. Design for graceful degradation and implement aggressive circuit breakers.

For Architects: Cross-region and cross-service dependencies should be mapped explicitly. A “regional” failure in us-east-1 became global because of hidden dependency chains.

For All Engineers: Even AWS, with world-class engineering, extensive testing, and battle-hardened systems, can experience cascading failures from subtle race conditions. Defense in depth, observability, and chaos engineering are not optional.

Looking Beyond the RCA: While understanding root causes is critical, resilience is as much about strategy as it is about uptime. How organizations prioritize risk, allocate resources, and make trade-offs around rare outages is explored in detail in my follow-up article: Resilience Is About Strategy, Not Uptime

References / Links:

AWS Official Outage Message