The Day the Internet Went Dark: Inside the 2017 Amazon S3 Outage

How a single typo in a maintenance script took down a global cloud platform — and changed how the world thinks about resilience engineering.

On the morning of February 28, 2017, millions of people sat down at their computers and found that the internet was broken. Slack was sluggish. Trello was down. Quora, Pinterest, Imgur — all showing errors. Even Amazon’s own S3 service health dashboard couldn’t load, because it too was hosted on S3.

The culprit was not a cyberattack, not a hardware failure, not a months-long software bug. It was four hours of global disruption caused by a single mistyped integer in a maintenance script, run by one engineer, on one Tuesday morning.

“We’ve identified the root cause of increased error rates in Amazon S3 US-EAST-1. Human error during a debugging exercise.” — AWS Post-Incident Report

S3 Architecture — How It Normally Works

To understand the failure, you first need to understand what S3’s internal architecture looks like under the hood. S3 is not a single database — it’s a layered system of interdependent subsystems, each responsible for a specific job.

Client / AppAPI call

Fig 1 — S3’s internal architecture under normal operating conditions. Both INDEX and PLACEMENT must be healthy for any request to succeed.

Every single S3 request — whether a PUT, GET, or LIST — passes through the API Gateway and then depends on both subsystems working in tandem. INDEX tells the system where an object lives. PLACEMENT tells it where to put new objects. Remove either, and S3 cannot function.

What Happened

On February 28, 2017, an engineer was running a standard debugging playbook for a billing system performance issue. The playbook called for removing a small subset of servers from the S3 subsystem to observe behaviour under reduced load — a routine capacity adjustment.

The engineer typed a parameter. The parameter was supposed to be small. It wasn’t. Due to a typo, it specified a much larger number — removing a significant portion of both the INDEX and PLACEMENT subsystems simultaneously.

The Failure Cascade — How One Typo Became a Global Outage

The failure didn’t stop at S3. Because US-EAST-1 is the oldest AWS region — and the one most internal AWS services defaulted to — the cascade spread rapidly outward.

Fig 2 — The failure cascade. One mistyped command created two simultaneous crises: immediate S3 failures and a slow, unavoidable cold restart. Both paths led to the same 4-hour outage.

Notice the ironic side branch: the AWS service health dashboard — the page millions of customers relied on to check if AWS was down — was itself hosted on S3 in US-EAST-1. When S3 went dark, so did the status page. Engineers had to rush to stand up a plain HTML page just to communicate the incident.

Timeline of the Outage

Here’s a minute-by-minute breakdown of how the morning unfolded, from the first erroneous command to full recovery.

Fig 3 — Recovery timeline. The root cause was found quickly (~23 min), but the cold restart of two large subsystems took nearly 3 additional hours.

Why It Happened — The Deep Causes

1. No rate limiting on critical subsystem changes

The maintenance tool had no minimum-capacity floors. It would remove any number of servers — including all of them — in a single command execution. No confirmation prompt, no ceiling, no guard.

2. Cold restart time had never been measured at scale

The INDEX and PLACEMENT subsystems had been running continuously for years. Nobody had recently benchmarked — or rehearsed — how long a full cold restart would take at 2017 S3 scale. The answer turned out to be hours, not minutes. That latent fact was invisible until it became a crisis.

3. The blast radius of US-EAST-1

US-EAST-1 is AWS’s original region. Over the years, a disproportionate number of AWS-native services quietly defaulted to it. When S3 there failed, the blast radius included CloudFront, EC2 API endpoints, Lambda console, and yes — the health dashboard itself.

4. The status page problem

The AWS service health dashboard was hosted on S3 in US-EAST-1. This is the engineering equivalent of keeping your fire extinguisher inside the room that’s on fire. Engineers had to scramble to stand up a plain HTML page to communicate the incident while it was still happening.

How It Was Solved

The recovery was simple in principle: restart both subsystems. The pain was in execution. At S3’s scale, the INDEX subsystem needed to rebuild its internal state — effectively re-indexing metadata for billions of stored objects — before it could serve requests reliably. This is not a process you can speed up. You wait.

While engineers waited, the AWS team worked in parallel: communicating with customers, mitigating impact where CDN caching could help, and documenting the incident in real time to accelerate the post-mortem.

By 1:18 PM, all subsystems were restored and S3 error rates returned to normal.

What Amazon Changed After

• Minimum capacity floors enforced in maintenance scripts — physically impossible to remove more servers than the system can afford to lose
• Slow, health-checked ramp-downs — server removals now happen gradually with automatic halt if error rates spike
• Restart time drills — regular exercises to measure and document cold restart durations at current scale
• Independent status infrastructure — the health dashboard migrated off S3 US-EAST-1 to an isolated, multi-region setup

Key Learnings

Final Thoughts

The 2017 S3 outage is a masterclass in how complex systems fail. It wasn’t negligence, it wasn’t a bad engineer. It was a normally functioning system that had quietly accumulated invisible fragility — a missing guardrail here, an untested restart path there, a status page in the wrong place — until one Tuesday morning, a single mistyped number found every crack at once.

“Resilience is not about preventing failure. It’s about designing systems that fail gracefully and recover quickly.” — Werner Vogels, Amazon CTO

The next time you write a script that touches production infrastructure, ask yourself: if someone entered the largest possible number here, what happens? That question — simple as it sounds — might be the most important guardrail you ever add.