Eleven nines of durability. I finally looked up what it actually means.

Eleven nines of durability. I finally looked up what it actually means.

I spent almost four years writing backend code that talked to DynamoDB and S3 every single day. Device states, user configs, firmware updates for hundreds of thousands of IoT devices, all of it sitting in AWS, never losing data, never going down.

At some point, I must have read the words “99.999999999% durability” in the AWS docs and moved on. Just another number in a spec sheet.

Then one day I actually stopped and asked: What does that number mean? And more importantly, how does AWS pull it off?

This is my attempt to answer that question in plain terms. No prior experience needed.

What does 99.999999999% durability actually mean?

Let’s make the number concrete.

99.999999999% durability means that if you stored 1 billion files in S3, you’d expect to lose less than one per year. Not per day. Per year.

A billion files. One loss.

That’s not luck. That’s a lot of careful engineering. And once I started pulling on that thread, I found it went surprisingly deep.

Your data is not in one place

First thing to understand: when you save something to DynamoDB or S3, it doesn’t go to one hard drive somewhere in Virginia. It gets copied to multiple physical data centers automatically.

AWS calls these data centers Availability Zones, or AZs. Each AZ is a separate building with its own power supply, its own cooling systems, its own internet connections. A single AWS region like US-East-1 has three or more of these AZs spread across different parts of a city or area.

Why does this matter?

Because hardware fails. Servers crash. Data centers flood. Power goes out. If your data only lived in one building and that building lost power tonight, your data would be gone. But if three separate buildings each hold a copy, you’d need all three to fail at the same time. That’s much harder to pull off by accident.

Keeping identical copies in multiple places is called replication. It’s the foundation of how AWS gets anywhere near that 11-nines number.

Copies create a new problem

Replication sounds simple. Copy the data to three places and you’re done, right?

Not quite.

When you write something to DynamoDB, AWS needs to push that write to all three AZs. But copying takes time. Even a few milliseconds. So for a brief window, AZ-1 might have your new data while AZ-2 and AZ-3 are still catching up.

What happens if someone reads your data during that window? They might get the old version.

This is called eventual consistency. The system will eventually get all copies to agree. But “eventually” is not “immediately.”

Think of it like a group chat. You send a message. Your phone shows it instantly. But your friend in another country sees it two seconds later. Eventually, everyone has the same messages. But for a moment, you’re out of sync.

Some applications are totally fine with this. If you’re loading someone’s profile picture, a one-second delay before it updates everywhere is not a big deal.

But some applications cannot afford it. Imagine a banking app. You transfer money to a friend. The write goes through. You immediately check your balance on a server that hasn’t caught up yet. It shows you still have the full amount. That’s a real problem.

For cases like that, AWS offers strong consistency. When you make a write, the system waits until all copies confirm before telling you it succeeded. A bit slower, but no surprises.

You pick which one your application needs. The tradeoff is always speed vs guarantee.

Eventual vs strong consistency handles the reading side of the problem. But there’s a messier problem on the write side that replication alone cannot fix.

Press enter or click to view image in full size

Now here’s the harder problem

Imagine two people make requests to DynamoDB at exactly the same moment. One request says, “delete this user record.” The other says, “update this user’s email address.”

Both requests hit AWS. The system sends them to different AZs.

AZ-1 gets the delete. It deletes the record. AZ-2 gets the update. It updates the email.

Now AZ-1 says the record doesn’t exist. AZ-2 says it exists with a new email. They disagree about the state of your database.

When they sync up, which one wins? If the update wins, you have a record that was supposed to be deleted. If the delete wins, you’ve lost a legitimate update.

The order in which these two operations happen completely changes your final database state. And two different AZs just processed them in two different orders.

This is called the ordering problem. It’s one of the hardest things to get right in distributed systems.

Paxos

To solve the ordering problem, the AZs need to agree on write order before executing anything. This is the consensus problem. Paxos, published by Leslie Lamport in 1989, is the algorithm that solved it.

Why a simple vote breaks

The obvious approach: vote on it. Majority wins.

The problem is who gets to call the vote. If two AZs try to call one at the same moment, AZ-1 proposes “delete goes first” and AZ-2 proposes “update goes first.” Nodes split their votes. You end up with two decisions both claiming majority. That is worse than no decision at all.

Paxos adds a step before the vote: one proposer has to earn the exclusive right to propose. Once that right is established, competing proposals are dead.

Round 1: claim the right to propose

Every proposal carries a number, strictly increasing and unique across all proposers. When an AZ wants to propose something, it broadcasts: “I am starting proposal 42. Promise me you will not accept anything with a lower number.”

If a majority responds with that promise, any competing proposer with a lower number is finished. The majority will reject it outright. Proposal 42 now has a clear field.

But each AZ that sends back a promise also reports one more thing: the highest-numbered proposal it has already accepted, if it accepted anything before. Not a list of everything it ever accepted. Just the single highest one.

The proposer collects these responses. If multiple AZs report previously accepted values, it takes the one with the highest proposal number and uses that as its own value. Its original preference is dropped.

This is the part worth pausing on.

Say proposal 38 ran earlier. AZ-1 accepted it. AZ-2 accepted it. That is a majority, so the value was potentially committed. But before AZ-3 heard about it, something crashed. Now proposal 42 starts. AZ-1 and AZ-2 both report: “I accepted a value in round 38. It was: delete goes first.” The proposer for 42 is now required to carry that forward. It cannot propose something different.

The reason comes down to one thing: in Paxos, “committed” does not mean the client got a response. It means a majority of nodes accepted a value. The client not getting a response just means the client needs to retry. The value may already be sitting committed at the node level.

When proposal 42 starts, it cannot know whether round 38 fully committed or half-committed before crashing. So it assumes the worst: round 38 might be final. If 42 ignores it and proposes something different, it risks overwriting a decision that was already made. Paxos treats any previously accepted value as potentially final and forces the new proposer to preserve it.

Picking the highest-numbered accepted value is always the right call because any value that a prior majority committed will be the highest-numbered one reported back. Anything lower was superseded before it could commit.

If nobody in the majority has accepted anything, the proposer is free to use its own value.

Round 2: the vote

The proposer sends the actual proposal to everyone: “Proposal 42, value: delete goes first. Accept this.” Each AZ accepts it if it has not already promised to ignore 42 or higher. Once a majority accept, the value is decided.

Two rounds total. Round 1 locks out competing proposers and surfaces any previously committed values. Round 2 is the vote.

Press enter or click to view image in full size

Why majority and not everyone

In a three-AZ setup, any two AZs form a majority. One AZ can be completely unreachable and decisions still get made. When it comes back, it asks what it missed and replays every write in the agreed order. No gaps, no conflicts.

How all of this fits together in a single DynamoDB write

When you call putItem from your laptop, here's roughly what happens:

Your request hits AWS and gets routed to one of the AZs in your region. That AZ doesn’t just write the data locally.

If you ask for strong consistency, DynamoDB waits until all copies confirm before responding to you. If you asked for eventual consistency, it responds as soon as the majority agrees and lets the rest sync on their own.

Press enter or click to view image in full size

All of that happens before you finish blinking.

The eleven nines are an outcome, not a setting

There’s no checkbox in AWS that says “enable 11 nines of durability.” It’s the result of all these pieces working together: replication across AZs, consensus through Paxos, careful ordering of operations, and a clean recovery path when things go wrong.

The next time you call putItem and get a 200 back in 10ms, there's a vote that just happened across multiple data centers, a position in a log that was agreed on, and a recovery plan already in place for when something eventually goes wrong.

Someone thought all of that through, so you didn’t have to.