MicroNotes XIII: Resiliency, Observability, and the Reality of Distributed Systems

Notes from Sam Newman’s Microservices Workshop — Bangkok 2025

https://www.rosenaviation.com/wp-content/uploads/2024/03/failed-airplane-engine-Rosen-Aviation-scaled.jpeg

This is the final part of the MicroNotes series. We’ve covered everything from understanding what microservices are to breaking apart databases. You know the challenges. You know the options. You know the patterns.

But here’s the thing:

“When things go wrong in a distributed system, they go wrong in interesting ways.”

A service might be slow. Another service might be down. The network might be dropping packets. Multiple things might be failing at once. And the way these failures interact can create problems you never expected.

Building systems that can handle failure requires thinking about resiliency. Knowing what’s happening when something breaks requires observability.

Making tradeoffs when you can’t have everything requires understanding the constraints.

Let’s start with what resiliency actually means.

https://unsplash.com/photos/lone-tree-in-middle-of-desert-during-daytime-ulquHF89xuY

What Is Resiliency?

Resiliency isn’t a thing you have. It’s not binary. You don’t either have resiliency or you don’t.

It’s a characteristic of how your system performs. Like security. Like usability. It’s a quality attribute.

That means it’s variable. You define what “good enough” looks like. The concept of resilience isn’t absolute. It’s about what people expect it should be.

A system that’s resilient for a blog might not be resilient enough for a payment system. A system that’s resilient for a payment system might not be resilient enough for a medical device. It depends on what you’re building and what the consequences of failure are.

Here’s a definition that captures this:

A system is resilient when it sustains required operations under both expected and unexpected conditions by adjusting its functioning prior to, during, or following events, changes, disturbances, or opportunities.

Let’s break that down with some examples.

Sustains required operations

When something goes wrong, certain things still need to happen.

If a plane has an engine failure, the plane still needs to not crash. The people still need to be safe.

But the plane might not be able to fly as far. It might have to land at a different airport. It might have to cancel some flights. Those are things you’re willing to give up to make sure the required operations continue.

This implies there are some things you’re happy to give up on when things go wrong.

Being resilient isn’t about trying to do everything even when things are going wrong. It’s about accepting that you might have to give up on certain things to make sure the required operations continue.

For a web service, maybe you turn off non-essential features when things are going wrong. Maybe you show a simpler version of the page. Maybe you disable recommendations or personalization. The core functionality still works. The important things still happen.

Expected and unexpected conditions

You have to be prepared for both. You know engines can fail. That’s expected. You know servers can crash. That’s expected. You know networks can have problems. That’s expected.

But you also have to handle surprises. Volcanic ash. Pandemics. A database corruption that you didn’t see coming. A bug that only appears under specific conditions. Things you didn’t plan for.

The expected failures are easier. You can plan for them. You can build redundancy. You can have backup systems.

“The unexpected failures are harder. That’s where the human side of resilience comes in.”

Adjusts its functioning

You might have to change how things work. The plane might have to land at a different airport. The system might have to degrade gracefully.

A service might have to use cached data instead of fresh data. A system might have to route around a problem.

This isn’t just about automatic recovery. It’s also about people making decisions.

“It’s about having the flexibility to change when something unexpected happens.”

Resilience Engineering vs. Safety Management

Traditionally, the focus has been on safety management:

making sure nothing goes wrong. Reduce adverse events. Stop bad things from happening. Prevent failures.

That works when you have a small number of components. If you have three servers, you can try to make sure none of them fail. You can buy really good hardware. You can have redundant power supplies. You can monitor everything closely. You can try to prevent every possible failure.

But as systems get bigger, as you have more components, the math changes.

If you have a component with a 1 in 100 chance of failure, and you have 100 of those components, you’re approaching near certainty that one of them will fail on any given day. You can’t prevent all failures. The math doesn’t work.

So the focus shifted to resilience engineering: making sure as many things go right as possible, and accepting that something might go wrong around the edges.

At that scale, resilience engineering matters more:

Make systems that keep working even when parts fail.

The classic example

Google’s early servers are a perfect example of this shift. They had so many computers that failures were inevitable. They couldn’t prevent all failures. So they built their software to tolerate computers failing, and they made their computers incredibly easy to replace.

https://blog.kevinchisholm.com/internet/googles-first-production-server-rack-circa-1998/

Hard drives weren’t screwed in. They were held with Velcro. When a drive failed, you could rip it out and stick a new one in. No tools needed. No downtime. Just swap it out.

They weren’t trying to prevent failures. They were trying to make failures cheap and easy to recover from.

This is a fundamental shift in thinking. Instead of “how do we prevent failures” it’s “how do we make failures not matter.” Instead of “how do we make things perfect,” it’s “how do we make things good enough and easy to fix.”

Failures weren’t prevented. They were expected.

The Four Concepts of Resilience

David Woods identified four concepts of resilience. These aren’t just technical concepts. They apply to how systems work, but also to how organizations work.

1. Rebound

Your system’s ability to recover after something goes wrong. This could be manual or automated. For rebound, you have to be able to model in advance what can go wrong so you know how to recover.

Basic automatic recovery is possible. Kubernetes desired state management. Auto-scaling groups. You say “I want five copies of this service.” If one fails, it brings back a replacement. That’s self-healing at a simple level.

This can be automated:

• Kubernetes self-healing
• Auto-scaling groups
• Desired state management

But rebound can also be manual. Someone notices something is wrong. They investigate. They fix it. They restore service. The key is that you have a plan. You know what to do when things go wrong.

The challenge is that you can only plan for things you can imagine. If something completely unexpected happens, you might not have a plan. That’s where the next concept comes in.

“When something bad happens, how easily are you able to recover?”

2. Robustness

Your system’s ability to absorb known events. You think ahead of time about what can go wrong, and you build your system to handle those scenarios. Engines can fail, so you have two engines. Machines can fail, so you have multiple copies of your service.

This is the most straightforward concept.

You identify risks. You build defenses.

You add redundancy. You make things more reliable.

“How well we can handle expected issues?”

But here’s the paradox: we have a lot of things to take care of.

More robustness → more complexity → more new ways to fail.

If you have two services for redundancy, you now have to keep them in sync. You have to handle split-brain scenarios. You have to deal with network partitions. The complexity of managing two services is higher than managing one.

Sometimes making your system more robust isn’t worth it. The cost or complexity isn’t justified. That’s a conscious decision. And that’s okay. Not everything needs to be highly available. Not everything needs redundancy.

A few years ago, a cloud region went down and affected lots of big tech companies. Zoom, Reddit, ChatGPT were all affected. People asked: “Why are you running out of one cloud region? Didn’t you know a cloud region could go down?”

Of course they knew. Could they have run a multi-region setup? Potentially. Would that have been very complicated and very costly? Yes. Would customers have been happy paying that cost for a situation that’s incredibly rare? Probably not.

Plus, by increasing that complexity to handle multi-region failover, you’re introducing new potential sources of failure. Many organizations decided that’s where they draw the line. It’s a conscious decision.

“We will spend a lot of our time looking at robustness”

3. Graceful extensibility

Your ability to handle the unexpected. This is typically where enterprise organizations fail. They build systems that are robust for the known. They have procedures for the known. Then something unexpected happens, and they’re stuck.

This isn’t about technology. It’s about people, culture, and process. If you have rigid roles, little autonomy, lots of controls, you can’t handle surprise. If something unexpected happens, you can’t deal with it because your structure doesn’t allow it.

You cannot foresee everything. Did we expect Covid? Not to the extent it happened. Many years ago, flights in Europe got grounded because a volcano in Iceland erupted. The reason they grounded flights is because many years earlier they found out that flying through volcanic ash was a bad idea. They didn’t know at the time.

There was a British Airways flight that flew at altitude through a plume of volcanic dust. That volcanic dust went into the jet engine intakes and froze the engines. The jet engines stopped working. The plane descended rapidly. The pilot and copilot were able to calmly get the plane down to an altitude where the ash defrosted and they could restart the engines. They weren’t expecting that. But their training was about dealing with the unexpected. How do we keep calm? What are we going to do?

That’s graceful extensibility. Handling something you didn’t plan for.

Imagine a scenario where something completely unexpected happens. Maybe a new type of attack. Maybe a combination of failures that you never anticipated. Maybe a business requirement that changes everything.

In a rigid organization, people have to escalate. They have to get approval. They have to follow procedures. By the time they can respond, it’s too late.

Learning organizations are more flexible. They give people autonomy. They trust people to make decisions. They can handle surprise better because people can adapt without waiting for permission.

“You cannot anticipate every eventuality. Surprising things will happen.”

4. Sustained adaptability

How well you learn from what happened. This is where a lot of organizations struggle. You need psychological safety. If people are afraid to admit mistakes, you can’t learn from them.

In the case of the volcanic ash, they dealt with it in the moment. Then they learned: don't fly through volcanic ash. That's sustained adaptability. Learning from what happened.

Some organizations celebrate learning. They give awards to the engineer who caused the biggest problem, because that’s where the most learning happened. They take the blame out of it. The focus is on learning, not on finding someone to blame.

This is about creating a culture where failure is seen as an opportunity to learn, not as something to be punished. It’s about post-incident reviews that focus on understanding what happened, not on finding who to blame.

The first two concepts are primarily technical. They’re about thinking in advance about problems. The last two are about people and culture. If you do the first two without the last two, you’re not resilient. You’re just one surprise away from trouble.

You can have all the redundancy in the world. You can have all the automatic recovery. But if your organization can’t handle surprises, and if you can’t learn from what happened, you’re not resilient. You’re just robust until something unexpected happens.

So you understand what resilience means. But building it requires a foundation. Here’s where to start.

The Dickerson Hierarchy

Making your service reliable requires a foundation. Mike Dickson, who was a site reliability engineer at Google and worked on healthcare.gov, created a hierarchy based on Maslow’s hierarchy of needs. The idea is you should do some of the things at the bottom before you do things at the top.

If you try to do things at the top without the foundation, you’re building on sand. It might work for a while, but it’s not sustainable.

https://medium.com/@brouwers.mike/enabling-sre-step-by-step-69329de39dc9

1. Monitoring (Observability)

Start here. You need to know what’s happening. If you don’t know what’s happening, you can’t fix problems. You can’t improve things. You’re flying blind.

The “golden signals” are latency, traffic, error rates, and saturation. These are the four things you need to watch. But look at these together, not in isolation. Low latency with 100% error rate isn’t good. High latency with quadrupled traffic might be fine. You need to understand the relationships.

Latency is how long things take. Traffic is how much load you’re handling. Error rates are how often things fail. Saturation is how much of your capacity you’re using.

If latency is high but traffic is also high, that might be expected. If latency is high and traffic is low, that’s a problem. If error rates are high, that’s always a problem. If saturation is high, you might be running out of capacity.

You need to see these signals. You need dashboards. You need alerts. You need to be able to dig into the details when something looks wrong.

Golden signals:

• Latency
• Traffic
• Errors
• Saturation

2. Incident response

When something goes wrong, you need to know who’s on call, how they get information, and how they escalate. You need at least basic monitoring before you can do incident response. Otherwise, someone says “something’s wrong” and you have no idea what.

This is about having a process. Who gets paged? How do they get notified? What information do they have? How do they escalate if they need help? How do they communicate with other people?

Without monitoring, incident response is just guessing. Someone says “the site is slow.” You have no idea why. You have no idea where the problem is. You’re just trying things randomly.

With monitoring, someone says “the site is slow,” and you can see that latency is high, error rates are up, and one specific service is saturated. Now you know where to look.

“Who’s on call?”

“How they get alerted?”

“How they escalate?”

3. Post-incident review

Not “root cause analysis.” There’s rarely a single root cause. There are usually multiple things that contributed. Focus on understanding what happened, not on finding a smoking gun. Make it blameless. Gather data. Learn.

This is about learning from incidents. What happened? Why did it happen? What can we do to prevent it from happening again? What can we do to detect it faster if it happens again?

The key is making it blameless. If people are afraid they’ll be blamed, they won’t be honest. They won’t share what they know. You can’t learn if people aren’t honest.

Some organizations do this really well. They have post-incident reviews that focus on understanding, not on blaming. They document what happened. They identify improvements. They actually implement those improvements.

4. Testing & release

Functional and non-functional tests. But don’t do load testing before you understand what’s coming into your system. That’s a waste of time.

This is about having confidence that your software works. But it’s also about understanding what “works” means. Does it work under normal load? Does it work under high load? Does it handle errors gracefully?

The key insight here is that you need to understand your system before you can test it effectively. If you don’t know what normal traffic looks like, how can you test for high traffic? If you don’t know what errors look like, how can you test error handling?

5. Capacity planning

Making sure you have the computing resources you need, even when things fail. Redundant capacity. Failover. Self-healing. This is where you might focus on self-healing systems.

This is about having enough resources. But it’s also about having resources in the right places. If all your capacity is in one data center, and that data center goes down, you’re in trouble.

You need to think about redundancy. You need to think about failover. You need to think about self-healing. If a service instance fails, can it be replaced automatically? If a data center goes down, can traffic be routed elsewhere?

6. Development practices

Building things into your software to handle failure better. Idempotency. Retries. Timeouts. Circuit breakers. You might pull some of this forward, but it’s hard to do well without the building blocks underneath.

This is about writing code that handles failure gracefully. If a call to another service fails, what do you do? Do you retry? Do you fail fast? Do you use a circuit breaker? Do you have a fallback?

These are important patterns. But they’re hard to implement well if you don’t have monitoring. How do you know if your retries are working? How do you know if your circuit breakers are opening at the right time? You need observability.

7. Product-level features

Product features, go-to-market features. Canaries. Progressive delivery. This is where you think about reliability features.

This is about features that help with reliability. Canary deployments. Blue-green deployments. Feature flags. A/B testing. These are powerful tools, but they require the foundation underneath.

The idea is: if you’re looking at work items and most of them are up here, and you’ve done nothing down here, that’s a problem. You need the foundation first.

You can’t do capacity planning if you don’t know what your current capacity is. You can’t do incident response if you don’t have monitoring. You can’t learn from incidents if you don’t do post-incident reviews.

Start at the bottom. Build the foundation. Then work your way up.

There’s also a fundamental constraint you need to understand about distributed systems.

If you’re doing the top without the bottom,

you’re building a castle on sand.

CAP Theorem

CAP theorem is a formal mathematical proof about the inherent tradeoffs in distributed systems. You can only guarantee two of three things: consistency, availability, or partition tolerance.

This isn’t a choice you make once. It’s a constraint you have to work within. And it affects every decision you make about how your system works.

A mathematical proof: You can only choose two:

Consistency:

Every read receives the most recent write or an error. Your system can be entirely consistent if everything errors. That’s consistent, just not desirable.

Consistency means that all nodes see the same data at the same time. If you write something to one node, all other nodes immediately see that write. Or they get an error.

Availability:

Every request receives a non-error response, without the guarantee that it contains the most recent write.

Availability means that the system continues to operate even if some nodes are down. Every request gets a response. It might not be the most up-to-date data, but you get a response.

Partition tolerance:

The system continues to operate despite an arbitrary number of messages being dropped or delayed by the network.

Partition tolerance means that the system can handle network problems. Messages can be lost. Messages can be delayed. The system still works.

Here’s the thing: if you’re building a distributed system, you have to tolerate partitions. You can’t guarantee the speed of communication. You can’t guarantee packets won’t get lost. Networks are unreliable. So in practice, you’re choosing between consistency and availability.

THE TWO GOLDEN RULES OF DISTRIBUTED COMPUTING

Sometimes, you simply can’t reach the thing you want to talk to. And when that happens, the network has failed — which means your system must tolerate partitions.

In practice, that failure forces a trade-off. During a partition, you don’t get everything — you choose between consistency and availability.

You’re not picking the “best” one.

You’re picking the one you’re willing to lose last.

But you can make different tradeoffs in different parts of your system. For some functionality, you might favor availability. For other functionality, you might favor consistency.

Let’s make this concrete. You’re selling CDs online. You have an order aservice, an inventory service, a catalog service, and a payment service.

A partition happens. You can’t reach the inventory service. You don’t know if you have stock. But you know how much things cost. And you can take payments.

Would you still sell the item?

It can behave differently depending on the business:

• E-commerce: sell anyway → cancel or back-order later
• Concert tickets: absolutely not. Fixed inventory. High customer anger potential.

If you’re running an e-commerce company, you probably sell the CD even though you don’t know if you have it. You can cancel the order later if you don’t have stock. You can offer to back-order it. If you don’t sell anything, customers go somewhere else. You might have had the stock, but you lost the sale.

Availability wins. You keep selling. You deal with the inconsistency later.

But if you’re selling concert tickets, same system, same partition, you probably don’t sell the ticket if you’re not sure you have it. You can’t back-order a concert ticket. It’s fixed inventory. If you sell a ticket you don’t have, fans get angry. Very angry.

Consistency wins. You stop selling. You wait for the partition to resolve. You make sure you don’t oversell.

Technically, it’s the same system. But the business context is different. The tradeoffs are different.

Here’s another example that shows how business context changes everything.

Sam told us about explaining this to a team in Romania once. They looked at him blankly. “That’s not how it works” they said.

In Romania, when you buy things online, you don’t pay and then it gets delivered. You pay on delivery. The item comes to your door, and you give your money to the courier.

So the whole discussion about whether to sell when you can’t check inventory doesn’t even come up. If the item doesn’t arrive at your door, you don’t pay. The business context is entirely different. This is why foreign companies like Amazon had trouble moving into e-commerce in Romania. They said “pay now,” and people said “no, I’ll pay when it arrives.” Given Romania’s history, they had reasons not to trust people from outside the country.

Same technical system. Completely different business context. Completely different tradeoffs.

That’s CAP theorem in practice. When you have a partition, you have to make a choice. The right choice depends on the business context. Talk to your business stakeholders. They’re the ones who can make that decision.

Engineers can’t make this decision alone. You need to understand the business. You need to understand what happens if you choose availability over consistency. You need to understand what happens if you choose consistency over availability. Then you can make the right choice.

But to make these decisions, and to handle failures when they happen, you need to know what’s actually going on in your system.

Observability

You need to know what’s happening in your system. That’s observability.

This isn’t just about having logs. It’s about being able to understand what’s happening, why it’s happening, and what you can do about it.

Start with the golden signals: latency, traffic, error rates, saturation. These tell you the health of your system. But they’re just the beginning.

Add logs.

Logs tell you what happened. But raw logs are hard to work with. You need structured logging. You need to be able to search. You need to be able to correlate events.

Add traces.

Traces tell you how requests flow through your system. A request comes in. It calls service A. Service A calls service B. Service B calls service C. You can see the whole path. You can see where time is being spent. You can see where errors are happening.

Add metrics.

Metrics tell you about the state of your system. How many requests per second? How much memory is being used? How many database connections are open? These are numbers that change over time.

For getting information out of your services, use OpenTelemetry. It’s an open standard. You use a library in your programming language. It sends metrics and traces. Then you pick a tool that supports OpenTelemetry. Honeycomb. DataDog. New Relic. Whatever.

The benefit of OpenTelemetry is you can switch vendors later without changing your code. Your services don’t change. You just point them at a different backend. You’re not locked in.

This is important because observability tools change. New tools come out. Prices change. Features change. You want to be able to switch without rewriting all your code.

For logging, if you’re not using OpenTelemetry, use structured logging. Standardize your formats. Make it easy to search and analyze. JSON is a good format. It’s structured. It’s easy to parse. It’s easy to search.

The key is: start simple. Get the basics working. Get the golden signals. Get basic logging. Get basic error tracking. Then you can add more sophisticated tooling later.

Don’t try to do everything at once. Don’t try to instrument every single line of code. Start with the important things. Start with the things that will help you understand what’s happening.

As you learn more about your system, you can add more instrumentation. You can add more detailed traces. You can add more metrics. But start simple.

The goal is to be able to answer questions. When something goes wrong, you should be able to answer: What happened? When did it happen? Where did it happen? Why did it happen? What can we do about it?

If you can answer those questions, you have observability. If you can’t, you need to add more instrumentation.

What to Remember

So what does this mean for you? Resiliency isn’t about preventing all failures. It’s about handling failures when they happen. It’s about making sure the important things still work.

Start with the foundation:

monitoring, incident response, post-incident reviews. You need to know what’s happening. You need to be able to respond when things go wrong. You need to learn from what happened.

Then you can build on that: testing, capacity planning, development practices, product features. But you need the foundation first.

Resilience has two sides:

the technical side (robustness and rebound) and the human side (graceful extensibility and sustained adaptability). You need both. If you only do the technical side, you’re not resilient. You’re just one surprise away from trouble.

CAP theorem tells you that in a distributed system, you have to make tradeoffs. When there’s a partition, you choose between consistency and availability. The right choice depends on your business context. Talk to your stakeholders. They can help you make that decision.

For observability, start simple.

Get the golden signals working. Add structured logging. Use OpenTelemetry so you can switch tools later. Don’t try to do everything at once.

Think of it this way: you can’t prevent all failures. But you can make failures cheap. You can make recovery fast. You can learn from what happened. That’s resilience.

“Resilience isn’t about never failing. It’s about failing gracefully and learning from it.”

That’s the difference between a brittle system and a resilient one.

https://unsplash.com/photos/person-throwing-rock-qKlUdr1qOR8

***All Notes

---

***I: What Are Microservices? II: Forget Service Size — Focus on What Your Team Can Manage III: Microservices Aren’t About Technology — They’re About Team Autonomy IV: Nobody Cares About Your Microservices — Only the Outcome V: Information Hiding — The Discipline That Makes Microservices Work VI: Request-Response vs Event-Driven — Choosing How Services Talk VII: Distributed Transactions Are Sad — Use Sagas Instead VIII: Designing Microservices for the Edge — Lessons from Fish Farming IX: Testing Microservices — Beyond the Test Pyramid X: Migrating to Microservices — Changing the Wheels While the Car Is Moving XI: Domain-Driven Design — Speaking the Same Language XII: Breaking Apart Databases — When Data Becomes the Problem XIII: Resiliency, Observability, and the Reality of Distributed Systems

---