8 Distributed System Patterns Every Backend Engineer Will Eventually Need

Saga, CQRS, Circuit Breaker, and the rest — explained with apps you already use. No PhD required.

Part 1 of 2. This part covers the heavy-hitters — the patterns that show up in nearly every distributed system. Part 2 digs into the underrated underdogs: lesser-known patterns that rarely make the headlines yet quietly keep real systems alive.

When you split a system into services, you trade one kind of problem for another. The single big application was hard to scale and scary to deploy, but at least it was honest: one database, one process, one place where the truth lived. The moment you break it apart, you inherit a new reality where the network can fail, services answer slowly, and “did the payment go through?” becomes a genuinely hard question.

Distributed system patterns are the accumulated answers to that new reality. They aren’t academic trivia — each one exists because a real team got burned and figured out a repeatable fix. The trick to learning them is to stop memorizing definitions and start recognizing the problem each one solves. Once you can feel the problem, the pattern becomes obvious.

This article walks through eight of the most useful ones. For each, we’ll look at the pain that creates it, the shape of the fix, a concrete example from an app you’ve probably used, a simple diagram, and the gotcha nobody mentions until it bites you.

Let’s go.

1. API Gateway — the front door

The problem. Imagine a food delivery app. Behind the scenes there’s a service for restaurants, one for orders, one for payments, one for the courier tracking, maybe a dozen more. If the mobile app had to know the address of every single service, handle authentication for each, and stitch responses together, the client would become a tangled mess — and every internal change would ripping through every app on the App Store.

The pattern. Put one service in front of all the others. The client talks only to the gateway. The gateway handles the cross-cutting concerns — authentication, rate limiting, routing, sometimes combining several internal calls into one response — and forwards traffic to the right place.

API Gateway. One stable entry point.

Why it helps. The client gets one stable, well-documented entry point. Security lives in one place instead of being re-implemented in every service. You can reshape your internal architecture freely as long as the gateway’s contract stays the same.

When to reach for it. Almost any system with more than a handful of services and multiple client types (mobile, web, partner APIs).

The gotcha. The gateway is now a single point of failure and a potential bottleneck — it has to be highly available and fast. And it’s tempting to dump business logic into it (“just add this little rule here”). Resist. The gateway routes and protects; it should not become a second monolith.

2. Circuit Breaker — stop hammering a service that’s already down

The problem. A stock trading app shows live prices by calling an external market-data provider. One afternoon that provider gets slow — responses that normally take 50ms now take 30 seconds. Your app keeps calling it anyway. Threads pile up waiting for replies, memory fills with pending requests, and within minutes your entire app is frozen — not because your code is broken, but because it politely kept knocking on a door no one was answering.

The pattern. Wrap the risky call in a “circuit breaker,” borrowed directly from electrical wiring. It watches for failures. When failures cross a threshold, it trips — and for a cooldown period, it stops making the call entirely and fails fast (returns a cached price, or a clear “temporarily unavailable”). After the cooldown, it lets one test request through. If that works, it closes again and resumes normal traffic.

Circuit Breaker. Fail fast, don’t wait.

Why it helps. A slow dependency stays their problem instead of becoming your outage. Failing fast frees up your resources, and the cooldown gives the struggling service room to recover instead of being pounded while it’s down.

When to reach for it. Any time you call something you don’t control — a third-party API, a payment processor, another team’s service that’s been flaky.

The gotcha. You need a sensible fallback. Failing fast is only useful if “fast failure” means something graceful: a stale-but-recent price, a default value, a queued retry. Tripping the breaker just to throw an error in the user’s face isn’t much of a win.

3. Bulkhead — contain the blast radius

The problem. Ships are divided into watertight compartments called bulkheads, so a hole in one section doesn’t sink the whole vessel. Software has the same failure mode. Picture a ride-hailing app where one shared pool of 100 worker threads handles three jobs: surge pricing, map rendering, and push notifications. One day the maps provider slows down. Map requests start hogging threads, waiting. Soon all 100 threads are stuck on maps — and now pricing and notifications, which were perfectly healthy, can’t get a single thread either. One slow dependency took down two unrelated features.

The pattern. Don’t share one pool. Give each dependency (or each tier of traffic) its own isolated pool of resources — threads, connections, whatever’s scarce.

Bulkhead design pattern

Why it helps. A failure in one area is contained. Maps can be completely down and your users can still see prices and get notifications.

When to reach for it. When a single process handles work of varying importance or talks to multiple dependencies with different reliability. Pairs naturally with the circuit breaker — bulkheads limit how much one failure can spread, breakers stop you from feeding the failure.

The gotcha. Partitioning resources means you can’t pool them for peak efficiency. You’re trading some raw throughput for resilience. Size the compartments deliberately; too many tiny pools and you’ll starve everything at once.

4. Saga — transactions across services that can’t share a transaction

The problem. This is the big one. In a single database, “place an order” is easy: start a transaction, charge the card, reserve the inventory, create the shipment, commit. If anything fails, the database rolls everything back atomically. But in a microservices world, Payments, Inventory, and Delivery each own their own database. There is no shared transaction to roll back. So what happens when the card is charged, the stock is reserved, and then delivery assignment fails? You’ve taken someone’s money for an order that can’t ship.

The pattern. A Saga breaks one big transaction into a sequence of local transactions, one per service. Each step has a paired compensating action that undoes it. If step 4 fails, you run the compensations for steps 3, 2, and 1 in reverse — refund the payment, release the stock, cancel the order.

Saga design pattern

There are two flavors:

Choreography — services react to each other’s events, no central boss:

Order Service ──"OrderPlaced"──▶ Payment Service
                                     │ "PaymentTaken"
                                     ▼
                              Inventory Service
                                     │ "ItemsReserved"
                                     ▼
                              Delivery Service
                                     │
                       (any failure triggers compensating
                        events that flow backward: refund,
                        release stock, cancel order)

Orchestration — one coordinator tells each service what to do and tracks the outcome:

              ┌──────────────────────┐
              │  Order Orchestrator  │
              └──────────┬───────────┘
        1. charge        │  2. reserve     3. assign courier
           ┌─────────────┼──────────────┬──────────────┐
           ▼             ▼              ▼              ▼
        Payment      Inventory      Delivery     (on any failure,
                                                 fire compensations
                                                 in reverse order)

Why it helps. You get end-to-end consistency without a distributed transaction — which, at scale, is the only realistic option.

When to reach for it. Any business process that spans multiple services and must either fully succeed or cleanly undo: checkout, booking a trip, opening a bank account.

The gotcha. Sagas give you eventual consistency, not instant consistency. For a brief window, the system is in a partial state (money taken, order not yet confirmed). Your UX and your logic have to tolerate that. Choreography is simple for short flows but turns into spaghetti as steps multiply — there’s no single place to see “where is this order right now?” Orchestration adds a coordinator but makes the flow visible and debuggable, which is usually worth it past three or four steps.

5. CQRS — split reading from writing

The problem. Go back to that stock trading app. The data model that’s perfect for writing — normalized tables, careful constraints, one row per trade — is terrible for the reading the app actually does most: “show me this user’s portfolio with live valuations, sorted by daily gain, across five account types.” That read might join a dozen tables and run thousands of times per second, while writes are comparatively rare. Forcing both through one model means every query fights the write constraints, and you can’t optimize for either.

The pattern. CQRS — Command Query Responsibility Segregation — is just this: use one model for writes (commands) and a separate model for reads (queries). The write side stays normalized and correct. The read side is denormalized, pre-shaped exactly for how the screen displays it, and kept in sync via events.

CQRS design pattern

Why it helps. Each side scales independently. The read side can be replicated heavily, cached aggressively, and shaped per screen. Reads stop competing with writes for the same resources.

When to reach for it. Read-heavy systems where read and write needs genuinely diverge — dashboards, feeds, search-heavy product catalogs, analytics views.

The gotcha. Two models means the read side lags the write side slightly — eventual consistency again. A user might place a trade and not see it reflected for a moment. That’s fine for a portfolio summary, not fine for “do I have enough balance to place this order?” Use the write model for decisions that must be exact, and never apply CQRS to a simple CRUD service just because it sounds sophisticated. It’s overhead you’ll regret.

6. Event Sourcing — store the changes, not just the result

The problem. A bank account row says balance: 150. That single number is the current truth, but it threw away the story. How did we get to 150? When? In what order? If a number looks wrong, you can't reconstruct what happened — the history is gone, overwritten with every update. For a bank, an audit trail isn't a nice-to-have; it's the law.

The pattern. Instead of storing the latest state, store every change as an immutable event, in order. The current state is derived by replaying those events.

Event Sourcing design pattern

Why it helps. You get a complete, tamper-evident history for free. You can rebuild state at any past point in time (“what was the balance last Tuesday?”), debug by replaying exactly what happened, and feed those same events into other systems — fraud detection, reporting, a CQRS read model.

When to reach for it. Domains where history and auditability are first-class requirements: finance, healthcare, anything regulated, or systems where understanding “how did we get here” matters as much as “where are we.”

The gotcha. This is a heavyweight pattern. Reconstructing state by replaying thousands of events is slow, so you need snapshots (periodic saved checkpoints). Events are immutable, so a bug in past events can’t be edited away — you fix it forward with corrective events. And changing the shape of an event over time (schema evolution) is a genuine, ongoing engineering tax. Don’t event-source your user-settings table. Do consider it for the money.

7. Transactional Outbox — publish events without losing them

The problem. A service does two things when an order is placed: it saves the order to its database, and it publishes an “OrderPlaced” event to a message broker like Kafka so other services (notifications, analytics, delivery) can react. The trap: these are two separate systems. If you save to the DB and then the app crashes before publishing the event, the order exists but nobody downstream ever hears about it — no confirmation email, no courier assigned. Flip the order and you get the opposite ghost: an event for an order that didn’t actually save. You cannot wrap a database write and a message-broker publish in one atomic transaction.

The pattern. Don’t publish to the broker directly. In the same database transaction that saves the order, also insert the event into a plain “outbox” table. Since both writes are in one transaction, they either both happen or neither does — no ghost states. A separate relay process then reads unpublished rows from the outbox and pushes them to the broker, marking them sent.

Transactional Outbox

Why it helps. It guarantees that an event is published if and only if the business change actually committed. No lost events, no phantom events. It’s the quiet workhorse that makes event-driven systems trustworthy.

When to reach for it. Any time a service must reliably emit an event after a state change — which, in an event-driven architecture, is constantly.

The gotcha. The relay can occasionally publish the same event twice (it sent it, then crashed before marking it done). So consumers must be idempotent — processing the same event twice has to be safe (e.g. “send confirmation email” should check whether one was already sent). “At-least-once delivery” is the rule almost everywhere; design for duplicates, not against them.

8. Strangler Fig — replace a monolith without a rewrite

The problem. You’ve inherited a ten-year-old insurance monolith. It works, it makes money, and everyone’s afraid to touch it. The fantasy is a big-bang rewrite — pause everything for a year, build the shiny new system, flip a switch. In reality, big-bang rewrites are where projects go to die: requirements drift, the old system keeps changing under you, and you can’t ship anything until the very end.

The pattern. Named after a vine that grows around a tree, slowly replacing it until the original is gone. Put a proxy in front of the monolith. Then, one capability at a time, build a new service and route just that slice of traffic to it. The monolith keeps handling everything else. Repeat until there’s nothing left inside the old system — and then you delete it.

Strangler Fig

Step 1:  Client ──▶ [ Proxy ] ──▶ Legacy Monolith (handles everything)

Step 2:  Client ──▶ [ Proxy ] ──┬─▶ New Payments service
                                └─▶ Legacy Monolith (everything else)

Step 3:  Client ──▶ [ Proxy ] ──┬─▶ Payments service
                                ├─▶ Orders service
                                └─▶ Legacy (shrinking…)

Why it helps. You ship value continuously and de-risk every step. Each migrated slice can be tested, rolled back, and proven in production on its own. The business never has to stop while you “go dark” for a rewrite.

When to reach for it. Modernizing legacy systems, breaking up a monolith, or migrating to a new platform — basically any large rewrite that would otherwise be a single terrifying leap.

The gotcha. For a long stretch, you run both systems at once, and often the same data lives in two places — keeping them in sync is the hard, unglamorous part. The migration can also stall: teams peel off the easy pieces, declare victory, and leave a stubborn core of the monolith alive for years. Pick the order of extraction deliberately, and commit to finishing.

How they fit together

These patterns aren’t a menu where you pick one. Real systems layer them. Trace a single “place order” tap through a mature food delivery backend:

The request enters through the API Gateway, which authenticates it and routes it to the Order service. Placing the order kicks off a Saga spanning payment, inventory, and delivery. When each step commits its local change, it reliably emits its event using the Transactional Outbox. The call to the external payment processor is wrapped in a Circuit Breaker, and the threads handling payments are isolated in their own Bulkhead so a payment slowdown can’t freeze order creation. Every state change is appended as an immutable event via Event Sourcing, giving the finance team a perfect audit trail. And the snappy “your orders” screen reads from a denormalized CQRS read model, not the transactional core. If this whole system grew out of an older monolith, all of it arrived gradually, behind a proxy, courtesy of the Strangler Fig.

None of these is exotic. Each is just a named, battle-tested answer to a specific question the network forces on you the moment you go distributed.

A word of caution

Every pattern here adds complexity, and complexity is a cost you pay forever. The mark of a good engineer isn’t knowing all eight — it’s resisting the urge to use them until the problem is real. A two-service app doesn’t need Sagas. A CRUD admin panel doesn’t need CQRS. Reach for a pattern when you can clearly name the pain it relieves. Until then, the simplest thing that works is almost always the right architecture.

Learn the problems first. The patterns will be waiting when you need them.

Coming in Part 2 — the underdogs. The eight patterns above are the famous ones. But there’s a second tier of lesser-known patterns — the Anti-Corruption Layer, Sidecar, Dead Letter Queue, Leader Election, and more — that rarely make the headlines yet quietly hold real systems together. That’s what Part 2 is all about.

If you found this useful, the best way to internalize these is to pick one pattern and trace it through an app you use every day. Once you start seeing them in the wild, you can’t unsee them.