The distributed systems lessons you only learn after production

Production has a way of exposing every assumption you thought was safe

Photo by Lavi Perchik on Unsplash

Every distributed system looks correct on a whiteboard.

The nodes talk to each other, the queue absorbs the spikes, the retries handle the blips, and the diagram has just enough arrows to feel thorough.

Then it goes live, and production starts asking questions the whiteboard never had to answer: what happens when this call succeeds but the acknowledgment is lost, what happens when two nodes disagree about what time it is, what happens when the “temporary” outage lasts eleven minutes instead of eleven seconds.

None of this is about bugs, this is about assumptions were safe in dev and staging environments and stopped being safe the moment of real traffic, real network partitions and real humans get involved.

Here are the lessons that tend to survive contact with production, with Go examples for the ones that are easiest to get wrong quietly.

At-least-once delivery means your handler is not optional idempotency, it’s mandatory idempotency

Message brokers like Kafka and Redpanda are honest about their contract: at-least-once delivery.

That “at least” is doing a lot of work.

A consumer can crash after processing a message but before committing its offset, and when it restarts, that message comes right back.

If your handler debits an account or issues a loan disbursement, and it isn’t idempotent, you don’t have a distributed system, you have a random number generator for your ledger.

The fix isn’t clever, it’s disciplined: every mutating operation needs a natural or synthetic idempotency key, checked and recorded atomically with the mutation itself.

func (s *Service) ProcessDisbursement(ctx context.Context, req DisbursementRequest) error {
    return s.db.ExecTx(ctx, func(q *repository.Queries) error {
        exists, err := q.IdempotencyKeyExists(ctx, req.IdempotencyKey)
        if err != nil {
            return fmt.Errorf("checking idempotency key: %w", err)
        }
        if exists {
            return nil // already processed, safe no-op
        }

        if err := q.InsertLedgerEntry(ctx, repository.InsertLedgerEntryParams{
            AccountID: req.AccountID,
            Amount:    req.Amount,
            EntryType: "disbursement",
        }); err != nil {
            return fmt.Errorf("inserting ledger entry: %w", err)
        }

        return q.RecordIdempotencyKey(ctx, req.IdempotencyKey)
    })
}

The key insight is that the idempotency check and the mutation have to initialize inside the same transaction.

If they’re separate calls, you’ve just moved the race condition somewhere less visible instead of removing it.

Retries without jitter turn a small outage into a thundering herd

A downstream dependency hiccups for two seconds.

Every one of your service instances notices at roughly the same moment, and every one of them retries on roughly the same schedule.

Now the dependency, which was recovering, gets hit with a synchronized wave of requests right as it comes back up. This is how a two-second blip becomes a ten-minute outage.

Exponential backoff is necessary but not sufficient. Jitter is what actually breaks the synchronization.

func retryWithJitter(ctx context.Context, maxAttempts int, fn func() error) error {
    base := 100 * time.Millisecond
    var lastErr error

    for attempt := 0; attempt < maxAttempts; attempt++ {
        if err := fn(); err == nil {
            return nil
        } else {
            lastErr = err
        }

        backoff := base * time.Duration(1<<attempt)
        jitter := time.Duration(rand.Int63n(int64(backoff)))
        wait := backoff/2 + jitter/2

        select {
        case <-time.After(wait):
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return fmt.Errorf("after %d attempts: %w", maxAttempts, lastErr)
}

The other half of this lesson is that retries need a ceiling somewhere in the system, not just in each client.

If every layer retries independently — the client, the gateway, the service mesh — you can accidentally multiply a single request into dozens of attempts against a struggling dependency.

A timeout that isn’t propagated isn’t a timeout, it’s a suggestion

It’s common to set a client timeout on the outermost call and assume it protects everything downstream.

But if that deadline isn’t threaded through every subsequent call in the chain, an inner call can happily keep running long after the outer caller has given up and moved on, quietly burning connections, goroutines, and database locks for a response nobody is waiting for anymore.

In Go this is usually a context.Context discipline problem more than a networking one.

func (s *Service) HandleLoanApproval(ctx context.Context, req ApprovalRequest) (*Approval, error) {
    ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
    defer cancel()

    creditScore, err := s.creditClient.FetchScore(ctx, req.MemberID)
    if err != nil {
        return nil, fmt.Errorf("fetching credit score: %w", err)
    }

    // ctx, with its remaining deadline, must keep flowing forward
    decision, err := s.rulesEngine.Evaluate(ctx, creditScore, req)
    if err != nil {
        return nil, fmt.Errorf("evaluating rules: %w", err)
    }

    return decision, nil
}

The rule of thumb: if a function accepts a context, it must pass that same context (or a derived one) to everything it calls that does I/O. A dropped context is a silent promise that nothing downstream will honor.

Partial failure is the default failure mode, not the edge case

In a monolith, a failed operation usually fails cleanly, the whole request rolls back.

In a distributed system, “the write to the ledger succeeded but the notification service timed out” isn’t a rare edge case, it’s Tuesday.

The system will spend most of its life in some partially completed state, and pretending otherwise is how you end up with silent data drift between services that never gets caught until a reconciliation job or an angry customer finds it.

This is the entire reason patterns like sagas and outbox tables exist.

Instead of hoping every step in a multi-service operation succeeds together, you design explicitly for the case where it doesn’t, with compensating actions and a durable record of what’s actually been done.

type OutboxEvent struct {
    ID        uuid.UUID
    EventType string
    Payload   []byte
    CreatedAt time.Time
    Published bool
}

func (s *Service) CreateShareCertificate(ctx context.Context, req ShareRequest) error {
    return s.db.ExecTx(ctx, func(q *repository.Queries) error {
        cert, err := q.InsertShareCertificate(ctx, req)
        if err != nil {
            return fmt.Errorf("inserting share certificate: %w", err)
        }

        payload, err := json.Marshal(ShareCreatedEvent{CertificateID: cert.ID})
        if err != nil {
            return fmt.Errorf("marshaling event: %w", err)
        }

        return q.InsertOutboxEvent(ctx, repository.InsertOutboxEventParams{
            EventType: "share.created",
            Payload:   payload,
        })
    })
}

A separate poller publishes rows from the outbox table to Redpanda and marks them published, so the database write and the event emission are never allowed to disagree with each other.

Nothing gets lost between “we saved it” and “we told everyone else about it.”

Clocks lie, so causality has to be earned, not assumed

Wall-clock timestamps feel authoritative because they’re just numbers, but across machines they drift, and under load, ordering by created_at can silently produce results that contradict what actually happened.

Two events a millisecond apart on different nodes can be recorded in the wrong order, and nothing in your code will complain, because nothing checked.

For anything where order genuinely matters — ledger entries, state transitions, event sourcing — you need something stronger than a timestamp: a monotonic sequence number, a vector clock, or a workflow engine like Temporal that gives you deterministic replay instead of hoping the clocks agreed.

Observability is a design decision, not an afterthought

The last lesson is less technical and more cultural: you cannot debug what you cannot see, and in a distributed system the interesting failures almost never show up in a single service’s logs.

They show up as a gap between two services’ understanding of the same event.

Structured logging with correlation IDs threaded through every hop, and distributed tracing that survives across your gRPC and message boundaries, aren’t nice-to-haves you add before a big launch.

They’re the only reason you’ll be able to answer “what actually happened” three months from now, at 2 a.m., when it matters.

If you look at all these lessons, none of them are exotic. They’re mostly about refusing to let convenience stand in for correctness — an idempotency check here, a propagated context there, an outbox table instead of a hopeful two-step write.

Production never punish complexity, it punishes the assumptions that were never actually tested.

Thanks for reading. Read these too

[embed]The common trade-offs Junior engineers rarely think about The mindset shift that separates beginners from experienced engineersmedium.com

[embed]The engineering habits that matter more than coding skills The difference between shipping code and becoming indispensableblog.devgenius.io

[embed]The architecture decision we didn’t regret 5 years later The quiet decision that prevented years of technical debtblog.stackademic.com