Message Ordering and Delivery Guarantees, Explained

Part 3 of the “Event-Driven Architecture” series. You assumed messages arrive once, in order. They don’t. That assumption is where event-driven systems quietly corrupt data.

You build an event-driven system (Parts 1–2) and naturally assume it behaves like ordinary code: each event happens once, in the order things occurred. Then production teaches you otherwise. An OrderShipped event gets processed before the OrderPlaced event that should have come first. A PaymentCharged event fires twice, double-charging a customer. A message vanishes entirely. None of this is a bug in your broker — it's the fundamental nature of distributed messaging, and assuming otherwise is how event-driven systems quietly produce wrong results.

The good news: once you know what messaging systems actually guarantee (and don’t), you can design for reality instead of being blindsided by it. Let me lay out the hard truths and how to handle each.

A message queue does not promise “exactly once, in order.” It promises “at least once, eventually.” The gap between what you assumed and what it actually guarantees is exactly where the data corruption lives.

Hard Truth #1: Delivery Is “At Least Once,” Not “Exactly Once”

Most message systems guarantee at-least-once delivery: a message will be delivered, but possibly more than once. This happens because of acknowledgements — if a consumer processes a message but crashes before confirming it, the broker assumes it failed and redelivers:

1. Consumer receives "PaymentCharged"
2. Consumer charges the card ✅
3. Consumer crashes BEFORE acknowledging
4. Broker didn't get the ack → redelivers "PaymentCharged"
5. A different consumer charges the card AGAIN ❌  (duplicate!)

“Exactly-once” delivery is famously near-impossible to guarantee end-to-end in a distributed system. So you must assume duplicates will happen. The fix is the recurring hero of this whole collection: idempotent consumers (the idempotency lesson). Dedupe on a unique message/event ID so processing the same event twice has the same effect as once:

def handle(event):
    if already_processed(event.id):   # seen this exact event? skip.
        return ack(event)
    do_work(event)
    mark_processed(event.id)
    ack(event)

Idempotency turns “at least once” into “effectively once” — without it, every duplicate is a bug.

Hard Truth #2: Order Is Not Guaranteed (Usually)

In a distributed system with multiple producers, consumers, and partitions, messages can arrive out of order:

Published:  OrderPlaced  →  OrderShipped
Received:   OrderShipped  →  OrderPlaced   (!!)  — shipped before placed?

This happens because messages may travel different paths, get retried, or be processed by parallel consumers at different speeds. If your logic assumes order (“an order must be placed before it can ship”), out-of-order delivery breaks it. Three ways to handle it:

• Partition by key for ordering where it matters. Systems like Kafka guarantee order within a partition. Route all events for the same entity (e.g. same order_id) to the same partition, and that order's events stay ordered — even though global order isn't guaranteed.
• Make handlers order-independent. Design so events can arrive in any order: e.g. an OrderShipped event that arrives first can be held or can create a placeholder until OrderPlaced catches up.
• Include version numbers / timestamps so a consumer can detect and ignore stale or out-of-sequence events.
• The cleanest is partitioning by entity key: you give up global ordering (which you rarely need) to keep per-entity ordering (which you usually do).

Hard Truth #3: Messages Can Be Lost (Without Care)

Without proper configuration, messages can vanish — a consumer acknowledges before finishing, a broker without persistence restarts, a queue overflows. Guard against it:

• Acknowledge AFTER processing, not before — so a crash mid-processing causes a redelivery, not a loss.
• Use durable/persistent queues — messages survive a broker restart (in-memory-only queues don’t).
• Dead-letter queues — messages that repeatedly fail go to a DLQ for inspection instead of being dropped silently (the message-queue lesson).
• The combination of at-least-once delivery + acknowledge-after-processing + durability is what makes the system reliable: it errs on the side of redelivering (which idempotency handles) rather than losing.

Fix: Design for the Guarantees You Actually Have

Put the three truths together into a design posture:

Assume: duplicates WILL happen   → make consumers idempotent (dedupe on event ID)
Assume: order is NOT guaranteed  → partition by key, or make handlers order-independent
Assume: messages CAN be lost     → ack after processing, durable queues, DLQs

Build every consumer as if the message might be a duplicate, out of order, and one of several attempts — because eventually it will be all three. Systems designed for these realities are robust; systems that assume "exactly once, in order" corrupt data the first time production proves them wrong.

Putting It Together

1. At-least-once, not exactly-once — duplicates happen; consumers must be idempotent.
2. Order isn’t guaranteed — partition by entity key, or design order-independent handlers.
3. Messages can be lost — ack after processing, use durable queues and DLQs.
4. Design for all three at once — assume duplicate, out-of-order, retried delivery by default.

The Takeaway

The most dangerous assumption in event-driven architecture is that messaging behaves like a function call — once, in order, reliably. It doesn’t: real message systems guarantee at-least-once, eventually, with no global ordering, and that gap between assumption and reality is precisely where double-charges and corrupted state come from. But these aren’t flaws to fight — they’re the nature of distributed systems, and they’re entirely manageable once you design for them. Make every consumer idempotent so duplicates are harmless, partition by entity key so the ordering you actually need is preserved, and acknowledge after processing with durable queues so nothing is lost. Build for the guarantees you genuinely have rather than the ones you wish you had, and your event-driven system stays correct under the messy reality of production.

This is Part 3 of the Event-Driven Architecture series. Next up: event sourcing — storing your data as a log of events, and the power and price that comes with it.

What’s the worst out-of-order or duplicate-message bug you’ve debugged? 👇