“Why Did My Kafka Event Disappear? The Outbox Pattern Explained with a Real Production Bug”

If you’ve built systems that update a database and publish events to Kafka or RabbitMQ, you’ve faced this nightmare — and there’s a battle-tested fix.

The problem, stated plainly

Every engineer who has worked on event-driven microservices eventually runs into one of these two failure modes:

“The write to the database succeeded, but the event was never published.”

“The event was published… but the corresponding database record doesn’t exist yet.”

Both are silent failures. No exceptions thrown, no retries triggered — just two systems quietly disagreeing about what reality looks like. This inconsistency exists because databases and message brokers operate under separate transaction boundaries. There’s no distributed two-phase commit to save you.

A production story: 4,000–5,000 events a day

Let me ground this with a real example. In one of our production projects, our service was responsible for publishing ad item change events to the Channel Integration (CI) service. The CI team would assign a correlation ID to those events and republish them to downstream consumers.

The incident: Squadcast was firing 20–30 alerts every single day. 4000–5000 events were missing correlation IDs. Consumers were failing silently. The monitoring dashboards in Datadog showed no consumer lag — the jobs appeared healthy.

The investigation started the usual way: check consumer lag, inspect the scheduled job, look at the offset window. Everything looked clean. The publish job was running on schedule with a 2-hour offset. No lag. No apparent errors.

Then we looked more carefully at the publisher code. The event publish was happening inside a @Transactional method. After each publish, the event was marked as published in the database — also inside the same transaction.

Here’s a simplified version of what it looked like:

@Transactional
public void publishPendingEvents(List<AdItemChangeEvent> events) {
    for (AdItemChangeEvent e : events) {
        adItemChangeEventGenerator.handleAdItemChangeEvent(e, details);
        // ⚠️ Kafka publish happens here — inside the transaction
        adItemChannelUpdateDao.updateStatus(SUCCESS, e.getEventId());
        // ⚠️ DB update also inside same transaction
    }
}

The problem: Kafka is not a transactional resource that participates in your database’s commit. When events were published to Kafka but the surrounding transaction rolled back — or when the transaction committed but the broker hadn’t yet acknowledged — the state between Kafka and the database could drift. Some events hit the CI service without a proper commit having been recorded. Others never arrived at all.

The fix was to decouple the publish from the transaction boundary:

@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void handleAfterCommit(AdItemChangeEvent event) {
    adItemChangeEventGenerator.handleAdItemChangeEvent(event, details);
}

Result: Squadcast incidents dropped to zero. Events reaching the CI service now always had a committed database record backing them. No more ghost events, no more missing correlation IDs.

This fix is the essence of the Outbox Pattern — and understanding why it works requires understanding the pattern more deeply.

What the outbox pattern actually does

The outbox pattern introduces a durable intermediate store — an outbox table — that lives in the same database as your business data. Instead of publishing to Kafka directly inside your business transaction, you write the event to this table as part of the same atomic operation.

Client Action
     ↓
DB Transaction (Atomic)
     ↓
[ Business Data Updated ]
[ Outbox Event Inserted ]
     ↓    (Later, reliably)
Outbox Processor → Kafka / Message Bus

The guarantees this buys you:

• If the DB write succeeds, the event exists.
• If the message publish fails, it can be retried — the event is still in the outbox table.
• No event is published before the data is committed.
• No data is committed without a corresponding event queued.

The tradeoff is that publishing is no longer synchronous with the business operation — but in most event-driven architectures, that’s exactly what you want.

The code that causes the problem

Here’s the naïve pattern most engineers start with:

@Transactional
public void updateOrder(Order order) {
    orderRepository.save(order);
    messageBroker.publish(order.toEvent()); // ❌ outside transaction boundary
}

This looks reasonable. It fails in several ways:

• The DB write commits successfully but the broker is temporarily unavailable — event is lost.
• The broker receives the event but the DB transaction rolls back due to an unrelated exception — you’ve published a ghost event.
• The thread is interrupted between the two calls — partial state everywhere.

Distributed systems don’t guarantee ordering or atomicity across components. Relying on sequencing two separate I/O calls is optimistic engineering.

Three implementation approaches

1. Polling Publisher (Simple)

A scheduled job reads pending rows from the outbox table and publishes them, marking each as done.

@Scheduled(fixedDelay = 200)
public void processOutbox() {
    List<OutboxEvent> pending = repo.findPending();
    pending.forEach(event -> {
        kafkaPublisher.publish(event.getPayload());
        event.markAsPublished();
        repo.save(event);
    });
}

Pros: Simple to implement, portable across any language or framework. Cons: Adds slight delay (milliseconds to seconds).

2. TransactionalEventListener (Spring)

This is what resolved the production incident described above. Spring’s application event system lets you fire an event from within a transaction, and the listener defers execution until after the transaction commits.

// Inside your service, within @Transactional:
applicationEventPublisher.publishEvent(new AdItemChangeEvent(data));

// The listener — runs AFTER the DB transaction commits:
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void handle(AdItemChangeEvent event) {
    kafkaPublisher.publish(event);
}

The guarantee: if the transaction rolls back, the listener never fires. If the transaction commits, the listener fires. Kafka only receives events that have a committed database state behind them.

Note: If the Kafka publish itself fails after the transaction commits, the event is still at risk. For true at-least-once guarantees, combine this with a fallback polling mechanism on a persisted outbox table.

Pros: Near-zero latency. Cons: Framework-specific (Spring).

3. CDC with Debezium

Debezium tails the database’s binary log (or WAL) and streams row-level changes — including inserts to your outbox table — directly to Kafka. No application code is involved in the publish path at all.

This is the most operationally robust approach, but it requires running Debezium as additional infrastructure and designing your outbox table schema to work with its routing capabilities.

Pros: Zero code coupling, highly reliable. Cons: Significant infrastructure complexity.

Idempotency: the part most engineers skip

Even with the outbox pattern in place, your consumers will receive duplicate messages. Retries, network hiccups, at-least-once delivery semantics — all of these mean a consumer should be prepared to see the same event more than once.

Common strategies:

• Dedup by event ID — Store processed event IDs in a table. Before processing, check if the ID was already handled. Simple and effective for low-throughput systems.
• Idempotent upserts — Design your writes so that applying the same event twice produces the same result. INSERT ... ON CONFLICT DO UPDATE in PostgreSQL is your friend.
• Versioning / optimistic locking — Attach a version number to your entities. Reject any event whose version doesn’t match what’s expected.

When to use it — and when not to

Use it when:

• Data change and event publish represent the same logical operation.
• Guaranteed delivery is required.
• You’re building across microservice boundaries with Kafka or a message bus.
• Consistency matters more than absolute real-time speed.

Skip it when:

• The system explicitly tolerates inconsistency.
• Events are purely advisory — logging, analytics, non-critical observability.
• The added operational complexity outweighs the consistency requirement.

Lessons from production

The incident described above — 4,000–5,000 Squadcast alerts per day from missing correlation IDs — was traced to a single architectural mistake: publishing to Kafka inside a transaction boundary, without accounting for what happens when those two systems fall out of sync.

What made it subtle: there was no consumer lag. The scheduled job was running. Datadog dashboards looked normal. The bug wasn’t in the consumer or the scheduler — it was in the assumption that a sequential pair of I/O calls is equivalent to an atomic operation.

The outbox pattern exists to eliminate that assumption entirely.

The hardest bugs in distributed systems aren’t the ones that throw exceptions. They’re the ones where two systems quietly disagree about what’s true — and neither one has any idea.

Summary

The outbox pattern solves one of the core challenges in distributed architecture: keeping state changes and published events in sync — always. It gives you reliability, deterministic event flow, resilience against partial failure, and traceability.

Whether you reach for a polling publisher, Spring’s @TransactionalEventListener, or a full CDC pipeline with Debezium depends on your latency requirements and infrastructure appetite. But the core principle remains the same: write the event to the same transaction as your business data, publish it after.

If this resonated, consider sharing it with your team. The outbox pattern is one of those patterns that feels unnecessary — until the day it isn’t.