A Guide to the Outbox Pattern: Why @Transactional Isn’t Enough for Kafka in Microservices Architecture

Stop losing data between your DB and Kafka. Learn why @Transactional doesn't roll back Kafka messages and how to implement the Outbox Pattern in Spring Boot for 100% consistency

Does @Transactional Cover Both DB and Kafka? No. In most applications, @Transactional only controls the database transaction, not Kafka. It works only with the transaction manager configured for that resource.

By default, @Transactional only manages the database, not Kafka.

Spring Boot typically auto-configures a JPATransactionManager for the DB.

This manager has no native control over external Kafka brokers or clusters.

If a DB rollback occurs, a message sent to Kafka remains sent (inconsistency).

Kafka operates on its own separate transaction protocol and infrastructure.

To link them, you must explicitly configure a KafkaTransactionManager.

Even with both, they remain two separate “units of work” by default.

The Outbox Pattern is the preferred way to ensure both sync perfectly.

In that pattern, you save the Kafka message to a DB table first.

This ensures the message only exists if the DB transaction successfully commits.

@Transactional
public void transfer() {

    transferRepository.save(transfer);

    kafkaTemplate.send("transfer-topic", event);
    throw new RuntimeException();  // (3) Crash!

}

Start DB Transaction
    |
    +-- Save transfer to DB
    |
    +-- Send message to Kafka (outside DB transaction)
Commit DB Transaction

What the Outbox Pattern Actually Solves ? The Outbox pattern ensures that database updates and event publishing stay consistent, even if failures occur.

Instead of writing to the database and Kafka separately, the service writes both the business data and the event into the same database transaction.

BEGIN TRANSACTION

Insert into transfers
Insert into outbox_events

COMMIT

both executed in same transaction boundary ,This guarantees that events cannot be lost.

When I first learned about the Outbox Pattern, the idea sounded reasonable to me — store the event in a separate table along with the business transaction so we never lose events.

But my first reaction was also very practical

Wait, now we are adding another table and another process. Isn’t that introducing more operational headache?

That thought is actually valid. While the Outbox pattern solves a very important reliability problem , it also introduces some trade-offs that engineers must understand before adopting it.

How the Event Eventually Reaches Kafka

The event stored in the Outbox table is not published immediately.

Instead, a separate process reads the outbox table and publishes events to Kafka.

Application
     |
     v
Database Transaction
     |
     +---- transfers table
     |
     +---- outbox_events table
             |
             v
      Outbox Publisher
             |
             v
            Kafka

Once the event is successfully published, the record is marked as processed.

Why Not Use Distributed Transactions?

One alternative is distributed transactions (2-phase commit) across Database and Kafka.

But this approach has major problems:

• Complex to implement
• Slower performance
• Not supported by many systems
• Reduces system availability

Trade-offs of the Outbox Pattern

While the Outbox pattern solves reliability issues, it introduces some trade-offs.

Let’s walk through those trade-offs in a practical way.

Additional System Complexity

Without the Outbox pattern, the code flow is very straightforward.

Save the record to DB

publish event to kafka

With the Outbox pattern, the architecture changes slightly.

Instead of publishing directly to Kafka, the service first writes the event to an Outbox table inside the same transaction as the business data.

Later, another component reads the Outbox table and publishes events to Kafka.

The flow becomes

Application
   |
   +-- Business Table (transfers)
   |
   +-- Outbox Table (events)
           |
           v
      Outbox Publisher
           |
           v
          Kafka

Now the system needs to manage additional pieces such as

• the Outbox table itself
• a publisher process that reads events
• retry mechanisms if publishing fails
• cleanup jobs to remove processed events

So while the pattern improves reliability, it does increase the architectural complexity of the system.

Eventual Consistency

Events are not published instantly. There may be a small delay between Transaction commit and Event appearing in Kafka.

Without the Outbox pattern, the event is usually published immediately after the database write.

With the Outbox pattern, the flow looks like this

Transaction commits
↓
Event stored in Outbox table
↓
Publisher reads Outbox
↓
Event sent to Kafka

This introduces a small delay between Database update and Event reaching Kafka , the system becomes eventually consistent rather than instantly consistent.

For most systems this delay is tiny (milliseconds or seconds), but this needs to be taken into account when designing.

Additional Database Load

Another side effect is increased database activity.

Earlier, a transaction might perform just one write — save record

With the Outbox pattern, each business operation performs two writes — Insert transfer and Insert outbox event

Under high traffic, this means:

• more database writes
• a rapidly growing outbox table
• increased indexing and storage requirements

Systems often handle this by using batch publishing and efficient indexing strategies.

Outbox Table Growth

Because every event is first stored in the database, the Outbox table can grow quickly.

If old events are never removed, the table may eventually become very large and impact performance.

Production systems usually implement mechanisms such as:

• scheduled cleanup jobs
• time-to-live (TTL) policies
• archiving old events

Without proper maintenance, the Outbox table can turn into a database bottleneck over time.

Possibility of Duplicate Events

If the publisher crashes during publishing, the event may be retried. This means consumers must handle duplicate events using idempotency.

For example

Event sent to Kafka
Publisher crashes before marking event processed

When the publisher restarts, it may read the same event again and publish it again.

This means downstream systems may receive duplicate events.

Because of this, consumers of the event must implement idempotent processing, meaning they can safely handle the same event multiple times.

Additional Infrastructure

In many production systems, teams do not implement the publisher manually. Instead, they use Change Data Capture (CDC) tools.

A common architecture looks like this

Database
   |
   v
Outbox Table
   |
   v
Debezium (CDC)
   |
   v
Kafka

Tools like Debezium automatically detect changes in the Outbox table and stream them to Kafka.

While this simplifies development, it introduces additional operational infrastructure that must be deployed and maintained.

When the Trade-off Is Worth It

Despite these trade-offs, the Outbox pattern is widely used in systems where reliability is critical.

It becomes particularly valuable when

• event loss is unacceptable
• services communicate heavily through events
• database state and events must remain consistent

Common domains

• Banking transactions
• Payment systems
• Order management
• Inventory updates
• Audit systems

In these systems, data consistency is far more important than architectural simplicity

Conclusion

The Outbox Pattern solves a critical problem in microservices ensuring that database updates and events remain consistent, even when failures occur.

However, that reliability comes with some trade-offs

• additional architectural complexity
• eventual consistency
• extra database load
• handling duplicate events
• operational overhead

In practice, teams adopt the Outbox pattern when guaranteeing reliable event delivery is more important than keeping the system simple.

If you found this helpful, don’t forget to give this article a clap 👏 and follow me for more tips and insights! Your support means a lot.