Stop Losing Events: The Transactional Outbox Pattern in .NET Microservices

Every event-driven microservice starts with the same 4 lines:

It feels safe. It passes code review.

It’s also wrong.

Not in a subtle way. “Your billing service will silently miss patients on the day Kafka has a hiccup, and you’ll only find out three weeks later from an angry CFO” wrong.

Why This Matters

One user action triggers a chain reaction:

• Patient created in PatientService DB
• PatientCreated event published to Kafka
• Billing service → creates account
• AI service → updates RAG embeddings

Now imagine that Kafka publish silently fails.

• The patient exists. The billing account doesn’t. The RAG index is stale. The audit log is missing an entry.
• Your database and your event stream now disagree about reality. Every downstream consumer is computing on lies.
• No alert. No automated recovery. Only divergence — discovered weeks later when someone notices the totals don’t match.

This is the dual-write problem, and it’s the most expensive bug in microservices.

The Bug in 4 Lines

Look at those 4 lines again. There are three failure modes hiding in them:

The database transaction does nothing to protect Kafka. The two systems have no shared notion of “commit.” You cannot atomically write to both.

This is not solvable by adding try/catch. It is structurally unfixable from inside the same transaction.

The Pattern: Outbox Table + Publisher

Stop trying to write to two systems at once.

The guarantee becomes:

• The event will be published if and only if the entity was committed.

If Kafka is down, the row stays in the outbox. The publisher retries. When Kafka returns — automatically — the event flows.

The database transaction is the source of truth. The publisher is the bridge.

The Outbox Row

The schema is intentionally boring:

Plus one composite index that makes the publisher’s “find unpublished, oldest first” query cheap forever:

No frameworks. No magic. A table and a flag.

Outbox holding events while Kafka is down After stopping Kafka and creating 5 patients via the API: 5 rows in OutboxMessages with IsPublished=false. The DB transactions committed. The events are safe.

The Atomic Write

The entire reliability guarantee lives in one SaveChangesAsync call:

EF Core wraps multiple Add calls in a single database transaction by default. Patient row and outbox row commit together, or roll back together. That’s it. That’s the magic.

The Publisher

Equally unremarkable:

A BackgroundService runs this loop every 10 seconds. That’s the entire reliability primitive.

Outbox drained after Kafka returns After restarting Kafka and waiting 10 seconds: same 5 rows, now IsPublished=true with PublishedAt timestamps. No manual intervention. No lost events. This is the recovery the pattern promises.

The Saga Twist Most Tutorials Skip

There’s a subtle bug I shipped in my first iteration — and almost every outbox tutorial gets it wrong.

• The naive flow looks fine: build the event payload, save entity + outbox row together in one SaveChanges.
• The problem: the entity’s auto-generated ID is zero at the moment you build the payload. EF hasn’t called the database yet. Every event you publish carries PatientId = 0.
• Downstream consumers receive PatientId = 0 for every create. The RAG index breaks silently. Billing accounts collide. The events publish successfully, consumers process happily, the data is just wrong.

This is saga-inspired rollback behavior within the Patient service boundary: if the gRPC call to Billing fails, everything rolls back — including the patient. No half-created state. No orphan events.

One unit of work, three side effects, atomic.

What Actually Matters

1. The Outbox Is Not Optional

• There is no “lightweight” version of event-driven microservices that skips the outbox. Either you have it, or you have silent corruption waiting for the day Kafka has a partition.
• Treat it like a foreign key — invisible 99% of the time, but the system collapses without it.

2. The DB Transaction Is Your Only Atomic Primitive

Anything that crosses a process boundary is not atomic with your DB write. The outbox cheats — it turns “publish to Kafka” into “insert a row” so it inherits the database’s atomicity.

3. Polling Sounds Dumb. It Isn’t.

• Polling every 10 seconds feels primitive. It isn’t.
• It’s simple (one query, one update), resilient (publisher crash = next poll picks up where it left off), observable (every event has a row, a retry count, an error message), and idempotent.
• For 10s freshness, polling beats LISTEN/NOTIFY, CDC, and Debezium. Save those for when you actually need sub-second event latency.

Loki view of the patient-service: the OutboxPublisher background worker logs every poll cycle and every successful publish. The spike at 21:36 is the visual signature of recovery — five trapped events drained across two poll cycles, fully automatic, no human in the loop.

4. The Event Envelope Is Forever

Every row in the outbox is a contract with every consumer that will ever exist. At minimum:

The EventId is the most important field — it’s what lets consumers deduplicate when the same event is delivered twice (which will happen with at-least-once delivery).

5. Don’t Forget The Cleanup Worker

Published rows pile up forever unless you delete them. A second background service that runs hourly:

Not glamorous. Absolutely required. Outbox tables that grow unbounded are how your “reliable event system” becomes “the table that ate the database.”

Key Takeaways

• Dual writes are structurally unfixable. Any code that writes to a DB and publishes to Kafka in the same operation has a silent corruption bug.
• The outbox makes “publish to Kafka” atomic with your DB write by turning it into a regular INSERT.
• A background polling worker is enough. 10-second freshness is fine for 99% of systems.
• Auto-generated IDs are a trap. Save the entity first, then build the outbox payload with the populated ID.
• The event envelope is a forever-contract. Get the fields right on day one.