From Message Loss to Safe Replays: Building Reliable Kafka Consumers with Retries, DLQs, and Idempotency

What happens after the broker accepts your message? That’s where production fails.

In my last article, I solved the publisher side of event-driven systems — every event that hits the database also reaches Kafka, atomically (the Transactional Outbox pattern).

But that’s only half the story.

The other half is the consumer. And consumers fail in spectacularly creative ways:

• The payload is malformed.
• A downstream service is down.
• The consumer crashes mid-processing.
• The same event arrives twice.

If you don’t engineer for these explicitly, you end up with silent data loss, infinite retry storms, or duplicate side-effects like double-charging a patient.

Phase 3 of PatientFlow-Platform builds a four-layer defense for the AI service that consumes patient events. Each layer answers one specific failure mode. Let me show you each one.

Layer 1 — Manual Offset Commit (no message left behind)

Kafka’s default is auto-commit every 5 seconds. Sounds harmless. It isn’t.

If your consumer crashes 3 seconds after starting to process a batch, Kafka has already advanced the offset for the previous batch. Your in-flight messages are now in no-man’s land: not yet processed, but no longer redeliverable. Silent loss.

The fix is one line:

EnableAutoCommit = false;

After successful processing, we commit explicitly:

var processed = await ProcessMessageAsync(envelope, stoppingToken);
if (processed)
    _consumer.Commit(consumeResult);

If ProcessMessageAsync throws or returns false, the offset never advances. On consumer restart, Kafka redelivers the exact message that failed. No loss.

Proof — Kafka’s __consumer_offsets topic

• Every successful commit becomes a record inside Kafka’s internal __consumer_offsets topic.

• That inner offset: 7 is the high-water mark for the ai-service-group on patient-created. If the container dies right now, the next instance resumes from offset 8 — not 0, not somewhere fuzzy, exactly 8.
• That single record is what EnableAutoCommit = false buys: deterministic, durable recovery.

Layer 2 — Retry Topics with Exponential Backoff (give transient failures a second chance)

Most consumer failures are transient: a downstream service is slow, Redis just restarted, a network blip. Instant retries would only amplify the problem.

The pattern: when processing fails, don’t retry in-process — republish the event to a <topic>-retry topic. A separate consumer waits, then re-attempts.

if (!processed && retryCount < _maxRetryAttempts)
{
    await SendToRetryTopicAsync(envelope, retryCount + 1);
    _consumer.Commit(consumeResult);  // commit original, work moved
}

The retry consumer applies exponential backoff before each attempt:

var delaySeconds = Math.Pow(2, retryCount);   // 2s, 4s, 8s
await Task.Delay(TimeSpan.FromSeconds(delaySeconds));

Proof — full retry cascade in Loki

I published a deliberately malformed event (Payload: “BROKEN” — invalid JSON) and watched it travel through the entire chain:

2s → 4s → 8s → DLQ, one event, single ID highlighted across 13 log lines. The whole story tells itself.

Layer 3 — Dead Letter Queue (quarantine the truly broken)

After three exponential retries, the event is poison. Reprocessing again will fail the same way. Continuing to retry wastes cluster resources and blocks the partition for healthy traffic.

So we wrap the failed event with diagnostic metadata and ship it to a separate <topic>-dlq topic where operators can inspect it without disturbing the main flow:

var dlqMessage = new {
    OriginalMessage = originalMessage,    // raw bytes, forensic integrity
    Reason          = "Max retries exceeded: 3",
    Topic           = _topic,
    FailedAt        = DateTime.UtcNow,
    ConsumerGroup   = _consumer.MemberId
};
await _producer.ProduceAsync(_dlqTopic, ...);

Note OriginalMessage is stored as a string, not a parsed object. The whole reason this event failed is that it isn’t valid — re-parsing it in the DLQ would crash the DLQ itself. Preserving raw bytes is forensic gold.

Proof — DLQ message in Kafka UI

Operators get what failed, why it failed, when it failed, and which consumer gave up — everything needed to triage in under a minute.

Layer 4 — Idempotency (the same event, twice, is harmless)

Kafka guarantees at-least-once delivery. A network blip during commit means the same event can be redelivered. If your consumer has side effects (write to DB, charge a card, send an email), duplicates can be catastrophic.

The fix is a Redis-backed dedup cache keyed by EventId:

var cacheKey = $"processed_event:{envelope.EventId}";
if (await redis.GetAsync(cacheKey) != null)
{
    _logger.LogInformation("Event {EventId} already processed, skipping", envelope.EventId);
    return true;   // commit offset, do nothing
}

// ... process ...

await redis.SetAsync(cacheKey, "1", ttl: TimeSpan.FromDays(7));

Proof — replay the same event twice

I republished a real event (EventId = 5bfd036e-…) that had been processed days earlier:

Three lines. No duplicate Redis write. No duplicate downstream call. Offset still committed (no infinite redelivery loop). Clean.

The Mental Model

When you put all four layers together, every consumer failure has a defined home:

That last row is the quiet superpower. Idempotency means operators can replay anything, anytime, without fear. Recovery becomes a button click instead of a war room.

Takeaways

1. EnableAutoCommit = false is non-negotiable for stateful consumers. One line eliminates an entire class of silent data loss.
2. Retry in another topic, not in-process. It survives crashes, frees the main consumer, and gives you exponential backoff for free.
3. DLQ wrappers must include diagnostic metadata — Reason, FailedAt, ConsumerGroup, original raw bytes. The wrapper is the message to your future self at 3 AM.
4. Idempotency keys by EventId in a short-TTL cache turn replays from hazardous to routine.
5. Run the experiments. Reading the code isn’t enough.

The screenshots in this post aren’t decorative. They are the acceptance tests for the architecture. Every layer above was proven to work, end to end, before this article was written.