What I Learned Building a Real-Time Event Processing Platform With .NET and Kafka

Managing reliability, idempotency, and distributed system failures without sacrificing performance.

Google AI studio by Author

1. Why the First Design Was Wrong

The system started as a background service consuming events from a few Kafka topics and writing results to PostgreSQL. Within three months it was handling forty-three topics, six downstream consumers, three external API integrations, and throughput an order of magnitude beyond the original estimate.

The first design did not survive that. Single hosted service per topic, synchronous processing inside the consume loop, no offset management strategy beyond letting the client library handle it, optimistic assumptions about at-least-once delivery. Every one of those decisions became a production incident eventually.

What we rebuilt taught me more about distributed systems than any prior project. Not because the problems were exotic, but because Kafka forces you to confront exactly the failure modes that most application-layer systems prefer to ignore. Offsets are explicit. Rebalancing is visible. Lag is measurable. There is nowhere to hide.

2. The Consume Loop Is Not Where Your Logic Should Live

The most common mistake I see in .NET Kafka consumers is treating the consume loop as the processing layer. You poll for a message, you process it, you commit the offset, you move on. This works until your processing is slow, your downstream dependency is unavailable, or you need to handle a message that consistently fails without blocking the entire partition.

The consume loop should do one thing: move messages off the wire and into a processing pipeline as fast as possible. Actual business logic belongs elsewhere.

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    _consumer.Subscribe(_topics);

    await foreach (var batch in _channel.Reader.ReadAllAsync(stoppingToken))
    {
        await _pipeline.ProcessBatchAsync(batch, stoppingToken);
    }
}

private async Task ConsumeLoopAsync(CancellationToken ct)
{
    while (!ct.IsCancellationRequested)
    {
        var result = _consumer.Consume(TimeSpan.FromMilliseconds(100));
        if (result is null) continue;

        await _channel.Writer.WriteAsync(
            new ConsumeResultEnvelope(result), ct);
    }
}

The Channel<T> here is doing important work. It decouples consumption rate from processing rate, gives you backpressure when the processing pipeline is under load, and keeps the Kafka heartbeat alive while downstream operations complete. Without that decoupling, a slow database write or a blocked HTTP call starves the poll loop, the broker stops receiving heartbeats, and you get a rebalance at exactly the moment you can least afford one.

3. Offset Commits Are a Contract, Not a Formality

Auto-commit is convenient and almost always wrong for production workloads. When auto-commit is enabled, the client periodically commits the current offset regardless of whether your application has successfully processed the messages at that position. A crash between the commit and your database write means those messages are silently lost. You have no idea they were dropped until a downstream system notices missing data.

Manual offset management forces you to be explicit about what processed means. The offset for a message should be committed only after the processing result has been durably stored — not after it has been sent to a downstream service, not after it has been written to an in-memory cache, not after the HTTP call returned 200. Durable storage.

The subtlety is that Kafka’s offset commit is per-partition, not per-message. Committing offset N implicitly acknowledges all messages up to N on that partition. If you process messages out of order or in parallel across a single partition, you need to track which offsets are safe to commit. We built a small watermark tracker that maintained the highest contiguous committed offset per partition and only advanced the commit position when there were no gaps below it. The overhead is minimal and the correctness guarantee is worth it.

4. Idempotency Is an Application Responsibility

Kafka’s at-least-once delivery guarantee is a design choice, not a bug. Under normal conditions messages are delivered once. Under rebalance, consumer restart, or commit failure, they may be delivered again. Your application owns that.

Idempotency keys stored alongside processing results are the standard solution. Before processing, check whether a record for that event ID exists. If it does, skip and commit. If not, process and write both the result and the idempotency record in the same transaction.

public async Task ProcessAsync(OrderCreatedEvent evt, CancellationToken ct)
{
    await using var tx = await _db.BeginTransactionAsync(ct);

    var alreadyProcessed = await _db.IdempotencyKeys
        .AnyAsync(k => k.EventId == evt.EventId, ct);

    if (alreadyProcessed)
    {
        await tx.CommitAsync(ct);
        return;
    }

    await _orderRepository.CreateAsync(evt.ToOrder(), ct);
    await _db.IdempotencyKeys.AddAsync(
        new IdempotencyKey { EventId = evt.EventId, ProcessedAt = DateTime.UtcNow }, ct);

    await tx.CommitAsync(ct);
}

The transaction boundary is non-negotiable here. Writing the business result and the idempotency record must be atomic. If you write the result and then fail before recording the idempotency key, you will process the same message twice on the next delivery. The entire point of the pattern collapses.

For high-throughput scenarios, idempotency key checks can become a bottleneck. We moved to a bloom filter as a first-pass check — fast, in-memory, with a configurable false positive rate — and only hit the database when the bloom filter indicated the key might exist. The false positive rate added a small number of unnecessary database reads. The true negative elimination removed the database read entirely for the vast majority of first-time events.

5. Partition Assignment and Parallelism Are Related

Kafka partitions are the unit of parallelism. Twelve partitions and three consumer instances means four partitions per instance. Processing speed scales with instance count up to the partition limit. Beyond that, additional instances sit idle.

Your partition count sets the horizontal scale ceiling for that topic. Setting it too low means re-partitioning later, which is disruptive for ordered event streams. We over-partitioned topics we expected to grow and accepted the metadata overhead.

Within a single instance, parallelizing across owned partitions is straightforward — each partition gets its own channel and offset tracker. Parallelizing within a single partition is riskier. Kafka’s ordering guarantee is per-partition, and if events are causally related, concurrent processing can violate that ordering. We enforced single-threaded processing per partition and relied on partition count for throughput scaling.

6. Dead Letter Topics Are Not Optional

Some messages will always fail processing regardless of how many times you retry them. A malformed payload, a schema version your consumer does not understand, a business rule violation that makes the event permanently unprocessable. If you retry these indefinitely, you block the partition. If you drop them silently, you lose data. The only reasonable path is a dead letter topic.

The dead letter topic receives the original message, the exception details, the consumer group identity, the source topic and partition, and a timestamp. This gives you enough context to replay the message after fixing the underlying issue, investigate the failure cause, and alert on anomalous dead letter rates.

What matters operationally is the dead letter rate monitoring. A sudden spike in dead letters almost always indicates a schema change, a dependency failure, or a bug introduced in a recent deployment. If you treat the dead letter topic as a write-only archive, you lose that signal. We alerting on dead letter rate per topic with a short evaluation window, which turned several potential extended outages into five-minute investigations.

Reprocessing from the dead letter topic requires the same idempotency guarantees as normal processing. The events being replayed may already have been partially processed if the failure occurred after a partial write. Do not assume dead letter events are clean.

7. Consumer Group Rebalancing Will Find Every Race Condition

Rebalancing — the process of redistributing partition ownership when a consumer joins or leaves a group — is the event that exposes every assumption you made about processing state. During a rebalance, partitions are revoked from current owners and reassigned. Any in-flight processing for a revoked partition needs to either complete and commit before the revocation completes, or be abandoned and reprocessed by the new owner.

The Confluent .NET client exposes SetPartitionsRevokedHandler for this purpose. We used it to drain the processing channel for affected partitions, wait for in-flight work to complete, and commit pending offsets before acknowledging the revocation.

_consumer.SetPartitionsRevokedHandler((c, partitions) =>
{
    foreach (var partition in partitions)
    {
        _partitionProcessors[partition.Partition].DrainAndFlush();
    }

    var offsets = _offsetTracker.GetCommittableOffsets(partitions);
    c.Commit(offsets);
});

Without this handler, a rebalance during active processing means the new partition owner replays messages that were processed but never committed by the previous owner. With at-least-once semantics and good idempotency handling, this is survivable. Without idempotency, it is a data corruption event.

The incremental cooperative rebalancing protocol introduced in newer Kafka versions significantly reduces the disruption by only revoking partitions that need to move, rather than revoking all partitions and reassigning from scratch. We migrated to CooperativeSticky partition assignment and the reduction in rebalance-related processing interruptions was immediately visible in our lag metrics.

8. Lag Is the Most Useful Signal You Have

Consumer lag — the difference between the latest offset on a partition and the last committed offset for your consumer group — is the primary operational health indicator for a Kafka-based system. Everything else is secondary.

Rising lag means your consumers are not keeping up with the producer. The causes are numerous: processing slowdown, downstream dependency degradation, insufficient consumer instances, a burst of unusually large messages. Lag tells you something is wrong before your end-to-end latency metrics do, and it tells you where to look.

We exported partition-level lag to our metrics system and built dashboards that showed lag trend alongside consumer processing rate and downstream dependency latency. The combination made it possible to distinguish between “we need more consumer instances” and “our database is slow” within seconds of an alert firing, rather than spending twenty minutes in logs trying to correlate events.

The important nuance is that aggregate lag across all partitions is less useful than per-partition lag. A healthy consumer group processing twelve partitions may have low average lag while one specific partition has runaway lag due to a stuck consumer or a high-cardinality key causing uneven load. Aggregates hide that. Watch per-partition.

9. Schema Evolution Without Coordination

In a long-running platform, producers and consumers evolve independently. A producer adds a field. A consumer not expecting it should not break. A producer removes a deprecated field. A consumer reading it needs to handle the absence. Schema contracts make this manageable.

We standardized on Avro schemas with the Confluent Schema Registry for all inter-team event contracts. The registry enforces compatibility rules — forward, backward, or full — before a schema change is accepted. A producer cannot publish a breaking change without explicitly upgrading the compatibility level and going through review.

The operational discipline this enforced was more valuable than the technical mechanism. Teams stopped treating event schemas as internal implementation details and started treating them as public API contracts. Breaking changes required a migration plan rather than a flag day. For consumer-side handling, generated C# classes from the Avro schemas with nullable fields for anything potentially absent made schema evolution visible at compile time rather than runtime.

10. The Operational Lessons Outlast the Code

The platform we shipped is not the system I would design today. The retry logic is more complex than it needed to be, the dead letter reprocessing workflow is too manual, and monitoring has gaps we work around. That is normal for a system that grew faster than anyone anticipated.

What held up is the foundational discipline: explicit offset management, transactional idempotency, partition-aware parallelism, graceful rebalance handling, and lag as the primary health signal. These are not Kafka-specific lessons. They are distributed systems lessons that Kafka makes unavoidable.

Most application-layer systems abstract away the failure modes Kafka surfaces explicitly. Message queues hide redelivery behind retry policies. Databases hide replication lag behind eventual consistency. Kafka hides nothing. Every failure mode is visible, every guarantee is bounded, and every architectural decision has an observable consequence. Building on it will make you a better distributed systems engineer regardless of what you build next.