Event-Driven Architecture with Kafka in .NET: A Modern Approach to Building Scalable Systems

Designing loosely coupled, highly scalable, and resilient distributed systems using Apache Kafka and .NET

Topic Overview — Introduction

Modern applications often need to process millions of events in real time. Traditional request-response architectures can become bottlenecks as systems grow and scale.

Event-Driven Architecture addresses this challenge by allowing services to communicate asynchronously through events, enabling better scalability, flexibility, and loose coupling between components.

In this article, we will explore how to build an Event-Driven system using Apache Kafka with .NET and understand how this architecture helps create scalable and resilient applications.

Key Features of Event-Driven Architecture / Kafka

• Asynchronous Communication: Services communicate via events without waiting for immediate responses.
• Loose Coupling: Producers and consumers do not need to know about each other.
• Scalability: Kafka supports horizontal scaling through partitions.
• High Throughput: Kafka is optimized for handling millions of messages per second.
• Fault Tolerance: Data is replicated across brokers to prevent data loss.
• Event Replay: Consumers can replay past events when needed.

Advantages of Event-Driven Systems

• Improved System Scalability: Services can scale independently.
• Better Resilience: If one service fails, others continue to operate.
• Real-Time Processing: Kafka enables near real-time data streaming.
• Flexibility in Integration: Easy to integrate new services without impacting existing ones.
• Better Observability: Event logs provide traceability and auditability.

Real-World Use Cases

Event-Driven Architecture with Kafka is ideal for:

• Order Processing Systems (E-commerce)
• Payment Processing Pipelines
• Real-Time Notifications
• Inventory Management
• Log Aggregation Systems
• IoT Data Streaming
• Microservices Communication

What is Apache Kafka?

Apache Kafka is a distributed event streaming platform used to publish, store, and process large streams of real-time data.

It was originally developed at LinkedIn and later open-sourced under the Apache Software Foundation.

Kafka Architecture Overview

The architecture of Apache Kafka is designed to handle high-throughput, real-time data streaming with strong scalability and fault tolerance. Kafka works on a distributed publish–subscribe model, where producers send events and consumers process them asynchronously.

Explain : Kafka acts as the central event broker. Producers publish events to topics. Topics are split into partitions, enabling parallel processing. Consumers within the same consumer group share partitions and scale message processing horizontally.

Topics and Partitions

Kafka organizes messages into topics, which are logical streams of events.

Each topic is divided into partitions, which allow Kafka to scale horizontally.

Benefits of partitions:

• Enable parallel processing
• Allow consumers to process messages independently
• Preserve ordering within a partition

Example:

Kafka guarantees message order only within a partition.

Producer

Kafka Producer — How It Works

When your application sends an order event to Kafka, it goes through 5 simple steps:

1. Order request — your app calls producer.send() with the order data.
2. Serialize — the order object is converted to bytes (JSON, Avro, etc.) before sending.
3. Connect to broker — the producer routes the message to the correct partition leader using a persistent connection.
4. Publish to topic — records are batched and sent to the Kafka topic for efficiency.
5. Store in partition — the broker appends the record to the partition log and assigns it a unique offset.

Consumer

Kafka Consumer — How It Works

A Kafka consumer reads events from a topic in 5 steps:

1. Join group — the consumer registers with the group coordinator and announces itself as part of a named consumer group.
2. Partition assigned — the broker assigns specific partitions to this consumer. Each partition is owned by exactly one consumer in the group at a time.
3. Poll broker — the consumer enters a continuous poll loop, fetching a batch of records starting from its last committed offset.
4. Deserialize records — raw bytes are converted back into usable objects (JSON, Avro, etc.) using the configured deserializer.
5. Process message — your business logic runs: save to a database, trigger a downstream service, update a cache, and so on.

After successful processing, the consumer commits its offset — telling Kafka how far it has read. On the next poll, it picks up from there.

Consumer Groups

Kafka consumers read messages as part of a consumer group, allowing multiple consumer instances to share the processing workload efficiently.

Key characteristics of consumer groups include:

• Each partition is assigned to only one consumer within a consumer group at any given time.
• Multiple consumers in the same group enable parallel processing across partitions.
• This partition-to-consumer assignment ensures message ordering within a partition and prevents duplicate processing within the same group.
• If a consumer fails, Kafka automatically rebalances the group and reassigns its partitions to other active consumers.

Example

Consider a topic with four partitions and a consumer group containing two consumers. Kafka may assign partitions 0 and 1 to Consumer A, and partitions 2 and 3 to Consumer B. If Consumer A crashes, Kafka automatically reassigns its partitions to Consumer B during the rebalance process.

In our code:

GroupId = "order-consumer-group";

All consumers using the same GroupId become members of the same consumer group and cooperate to process messages from the topic.

Consumer Group vs GroupId

These two terms are closely related but often confused.

• Consumer Group: A logical group of consumers that work together to consume messages from a topic
• GroupId: A configuration property used to uniquely identify that consumer group in Kafka

Example:

var config = new ConsumerConfig
{
 BootstrapServers = “localhost:29092”,
 GroupId = “order-consumer-group”,
 AutoOffsetReset = AutoOffsetReset.Earliest
};

All consumers using the same GroupId belong to the same consumer group.

Production Architecture Overview

In a modern Event-Driven Architecture (EDA), services communicate through events rather than direct synchronous API calls. Kafka acts as the central event backbone that enables asynchronous communication between independent services.

For example, when an Order Service creates a new order, it publishes an OrderCreated event to a Kafka topic. Multiple downstream services such as Inventory, Payment, Notification, and Analytics can consume the same event independently without impacting the producer.

Benefits of this architecture include:

• Loose coupling between services
• Independent scaling of producers and consumers
• Improved fault tolerance and resilience
• Better deployment flexibility
• Support for real-time event processing

A typical flow looks like:

Order API
    |
    v
Kafka Producer
    |
    v
+------------------+
|  Order Topic     |
+------------------+
   |      |      |
   v      v      v
Inventory Payment Notification
 Service   Service    Service

This approach allows organizations to build scalable distributed systems where services evolve independently while maintaining reliable communication.

Message Keys for Ordered Processing

This is important for real-world systems.

Add to Producer Code

await _producer.ProduceAsync("orders", new Message<string, string>
{
    Key = order.OrderId.ToString(),
    Value = JsonSerializer.Serialize(order)
});

Explain:

By using OrderId as the message key, Kafka ensures that all events for the same order are routed to the same partition, preserving order.

Offset Behavior Section

Kafka tracks message position using offsets.

Each message within a partition has a unique offset number.

Example:

Consumers store offsets to know where to resume processing.

Offsets represent positions within a partition log. They are not globally unique message identifiers and are only meaningful within a specific partition.

Consumer Configuration:

AutoOffsetReset = AutoOffsetReset.Earliest

Use cases:

• Earliest → replay events • Latest → real-time processing

Graceful Shutdown

Add to consumer code:

Console.CancelKeyPress += (_, e) =>
{
    e.Cancel = true;
    consumer.Close();
};

Explain:

Graceful shutdown ensures that the consumer commits offsets before stopping, preventing duplicate message processing.

Dead Letter Queue (DLQ)

Dead Letter Queue (DLQ) — Your Kafka Safety Net

A Dead Letter Queue is a special Kafka topic that catches messages your consumer failed to process — even after retrying. Instead of losing the message silently or blocking the entire partition, you route it to a DLQ and keep moving.

DLQs should be actively monitored. Failed messages should be reviewed, replayed, or resolved through operational processes to prevent silent message loss.

How it works:

1. Consumer receives a message from orders topic and tries to process it.
2. Processing fails (bad data, downstream service down, schema mismatch, etc.).
3. Consumer retries up to a configured limit (e.g. 3 attempts).
4. Retries exhausted → message is published to orders.DLQ along with error metadata (original topic, partition, offset, exception, retry count).
5. Your team can then inspect the failure, replay the message once the bug is fixed, alert on-call, or discard if the message is genuinely invalid.

Why it matters:

• No message is silently dropped — every failure is observable.
• The main consumer keeps moving — one bad message doesn’t block thousands of good ones.
• Replay is safe — fix the bug, re-publish from the DLQ, done.

Naming convention: orders.DLQ or orders.dead-letter — one DLQ per source topic is the common pattern.

Monitor it: Set an alert if DLQ message count grows — it means something in your pipeline is broken.

Idempotent producer

Idempotent Producer — Exactly-Once, No Duplicates

Idempotent producers prevent duplicate writes caused by producer retries but do not provide end-to-end exactly-once processing guarantees.

Without idempotence, a network hiccup causes a silent bug: the broker writes a message, the ack gets lost, the producer retries — and now the same message exists twice in the partition. Your order gets processed twice. Your invoice gets sent twice.

The idempotent producer fixes this with two simple things:

• Producer ID (PID) — the broker assigns a unique ID to each producer instance on first connect.
• Sequence number — the producer stamps every message with a per-partition counter that increments with each send.

When a retry arrives with the same PID + sequence number, the broker recognises it as a duplicate and silently discards it. Your consumer sees the message exactly once.

How to enable it:

properties

enable.idempotence=true

That single setting automatically enforces acks=all and retries=MAX_INT — the safest defaults.

What it covers and what it doesn’t:

• Exactly-once within a single partition — guaranteed.
• Exactly-once across multiple partitions or topics — you need the Kafka Transactions API (producer.beginTransaction() / commitTransaction()).

Overhead: Nearly zero. The broker only tracks the last 5 sequence numbers per producer per partition.

Bottom line: Always enable idempotence for any production producer writing critical data. It costs nothing and eliminates an entire class of subtle data bugs.

Schema Registry

Schema Registry — The Contract Between Producers and Consumers

In Kafka, messages are just bytes. Without a schema, nothing stops a producer from changing its payload structure and silently breaking every consumer downstream. Schema Registry solves this.

How it works:

1. Producer registers the schema (Avro, Protobuf, or JSON Schema) with the registry on first use. The registry returns a schema ID.
2. Each message carries only the schema ID (4 bytes) in its header — not the full schema. This keeps messages tiny.
3. Consumer reads the schema ID, fetches the full schema from the registry (cached after first fetch), and deserializes the bytes correctly.

Schema Evolution — Compatibility Modes:

ModeWhat it meansBACKWARD (default)New schema can read old messages — safe to add optional fieldsFORWARDOld schema can read new messages — safe to remove optional fieldsFULLBoth directions — safest, most restrictiveNONENo checks — use only in development

Why it matters:

• Breaking schema changes are rejected at publish time — not discovered when consumers crash in production.
• The schema is stored once, referenced by ID — smaller messages, faster serialization.
• Producers and consumers are decoupled from each other but coupled to the contract — the right tradeoff.

Popular implementations: Confluent Schema Registry, AWS Glue Schema Registry, Apicurio Registry.

Bottom line: If you’re using Kafka in production with more than one team touching the same topic, Schema Registry is essential — it’s the API contract for your event streams.

Partitioning Strategy

Kafka Partitioning Strategy — Choosing the Right One

Partitioning decides which partition a message lands in. The right strategy depends on whether you need ordering, throughput, or custom routing.

1. Key-based (default when key is set) hash(key) % numPartitions — same key always goes to the same partition. Guarantees ordering for all events sharing a key (e.g. all events for order-123 arrive in sequence). Best for: orders, user activity, sessions.

2. Round-robin (no key set, pre-Kafka 2.4) Messages spread evenly across all partitions one at a time. Maximum load balance, zero ordering guarantee. Best for: logs, metrics, analytics where order doesn’t matter.

3. Sticky (default no-key, Kafka 2.4+) Fills one partition’s batch completely before switching to the next. Produces fewer, larger batches — significantly better throughput than round-robin with the same no-ordering tradeoff. Best for: high-volume keyless workloads.

4. Custom partitioner Implement the Partitioner interface and return any partition number you want. Use it for VIP user routing, geo-based lanes, or priority queues (critical vs. background jobs on separate partitions).

Hot partition — the silent killer If a single key generates far more traffic than others, one partition absorbs all that load while others sit idle. The fix: append a random salt suffix to the key (user_id + "-" + random(0, N)) to spread the load — at the cost of losing strict per-user ordering.

Rule of thumb: Use key-based when order matters. Use sticky when it doesn’t and throughput does. Use custom only when your routing logic can’t be expressed as a key hash.

Batch Processing

Kafka Batch Processing — How Records Are Grouped and Sent

Kafka doesn’t send one message per network request. Instead, the producer groups records into batches before dispatching them — this is the primary reason Kafka can handle millions of messages per second.

How it works:

1. RecordAccumulator buffers records Every call to producer.send() places the record into an in-memory buffer called the RecordAccumulator — organized per (topic, partition) pair. Records sit here until the batch is ready to flush.

2. Two flush triggers A batch is sent when either condition is met first:

• batch.size — the batch has accumulated enough bytes (default 16 KB). Send immediately.
• linger.ms — the wait timer has expired (default 0 ms). Send whatever is buffered, even if the batch isn't full.

3. Sender thread compresses and dispatches A background Sender thread drains ready batches, optionally compresses them, and sends one network request to the broker carrying all records in the batch.

4. Broker appends the whole batch The broker writes the entire batch to the partition log in a single operation — far more efficient than writing records one at a time.

Tuning tradeoffs:

GoalSettingsLow latencylinger.ms=0, small batch.sizeHigh throughputlinger.ms=5–20, batch.size=64–512KBReduced I/Ocompression.type=lz4 or zstd

Rule of thumb: For real-time pipelines, keep linger.ms=0. For analytics or log pipelines where a few milliseconds don't matter, bumping linger.ms to 5–20ms can double or triple your throughput with zero code changes.

Health Check and monitoring

Kafka Health Check & Monitoring — What to Watch

Kafka exposes hundreds of metrics via JMX. These are the ones that actually matter in production.

Broker Health — The Non-Negotiables

MetricHealthy valueActiveControllerCountMust be exactly 1 — 0 means no controller, cluster is brokenUnderReplicatedPartitionsMust be 0 — any value means data isn't fully replicatedOfflinePartitionsCountMust be 0 — offline partitions mean data is unavailableDisk / CPU / NetworkWatch for saturation trends

Producer Metrics

• record-error-rate — should always be 0. Any errors mean messages are being dropped.
• request-latency-avg — how long the broker takes to ack. Alert if consistently above 100ms.
• record-queue-time-avg — time records spend waiting in the accumulator. A rising trend means the producer is falling behind.

Consumer Lag — Your Most Important Metric

Consumer lag = log-end-offset − committed-offset. It tells you how far behind your consumers are.

• A stable lag is fine. A growing lag means consumers can’t keep up with producers — scale up consumers or optimize processing.
• Track records-lag-max per partition and records-lag-avg per consumer group.
• If lag exceeds your retention window, consumers will start losing messages.

Topic Metrics

• MessagesInPerSec — write throughput per topic.
• BytesIn/OutPerSec — bandwidth usage, useful for capacity planning.
• LogEndOffset growth rate — how fast data is being written.

Alert Thresholds

• Page immediately: OfflinePartitions > 0, UnderReplicatedPartitions > 0, ActiveController != 1
• Warn: Consumer lag growing, record-error-rate > 0, disk above 80%

Recommended tools: Prometheus + Grafana (open source), Confluent Control Center (managed), Datadog or New Relic for full-stack observability.

Error Handling

Error Handling Strategy

Production systems must handle failures gracefully.

Producer Errors

Handle Kafka publish failures:

try
{
    await producer.ProduceAsync(topic, message);
}
catch (ProduceException<string,string> ex)
{
    Console.WriteLine($"Kafka publish error: {ex.Error.Reason}");
}

Consumer Errors

Possible failures:

• Deserialization errors • Business logic failures • External API failures

Best practice:

Use Dead Letter Topics (DLT) for failed messages.

Example: An Order Service publishes an OrderCreated event.

• Payment Service consumes it
• Inventory Service consumes it
• Notification Service consumes it

All independently.

Coding Example Architecture Diagram

Explain:

In this demo:

• Order API acts as the event producer • Kafka stores the event in the orders topic • Consumer service processes events asynchronously

Example Code

Let’s build a simple example:

Scenario: An Order API publishes an event to Kafka. A Background Worker consumes that event.

Prerequisites

• .NET 10+ SDK installed
• Kafka installed locally or via Docker
• Basic knowledge of ASP.NET Core
• Visual Studio / VS Code

To run Kafka using Docker:

docker run -p 29092:29092 apache/kafka

Required NuGet Packages

Install:

dotnet add package Confluent.Kafka

Package Used:

• Confluent.Kafka

Configuration

Add in appsettings.json:

{
  "Kafka": {
    "BootstrapServers": "localhost:29092",
    "Topic": "order-events",
    "GroupId": "order-consumer-group"
  }
}

Code Examples

1. Create Order Event Model

public class OrderCreatedEvent
{
    public string OrderId { get; set; }
    public string ProductName { get; set; }
    public double Price { get; set; }
}

2. Kafka Producer Service

using Confluent.Kafka;
using System.Text.Json;

public class KafkaProducer
{
    private readonly IProducer<Null, string> _producer;
    private readonly string _topic;

    public KafkaProducer(IConfiguration configuration)
    {
        var config = new ProducerConfig
        {
            BootstrapServers = configuration["Kafka:BootstrapServers"],
            Acks = Acks.All,
            EnableIdempotence = true,
            MessageSendMaxRetries = int.MaxValue,

            CompressionType = CompressionType.Zstd,
            LingerMs = 5,
            BatchSize = 64 * 1024

        };

        _producer = new ProducerBuilder<Null, string>(config).Build();
        _topic = configuration["Kafka:Topic"];
    }

    public async Task ProduceAsync(OrderCreatedEvent orderEvent)
    {
        var message = JsonSerializer.Serialize(orderEvent);

        await _producer.ProduceAsync(_topic, new Message<Null, string>
        {
            Value = message
        });
    }
}

Explain each setting

Acks = All

Producer waits until all replicas confirm.

Producer
   ↓
Leader Broker
   ↓
Replica 1
Replica 2

Safer but slightly slower.

EnableIdempotence = true

Prevents duplicate messages when retries happen.

Without:

Send OrderCreated
↓
Network timeout
↓
Retry
↓
Duplicate event

With idempotence:

Kafka stores only one copy.

CompressionType

Reduces network traffic.

100 MB
 ↓
20 MB

Faster and cheaper.

LingerMs

Waits a few milliseconds before sending.

Instead of:

1 message
1 request

1 message
1 request

Kafka batches:

100 messages
1 request

3. API Controller

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly KafkaProducer _producer;

    public OrdersController(KafkaProducer producer)
    {
        _producer = producer;
    }

    [HttpPost]
    public async Task<IActionResult> CreateOrder(OrderCreatedEvent order)
    {
        await _producer.ProduceAsync(order);
        return Ok("Order event published successfully");
    }
}

4. Kafka Consumer (Background Service)

using Confluent.Kafka;
using System.Text.Json;

public class KafkaConsumer : BackgroundService
{
    private readonly IConfiguration _configuration;

    public KafkaConsumer(IConfiguration configuration)
    {
        _configuration = configuration;
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = _configuration["Kafka:BootstrapServers"],
            GroupId = _configuration["Kafka:GroupId"],
            AutoOffsetReset = AutoOffsetReset.Earliest
        };

        var consumer = new ConsumerBuilder<Ignore, string>(config).Build();
        consumer.Subscribe(_configuration["Kafka:Topic"]);

        return Task.Run(() =>
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                var result = consumer.Consume(stoppingToken);
                var orderEvent = JsonSerializer.Deserialize<OrderCreatedEvent>(result.Message.Value);

                Console.WriteLine($"Order Received: {orderEvent.OrderId}");
            }
        }, stoppingToken);
    }
}

5. Register Services in Program.cs

builder.Services.AddSingleton<KafkaProducer>();
builder.Services.AddHostedService<KafkaConsumer>();

How It Works

• Client calls POST /api/orders
• API publishes event to Kafka topic
• Kafka stores event in partition
• Consumer reads event
• Business logic executes independently

Example Output

After running the ASP.NET Core API and Kafka consumer, we send a POST request:

POST /api/orders

Request Body:

{
  "orderId": "ORD-1001",
  "productName": "Laptop",
  "price": 75000
}

The following output confirms successful event publishing and consumption.

API Response

Order event published successfully

Consumer Console Output

Order Received: ORD-1001
Product: Laptop
Price: 75000

This confirms:

• Event was published
• Kafka stored the message
• Consumer processed the event successfully

All asynchronously. No direct service-to-service blocking.

The complete project is available on this **GitHub repository.**

Final Conclusion

Event-Driven Architecture with Kafka allows .NET applications to become more scalable, resilient, and loosely coupled.

By using Kafka as the event streaming platform, services can communicate asynchronously and handle massive workloads without tight dependencies.

As modern systems grow more distributed, adopting event-driven patterns becomes essential for building reliable and scalable applications.

Event-driven architectures require more than integrating Kafka into an application — they depend on thoughtful system design, resilient messaging patterns, and operational best practices. Simform helps organizations build scalable event-driven platforms that improve resilience, throughput, and long-term maintainability across distributed systems.

For more updates on the latest development trends, follow the Simform Engineering blog.

Follow Us: Twitter | LinkedIn