Building .NET 8 Microservices with Kafka (Complete Step‑by‑Step Guide)

Event‑Driven Architecture with ProductAPI & CustomerAPI

In this tutorial, we will build a complete .NET 8 microservices system using:

• .NET 8 Web API
• SQL Server
• Entity Framework Core
• Apache Kafka
• Docker
• Kafka UI

We will create:

• ProductAPI → Publishes ProductCreated event
• CustomerAPI → Consumes ProductCreated event

Architecture Overview

ProductAPI saves data and publishes an event. CustomerAPI listens to Kafka and reacts asynchronously.

Prerequisites

Install the following:

• .NET 8 SDK
• Visual Studio 2022 or VS Code
• SQL Server (Express or full)
• Docker Desktop (with WSL2 enabled on Windows)

Create Microservices

dotnet new webapi -n ProductAPI
dotnet new webapi -n CustomerAPI

Add EF Core + SQL Server

Inside both APIs:

dotnet add package Microsoft.EntityFrameworkCore.SqlServer
dotnet add package Microsoft.EntityFrameworkCore.Tools

ProductAPI Setup

Product Model

public class Product
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
}

ProductDbContext

public class ProductDbContext : DbContext
{
    public ProductDbContext(DbContextOptions<ProductDbContext> options)
        : base(options) { }

public DbSet<Product> Products { get; set; }

}

Register DbContext in Program.cs

builder.Services.AddDbContext<ProductDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection")));

appsettings.json

"ConnectionStrings": {
  "DefaultConnection": "Server=localhost;Database=ProductDb;Trusted_Connection=True;TrustServerCertificate=True;"
}

ProductController

[ApiController]
[Route("api/[controller]")]
public class ProductController : ControllerBase
{
  private readonly ProductDbContext _context;

  public ProductController(ProductDbContext context)
    {
        _context = context;
    }

    [HttpPost]
    public async Task<IActionResult> Create(Product product)
    {
        product.Id = Guid.NewGuid();
        _context.Products.Add(product);
        await _context.SaveChangesAsync();
        return Ok(product);
    }
}

CustomerAPI Setup

Customer Model

public class Customer
{
    public Guid Id { get; set; }
    public string Name { get; set; }
}

CustomerDbContext

public class CustomerDbContext : DbContext
{
    public CustomerDbContext(DbContextOptions<CustomerDbContext> options)
        : base(options) { }
    public DbSet<Customer> Customers { get; set; }
}

Register DbContext exactly like ProductAPI.

Setup Kafka with Docker

Create docker-compose.yml:

services:
zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    container_name: zookeeper
    restart: always
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    container_name: kafka
    restart: always
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "9093:9093"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_HOST://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    restart: always
    depends_on:
      - kafka
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

Run:

docker compose up -d

Open Kafka UI at:

http://localhost:8080

Install Kafka Package

Inside both APIs:

dotnet add package Confluent.Kafka

ProductAPI →Add Kafka Producer

Kafka Settings

"KafkaSettings": {
  "BootstrapServers": "localhost:9093",
  "ProductCreatedTopic": "product-created-topic"
}

KafkaSettings Class

public class KafkaSettings
{
    public string BootstrapServers { get; set; }
    public string ProductCreatedTopic { get; set; }
}

KafkaProducer Service

using Confluent.Kafka;
using System.Text.Json;

public class KafkaProducer
{
    private readonly IProducer<string, string> _producer;
    private readonly KafkaSettings _settings;
    public KafkaProducer(IOptions<KafkaSettings> options)
    {
        _settings = options.Value;
        var config = new ProducerConfig
        {
            BootstrapServers = _settings.BootstrapServers
        };
        _producer = new ProducerBuilder<string, string>(config).Build();
    }
    public async Task ProduceAsync<T>(string topic, string key, T message)
    {
        var json = JsonSerializer.Serialize(message);
        await _producer.ProduceAsync(topic, new Message<string, string>
        {
            Key = key,
            Value = json
        });
    }
}

Register in Program.cs:

builder.Services.Configure<KafkaSettings>(
    builder.Configuration.GetSection("KafkaSettings"));

builder.Services.AddSingleton<KafkaProducer>();

ProductCreatedEvent

public class ProductCreatedEvent
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
}

Publish Event in Controller

After saving product:

await _kafkaProducer.ProduceAsync(
    _kafkaSettings.Value.ProductCreatedTopic,
    product.Id.ToString(),
    new ProductCreatedEvent
    {
        Id = product.Id,
        Name = product.Name,
        Price = product.Price
    });

CustomerAPI — Add Kafka Consumer

Kafka Settings

"KafkaSettings": {
  "BootstrapServers": "localhost:9093",
  "GroupId": "customer-group",
  "ProductCreatedTopic": "product-created-topic"
}

ProductCreatedConsumer

using Confluent.Kafka;
using System.Text.Json;
public class ProductCreatedConsumer : BackgroundService
{
    private readonly KafkaSettings _settings;

    public ProductCreatedConsumer(IOptions<KafkaSettings> options)
    {
        _settings = options.Value;
    }
    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var config = new ConsumerConfig
        {
            BootstrapServers = _settings.BootstrapServers,
            GroupId = _settings.GroupId,
            AutoOffsetReset = AutoOffsetReset.Earliest
        };
        return Task.Run(() =>
        {
            using var consumer = new ConsumerBuilder<string, string>(config).Build();
            consumer.Subscribe(_settings.ProductCreatedTopic);
            while (!stoppingToken.IsCancellationRequested)
            {
                var result = consumer.Consume(stoppingToken);
                var productEvent = JsonSerializer.Deserialize<ProductCreatedEvent>(result.Message.Value);
                Console.WriteLine($"Received Product: {productEvent.Name}");
            }
        }, stoppingToken);
    }
}

Register in Program.cs:

builder.Services.AddHostedService<ProductCreatedConsumer>();

Test End‑to‑End

1. Run Docker Compose
2. Run ProductAPI
3. Run CustomerAPI
4. Send POST request to ProductAPI

You should see:

• Message inside Kafka UI
• CustomerAPI console logging the event

Production Recommendations

• Use manual offset commits
• Implement Outbox Pattern for reliability
• Use message keys for partitioning
• Add retry & error handling
• Monitor Kafka with proper observability tools

Conclusion

You have successfully built:

• Two .NET 8 microservices
• SQL Server integration
• Kafka producer & consumer
• Event‑driven architecture
• Kafka monitoring with UI

Source Code: https://github.com/hafizasad072/Microservices

This is the foundation of scalable, production ready microservices systems.