Designing Event-Driven Architecture in Ruby on Rails with Kafka

Building scalable, resilient, and loosely coupled Rails applications using Apache Kafka.

Introduction

As applications grow, the traditional request-response architecture starts showing its limitations. Features such as sending emails, updating analytics, processing payments, syncing third-party services, and generating reports often become tightly coupled within a single request.

This leads to:

• Slow API responses
• Tightly coupled services
• Difficult deployments
• Poor scalability
• Reduced fault tolerance

This is where Event-Driven Architecture (EDA) shines.

Instead of directly calling another service, your application publishes an event. Any interested service can subscribe to that event and react independently.

In this article, we’ll build a production-ready Event-Driven Architecture using Ruby on Rails and Apache Kafka, exploring real-world patterns, implementation techniques, and best practices.

What is Event-Driven Architecture?

Event-Driven Architecture is a software design pattern where components communicate by producing and consuming events rather than invoking each other directly.

Imagine an e-commerce application.

When an order is placed:

Customer
    │
    ▼
Rails API
    │
    ▼
Order Created

Instead of calling multiple services synchronously:

Order Service
    │
    ├── Email Service
    ├── Inventory Service
    ├── Analytics Service
    ├── Loyalty Service
    └── Notification Service

The Order Service simply publishes:

order.created

Then Kafka distributes the event to every interested consumer.

Kafka

order.created
     │
     ├── Email Consumer
     ├── Inventory Consumer
     ├── Analytics Consumer
     ├── Notification Consumer
     └── Billing Consumer

Each consumer works independently.

Why Kafka?

Apache Kafka is one of the most popular distributed event streaming platforms.

Benefits include:

• High throughput
• Fault tolerance
• Event persistence
• Horizontal scalability
• Ordered events
• Consumer groups
• Replay capability

It’s widely used by companies like LinkedIn, Uber, Netflix, Airbnb, and many large-scale SaaS platforms.

System Architecture

+------------------+
                | Ruby on Rails API|
                +------------------+
                        |
                        |
                Publish Events
                        |
                        ▼
                +------------------+
                |      Kafka       |
                +------------------+
                 |      |      |
        ----------      |      ----------
       ▼                ▼               ▼
Inventory       Email Service     Analytics
 Payment         Notification      CRM
 Search          Reporting         Billing

Setting Up Kafka

The easiest way to start locally is using Docker Compose.

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

Start Kafka.

docker compose up

Installing ruby-kafka

Add the gem.

gem 'ruby-kafka'

Install.

bundle install

Configuring Kafka

Create:

config/initializers/kafka.rb

KAFKA = Kafka.new(
  ["localhost:9092"],
  client_id: "rails_app"
)

Now Kafka is available globally.

Creating an Event Publisher

Instead of publishing events everywhere, create a reusable publisher.

app/services/event_publisher.rb

class EventPublisher
  def self.publish(topic, payload)
    producer = KAFKA.async_producer
    producer.produce(
      payload.to_json,
      topic: topic
    )
    producer.deliver_messages
  end
end  

Simple.

Reusable.

Testable.

Publishing Events

Suppose a customer places an order.

class OrdersController < ApplicationController
  def create
    order = Order.create!(order_params)
    EventPublisher.publish(
      "order.created",
      {
        order_id: order.id,
        customer_id: order.customer_id,
        total: order.total
      }
    )
    render json: order
  end
end

Notice the controller doesn’t know who will consume the event.

This is loose coupling.

Building a Consumer

Consumers subscribe to topics.

consumer = KAFKA.consumer(
  group_id: "inventory-service"
)
consumer.subscribe("order.created")

Process messages.

consumer.each_message do |message|
data = JSON.parse(message.value)
  Inventory.reserve_stock(
    data["order_id"]
  )
end

Inventory is completely independent from Orders.

Creating Multiple Consumers

The same event can power multiple systems.

order.created
│
        │
        ▼
Inventory
Email
Analytics
CRM
Notification
Fraud Detection
Billing

None of these services know about each other.

Event Versioning

Events evolve over time.

Bad practice:

{
  "order_id": 12
}

Better:

{
  "version": 2,
  "event": "order.created",
  "data": {
    "order_id": 12,
    "customer_id": 55,
    "total": 399
  }
}

Versioning prevents breaking older consumers.

Event Naming Convention

A good naming strategy is:

entity.action

Examples:

user.created
user.deleted
payment.completed
payment.failed
order.shipped
invoice.generated

Keep names meaningful and immutable.

Handling Failures

Consumers should never crash because of one bad event.

Wrap processing.

consumer.each_message do |message|
begin
    process(message)
  rescue => e
    Rails.logger.error(e)
  end
end

Production systems should also use:

• Dead Letter Queues (DLQ)
• Retry mechanisms
• Alerting
• Monitoring

Idempotency

Consumers may receive duplicate events.

Never assume Kafka delivers exactly once.

Bad:

Order.update(status: "paid")

Better:

return if Payment.exists?(event_id: event_id)
Payment.create!(...)

Always make event processing idempotent.

Partitioning

Kafka distributes messages into partitions.

Orders
Partition 1
Partition 2
Partition 3
Partition 4

Messages sharing the same key remain ordered.

producer.produce(
  payload,
  topic: "orders",
  partition_key: order.id
)

Ordering is preserved for each order.

Consumer Groups

Consumer groups enable horizontal scaling.

Inventory Group
Consumer A
Consumer B
Consumer C

Kafka automatically balances partitions across consumers.

This allows processing millions of events efficiently.

Retry Strategy

Avoid immediate retries.

Instead:

Main Topic
↓
Retry Topic
↓
Retry Topic 2
↓
Dead Letter Queue

This prevents endless processing loops and gives operators a safe place to inspect failed events.

Monitoring Kafka

Track key metrics:

• Consumer lag
• Failed messages
• Processing time
• Throughput
• Retry count
• Topic size
• Broker health

Tools commonly used:

• Datadog
• Prometheus
• Grafana
• Kafka UI

Monitoring is essential for production systems.

Security Best Practices

Always secure Kafka clusters.

Recommended practices:

• TLS encryption
• SASL authentication
• ACL authorization
• Separate topics by domain
• Encrypt sensitive payloads
• Never publish passwords or tokens

Events should contain only the data required by consumers.

Testing Event Publishing

RSpec makes event testing straightforward.

describe EventPublisher do

  it "publishes an event" do
    producer = instance_double("Kafka::AsyncProducer")
    allow(KAFKA).to receive(:async_producer)
      .and_return(producer)
    expect(producer)
      .to receive(:produce)
    EventPublisher.publish(
      "orders",
      { id: 1 }
    )
  end
end

This keeps tests fast and isolated from Kafka.

Common Mistakes

Avoid these pitfalls:

• Publishing entire ActiveRecord objects
• Creating too many topics
• Ignoring schema evolution
• Missing idempotency
• Blocking consumers with slow operations
• Forgetting retries
• Not monitoring consumer lag
• Treating Kafka as a request-response system

Production Tips

• Keep events small and focused.
• Use meaningful topic names.
• Version event payloads.
• Design consumers to be idempotent.
• Separate producers and consumers into dedicated services when scaling.
• Monitor lag, retries, and throughput continuously.
• Document your event contracts and treat them as public APIs.

Conclusion

Event-Driven Architecture transforms the way Rails applications scale. By introducing Kafka as an event backbone, you decouple services, improve resilience, and enable independent evolution of business capabilities.

Whether you’re processing orders, sending notifications, synchronizing external systems, or analyzing user behavior, Kafka provides a robust foundation for asynchronous communication. Combined with Ruby on Rails, it allows teams to build systems that are easier to maintain, simpler to scale, and better prepared for the demands of modern distributed applications.

As your application grows, adopting event-driven patterns isn’t just an optimization — it’s an architectural investment that pays dividends in flexibility, reliability, and long-term scalability.

Happy coding!