Real-Time Data Pipelines (Kafka + Stream Processing)

Real-time pipelines are built on an event-driven architecture.

Traditional systems:

• Request → Response (synchronous, tightly coupled)

Modern systems:

• Events → Continuous processing (asynchronous, loosely coupled)

An event represents a fact that has already occurred, not a command. For example, ORDER_CREATED is a record of something that happened, not an instruction to perform an action.

🚨 WANTED: TECH TALENT

💰 High Pay | 🌍 Remote | ⚡ Fast Hiring

Frontend • Backend • Full Stack • AI/ML • DevOps

**👉 APPLY NOW**

2. Real-World Scenario

Consider platforms like Swiggy or Zomato.

When a user places an order:

• Payment processing must begin
• Restaurant must be notified
• Delivery partner must be assigned
• Notifications must be sent
• Analytics must update dashboards

Instead of calling multiple services directly, the system emits a single event:

ORDER_CREATED

Different services independently consume this event and act accordingly. This removes tight coupling between services.

3. Role of Kafka

Apache Kafka acts as a distributed commit log.

Key characteristics:

• Immutable: events are not modified after being written
• Ordered: events are ordered within a partition
• Durable: events are stored on disk

Core concepts:

• Topic: logical category (e.g., orders)
• Partition: unit of parallelism
• Offset: position of an event in a partition

Kafka does not process data. It stores and delivers events.

4. Data Flow

The system follows this flow:

Order Service → Kafka → Stream Processing → Kafka → Consumers

Expanded view:

[Producer Services]
        ↓
Kafka (raw events)
        ↓
Stream Processing Layer
        ↓
Kafka (processed events)
        ↓
[Multiple Independent Consumers]

Each consumer operates independently without direct interaction with others.

5. Stream Processing

Stream processing systems continuously consume and process events.

Frameworks:

• Kafka Streams
• Apache Flink

Processing model:

while(true):
    read event
    process event

Unlike traditional services, stream processors do not wait for requests. They operate continuously.

6. State Management

Many use cases require maintaining state across events.

Example: “Number of orders per restaurant in the last 5 minutes”

This requires storing intermediate data:

Restaurant A → 120 orders
Restaurant B → 75 orders

State is:

• Stored locally for fast access
• Backed up in Kafka (changelog topics) for recovery

State enables:

• Aggregations
• Windowing
• Fraud detection
• Real-time analytics

7. Windowing

Windowing groups events based on time.

Types of windows:

• Tumbling window: fixed intervals (e.g., every 5 minutes)
• Sliding window: overlapping intervals
• Session window: based on user activity/inactivity

Windowing is essential for time-based computations.

8. Key Kafka Concepts

Partitioning

• Data is distributed across partitions
• Enables parallel processing

Trade-off:

• Ordering is guaranteed only within a partition

Offsets

Each event has a unique position:

offset 103 → ORDER_CREATED

Consumers track offsets to know what has been processed.

Replay Capability

Kafka allows re-reading past events.

Use cases:

• Rebuilding system state
• Fixing bugs and reprocessing data
• Debugging production issues

Fault Tolerance

If a consumer fails:

• Kafka retains data
• Consumer resumes from last committed offset

No data loss occurs, but duplicate processing is possible.

Processing Guarantees

• At most once
• At least once (most common)
• Exactly once (more complex)

9. Production Behavior

Traffic Spikes

• Kafka distributes load across partitions
• Multiple consumers process data in parallel

Consumer Failures

• events accumulate in Kafka
• Processing resumes when consumer recovers

Duplicate Events

Duplicates may occur due to retries.

Solutions:

• Idempotent processing
• Deduplication logic

10. When to Use

Use real-time pipelines when:

• Real-time processing is required
• System handles high event volume
• Multiple services depend on the same data stream

Examples:

• Fraud detection
• Live dashboards
• Recommendation systems

11. When Not to Use

Avoid when:

• System is simple CRUD-based
• Traffic is low
• Real-time processing is unnecessary

Using such architecture in these cases introduces unnecessary complexity.

12. Java Example (Kafka Streams)

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> stream = builder.stream("orders");
stream
    .filter((key, value) -> value.contains("ORDER_CREATED"))
    .to("processed-orders");
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

This simple code hides complex mechanisms:

• Partition assignment
• Offset tracking
• State management
• Fault recovery
• Parallel execution

13. Mental Model

• Kafka acts as a durable event log and transport layer
• Stream processors perform continuous computation
• Services consume events independently

This forms a scalable, fault-tolerant, and decoupled system.

Final Takeaways

• Real-time pipelines are event-driven, not request-driven
• Kafka functions as a durable log and buffer
• Stream processing enables continuous computation with state
• Partitioning enables scalability but affects ordering
• Failures do not cause data loss, but duplicates must be handled
• The system becomes more powerful but also more complex

Thank you for being a part of the community

Before you go:

👉 Be sure to clap and follow the writer ️👏️️

👉 Follow us: **Linkedin| [Medium](https://medium.com/codetodeploy)**

👉 CodeToDeploy Tech Community is live on Discord — **Join now!**

Disclosure: This post includes affiliate and partnership links.