Kafka Streams CDC Pipeline: Real-Time Change Data Capture at Scale

Introduction

In the modern data-driven enterprise, data is not static. Every transaction, every user action, every system event produces a change — and the ability to capture, propagate, and act on those changes in real time is a fundamental competitive differentiator. Change Data Capture (CDC) is the discipline of detecting and streaming data changes from source systems as they occur. Kafka Streams is Apache Kafka’s native stream processing library. Together, they form one of the most powerful, scalable, and operationally mature patterns for building real-time data pipelines. — -

What Is Change Data Capture?

CDC identifies and captures changes made to data in a source system — typically a relational database — and makes those changes available to downstream consumers in real time. A CDC system answers: what changed, when, and to what value? For every row in a database, CDC tracks three event types:

• INSERT: A new row was created
• UPDATE: An existing row was modified (with before and after values)
• DELETE: A row was removed

https://substackcdn.com/image/fetch/$s_!fUz2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65adea54-570e-4264-bed7-6b2ca3d0e8df_2456x1280.png

CDC Mechanisms: How Changes Are Captured

1. Log-Based CDC (Transaction Log Mining)

The gold standard of CDC. Every major relational database maintains a transaction log (write-ahead log / WAL). Log-based CDC tools read this log directly:

• PostgreSQL: Logical replication slots and pg_logical
• MySQL: Binary log (binlog) in ROW format
• Oracle: LogMiner
• SQL Server: SQL Server CDC feature

Debezium is the open-source leader — a Kafka Connect source connector that reads database transaction logs and produces Kafka events for every data change. Advantages: zero impact on source database, captures all changes including DELETEs, millisecond latency, no polling overhead.

2. Query-Based CDC (Timestamp Polling)

Periodically queries the source database for rows where updated_at is greater than the last poll time. Simple to implement but misses DELETE events, adds query load to the source DB, and latency is bounded by poll interval.

3. Trigger-Based CDC

Database triggers fire on INSERT/UPDATE/DELETE, writing change records to a shadow table. High database overhead; not recommended for high-volume production use.

Debezium + Kafka Connect: The CDC Foundation

The standard production architecture begins with Debezium running as a Kafka Connect source connector:

{
  "name": "postgres-cdc-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres-primary",
    "database.port": "5432",
    "database.user": "debezium",
    "database.dbname": "production_db",
    "database.server.name": "prod",
    "table.include.list": "public.orders,public.users,public.products",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_slot",
    "publication.name": "debezium_publication",
    "tombstones.on.delete": "true",
    "heartbeat.interval.ms": "10000"
  }
}

Debezium reads the PostgreSQL logical replication slot and produces messages to Kafka topics in format {server.name}.{schema}.{table} — e.g., prod.public.orders.

Debezium Event Schema

Each CDC event has a rich envelope:

{
  "before": { "id": 42, "status": "pending", "amount": 100.00 },
  "after":  { "id": 42, "status": "shipped", "amount": 100.00 },
  "source": {
    "connector": "postgresql",
    "ts_ms": 1706745600000,
    "table": "orders",
    "txId": 7291,
    "lsn": 24023120
  },
  "op": "u",
  "ts_ms": 1706745600123
}

• op: c (create), u (update), d (delete), r (read/snapshot)
• before: Row state before the change (null for inserts)
• after: Row state after the change (null for deletes)

Kafka Streams: Processing CDC Events

Kafka Streams provides the stream processing layer — transforming, enriching, aggregating, and routing change events to downstream systems.

Why Kafka Streams for CDC Processing?

• Native Kafka integration: Runs as a library in your application
• Stateful processing: Built-in RocksDB-backed state stores for joins and aggregations
• Exactly-once semantics: End-to-end EOS with Kafka transactions
• Fault tolerance: Automatic state recovery from changelog topics
• Scalability: Horizontally scalable via partition-based parallelism

Core Kafka Streams Topology for CDC

StreamsBuilder builder = new StreamsBuilder();
java
// Consume CDC events from Debezium topic
KStream cdcStream = builder
    .stream("prod.public.orders",
        Consumed.with(Serdes.String(), orderCdcSerde));
// Filter: only updates where status changed to 'shipped'
KStream shippedOrders = cdcStream
    .filter((key, event) ->
        "u".equals(event.getOp()) &&
        "shipped".equals(event.getAfter().getStatus()) &&
        !"shipped".equals(event.getBefore().getStatus()));
// Enrich with customer data via GlobalKTable join
GlobalKTable customerTable = builder
    .globalTable("customers",
        Consumed.with(Serdes.String(), customerSerde));
KStream enriched = shippedOrders
    .join(customerTable,
        (key, order) -> order.getAfter().getCustomerId(),
        (order, customer) -> new EnrichedShipmentEvent(order, customer));
// Route to notification topic
enriched.to("shipment-notifications",
    Produced.with(Serdes.String(), enrichedSerde));
// Aggregate: count shipments per region per hour
enriched
    .groupBy((key, event) -> event.getCustomer().getRegion())
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
    .count()
    .toStream()
    .to("shipment-counts-by-region-hourly");

Schema Management with Schema Registry

CDC events carry evolving schemas — as database tables gain or lose columns, event schemas must evolve without breaking consumers. Confluent Schema Registry manages this:

• Schemas registered in Avro, Protobuf, or JSON Schema format
• Compatibility rules (BACKWARD, FORWARD, FULL) enforced on schema evolution
• Debezium integrates natively with Schema Registry

Exactly-Once Semantics

In a CDC pipeline, duplicate processing can lead to double-counted analytics, duplicate notifications, and inconsistent replicated data. Kafka Streams provides exactly-once semantics (EOS):

Properties config = new Properties();
config.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
    StreamsConfig.EXACTLY_ONCE_V2);
config.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3);
config.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);

With EOS enabled, Kafka Streams uses transactional producers — all output records and state store updates for a batch are committed atomically.

Handling the Initial Snapshot

When a CDC connector starts for the first time, it captures the current state of the database before streaming ongoing changes. Debezium handles this by:

• Acquiring a consistent read snapshot of source tables
• Emitting r (read) events for every existing row
• Switching to streaming from the transaction log

Incremental snapshotting (Debezium 1.6+) addresses large tables — chunking the snapshot into smaller interleaved reads that don’t block ongoing CDC streaming. — -

Production Architecture

PostgreSQL Primary
    | (logical replication slot)
    v
Debezium Connector (Kafka Connect cluster)
    | (Avro events)
    v
Kafka Topics (prod.public.orders, prod.public.users, ...)
    |
    +-- Kafka Streams App 1: Order Processing Pipeline
    |       +-- Elasticsearch (search index updates)
    |       +-- notification-events topic
    |
    +-- Kafka Streams App 2: Analytics Aggregation
    |       +-- ClickHouse / BigQuery (real-time analytics)
    |
    +-- Kafka Streams App 3: Cache Invalidation
            +-- Redis PUBLISH (cache invalidation signals)

Monitoring and Observability

Common Pitfalls and Solutions

Replication slot bloat: If the Debezium connector stops, the PostgreSQL replication slot accumulates WAL. Set max_slot_wal_keep_size and monitor slot lag aggressively. Schema evolution failures: Always test schema changes against Schema Registry compatibility rules in staging before production deployment. Large transactions: A single database transaction producing millions of rows creates a burst of CDC events. Use Kafka Streams' built-in backpressure mechanisms. Tombstone handling: DELETE events produce tombstone records (null value). Ensure all consumers handle null values gracefully.

Conclusion

The Kafka Streams CDC pipeline is one of the most battle-tested patterns in modern data engineering. By combining Debezium’s log-based CDC with Kafka’s durable, scalable event backbone and Kafka Streams’ rich stateful processing, teams can build real-time data pipelines that are low-latency, exactly-once, fault-tolerant, and horizontally scalable. Mastering this stack is an essential skill for any engineer building event-driven, data-intensive systems.

Sources:

• 1. Apache Kafka. (2024). Kafka Streams Developer Guide. https://kafka.apache.org/documentation/streams/
• 2. Debezium Authors. (2024). Debezium Documentation: CDC for PostgreSQL, MySQL, MongoDB. https://debezium.io/documentation/
• 3. Narkhede, N., Shapira, G., & Palino, T. (2021). Kafka: The Definitive Guide, 2nd Edition. O’Reilly Media. https://www.oreilly.com/library/view/kafka-the-definitive/9781492043072/
• 4. Confluent. (2024). Schema Registry Documentation. https://docs.confluent.io/platform/current/schema-registry/
• 5. Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media. https://dataintensive.net/
• 6. PostgreSQL Global Development Group. (2024). Logical Replication Documentation. https://www.postgresql.org/docs/current/logical-replication.html