Using Debezium to Move Change Data from Azure Database for PostgreSQL into Databricks

CDC is the difference between stale snapshots and an actually usable lakehouse

A lot of modern data platforms still move operational data the old way: nightly extracts, periodic full refreshes, and brittle batch jobs that repeatedly copy the same rows. That approach is expensive, slow, and badly matched to how product and operational systems actually change.

If your source of truth is Azure Database for PostgreSQL and your analytics or AI estate runs on Databricks, the better pattern is usually change data capture (CDC). Instead of repeatedly re-reading whole tables, CDC lets you propagate inserts, updates, and deletes as they happen by reading the database’s write-ahead log.

This is where Debezium becomes useful.

Debezium is an open-source CDC platform built on Kafka Connect. It reads database transaction logs, converts row-level changes into ordered event streams, and publishes them to downstream consumers. In a Databricks architecture, that gives you a clean way to move from operational Postgres tables to Bronze, Silver, and downstream curated datasets without relying on wasteful bulk reloads.

The core idea is simple:

Let PostgreSQL produce the truth about what changed, let Debezium serialize those changes reliably, and let Databricks ingest that stream into Delta tables designed for replay, lineage, and incremental transformation.

That pattern is significantly better than trying to infer changes from timestamps or repeatedly diff large snapshots.

What the architecture actually looks like

At a high level, the flow usually looks like this:

1. Application writes data into Azure Database for PostgreSQL.
2. PostgreSQL records those changes in its write-ahead log (WAL).
3. Debezium reads the WAL using PostgreSQL logical replication.
4. Debezium emits change events into Kafka topics.
5. Databricks consumes those Kafka topics using Structured Streaming or a managed ingestion path.
6. Raw CDC events land in Bronze Delta tables.
7. Merge logic applies inserts, updates, and deletes into Silver tables that represent the latest business state.

That sequence matters because each layer has a separate responsibility:

• PostgreSQL is the transactional system of record.
• Debezium is the CDC extraction and serialization layer.
• Kafka is the durable transport and replay buffer.
• Databricks is the storage, transformation, and serving layer.

Trying to collapse those responsibilities into one job usually creates operational pain later.

Why Debezium fits this use case well

There are other ways to move data from Postgres into a lakehouse. Some teams use Azure-native ETL tools. Some use SaaS replication products. Some build custom pollers. Debezium tends to stand out when the requirement is high-fidelity, low-latency, replayable CDC with open control over the data path.

It is particularly strong for this Azure-to-Databricks pattern because it gives you:

• Log-based capture rather than query-based polling.
• Ordered change events with transaction context.
• Delete propagation, which many ad hoc pipelines handle badly.
• Schema change metadata, depending on how you configure converters and schema history.
• Replayability, because Kafka can retain the event stream independently of the target lakehouse state.
• Decoupling, because multiple downstream consumers can subscribe to the same operational changes.

That last point is often underrated. Once Debezium is in place, the same change stream can feed Databricks, operational caches, fraud systems, search indexes, or monitoring pipelines without adding new load to PostgreSQL.

The Azure PostgreSQL constraint that matters most: logical replication

The entire design depends on PostgreSQL exposing changes through logical replication. Debezium does not read tables by repeatedly querying them. It reads from the WAL using a replication slot and a publication.

That means the first design question is not “how do we connect Debezium to Databricks?” It is this:

Is your Azure Database for PostgreSQL deployment configured and permitted to support logical replication in the way Debezium needs?

In practice, you need to validate:

• the Azure PostgreSQL deployment model you are using
• whether logical replication is enabled
• whether WAL settings support the expected CDC load
• whether you can create replication users, publications, and slots
• network connectivity between the Debezium runtime and Azure Postgres

Depending on whether you are using Flexible Server and how tightly the environment is governed, these controls may require coordination with platform, security, and database administrators.

This is also where many CDC projects fail early. Teams start from the sink side, talk about Delta tables and dashboards, and only later discover that the source database has not been prepared for sustained logical replication.

Debezium event structure is not the same thing as a target table

One of the most important implementation details is that Debezium emits events, not ready-made analytics tables.

A typical Debezium PostgreSQL change event contains:

• the before row image
• the after row image
• an operation code such as create, update, delete, or snapshot read
• source metadata such as database, schema, table, LSN, and timestamp
• optional transaction metadata

Conceptually, the event looks like this:

{
  "before": { "id": 42, "status": "pending" },
  "after": { "id": 42, "status": "approved" },
  "op": "u",
  "ts_ms": 1710000000000,
  "source": {
    "db": "appdb",
    "schema": "public",
    "table": "orders",
    "lsn": 123456789
  }
}

That is ideal for a Bronze layer because it preserves the change history and replay context. It is not ideal as the final shape for downstream analysts or ML feature pipelines.

You usually want two distinct landing patterns in Databricks:

• Bronze CDC tables that store the raw Debezium envelope or a lightly normalized version of it.
• Silver current-state tables that apply CDC semantics and present one latest row per business key.

If you skip Bronze and only try to maintain current-state tables, you lose auditability and make backfills, debugging, and reprocessing much harder.

Snapshot plus streaming is usually the right starting mode

When Debezium starts against an existing production database, it typically needs to do two jobs:

1. Take an initial snapshot of existing table contents.
2. Continue streaming new WAL changes from that point forward.

That startup behavior is important because it determines how you bootstrap Databricks without missing rows or duplicating state. Debezium supports multiple snapshot modes, and the right choice depends on the operational constraints of the source system.

For most teams, the principle is straightforward:

• use the initial snapshot to create a baseline
• use the change stream to keep the lakehouse current
• store enough metadata in Databricks to deduplicate and apply events deterministically

The biggest mistake here is treating the snapshot as one-time batch load logic and the CDC stream as a completely separate pipeline. They should land into the same logical ingestion model so that downstream merge logic does not care whether a row originated from the initial snapshot or a later update.

The Databricks side: design for replay, not just ingestion

Databricks is a strong target for Debezium CDC because Delta Lake gives you transactional storage, streaming-friendly ingestion, and efficient merge semantics. But the design only works well if you treat CDC as a replayable event stream, not just a faster copy job.

The Bronze layer should generally retain:

• the business payload
• the Debezium operation type
• event timestamp
• source table metadata
• ordering metadata such as LSN or equivalent source position
• ingestion timestamp

That enables three critical downstream behaviors:

• reconstructing event history
• reprocessing target tables after logic changes
• debugging mismatches between source and lakehouse state

From there, Silver transformations apply business keys and CDC rules. For example:

• op = c or op = r means insert current row state
• op = u means update current row state
• op = d means mark deleted or physically delete, depending on your serving pattern

If you need type-2 history, you can derive that from the Bronze event stream as well, but do not confuse that requirement with the mechanics of raw CDC ingestion.

A practical reference pattern

In many real implementations, the stack looks something like this:

• Azure Database for PostgreSQL as source
• Kafka Connect running Debezium on AKS, Kubernetes, or another managed runtime
• Kafka hosted through Confluent Cloud, Azure Event Hubs for Kafka, or a self-managed cluster
• Databricks Structured Streaming reading Kafka topics
• Delta Bronze tables for raw CDC events
• Delta Silver tables maintained with MERGE or streaming upsert logic

That is not the only valid architecture, but it is a sensible one because each component does one thing well.

A simplified Kafka Connect connector configuration might look like this:

{
  "name": "postgres-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "your-postgres-host.postgres.database.azure.com",
    "database.port": "5432",
    "database.user": "debezium_user",
    "database.password": "***",
    "database.dbname": "appdb",
    "topic.prefix": "appdb",
    "plugin.name": "pgoutput",
    "publication.autocreate.mode": "filtered",
    "slot.name": "debezium_appdb",
    "table.include.list": "public.orders,public.customers",
    "snapshot.mode": "initial",
    "tombstones.on.delete": "false"
  }
}

This is illustrative rather than production-ready, but it highlights the main moving parts:

• source connectivity
• logical replication plugin
• replication slot naming
• publication behavior
• table scoping
• snapshot strategy

On the Databricks side, the raw stream consumer typically parses the Debezium envelope and writes it into Delta. The exact code depends on your serialization format, but the conceptual pattern is stable:

raw = (
  spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", kafka_bootstrap)
    .option("subscribe", "appdb.public.orders")
    .load()
)

bronze = parse_debezium_payload(raw)

(
  bronze.writeStream
    .format("delta")
    .option("checkpointLocation", checkpoint_path)
    .outputMode("append")
    .start(bronze_table_path)
)

The important part is not the syntax. It is the contract: Kafka stores the ordered CDC events, and Databricks persists them in a Delta format that supports downstream incremental processing.

Deletes are where weak CDC designs get exposed

A lot of replication pipelines look correct until the first serious delete workload arrives.

This is one reason Debezium is valuable. It carries explicit delete semantics rather than forcing downstream systems to guess whether a missing row means “deleted,” “not yet synced,” or “filtering bug.”

But that does not mean delete handling is automatic. You still have to choose the serving behavior in Databricks:

• hard delete rows from Silver tables
• soft delete rows with an is_deleted flag
• maintain both current-state and full-history representations

The right answer depends on downstream use cases. BI serving often wants current-state tables. Compliance, audit, and feature lineage often benefit from retaining historical change records. The important thing is to decide this explicitly rather than letting deletes disappear somewhere between Kafka and Delta.

Ordering and idempotency matter more than raw throughput

Teams often obsess over message volume first. In practice, the harder problems are usually event ordering, deduplication, and deterministic merge behavior.

Postgres transactions can update the same row multiple times. Network or consumer retries can cause reprocessing. Streaming jobs can restart. Schema changes can arrive at awkward moments. None of that is unusual. It is normal.

So your Databricks merge logic should be designed around:

• a stable business primary key
• source ordering metadata such as LSN and event timestamp
• idempotent application of repeated events
• explicit handling for out-of-order edge cases

If those controls are weak, your target tables may silently drift from source truth even while all jobs show green.

Schema evolution needs an opinionated policy

Debezium can surface schema changes, but that does not eliminate the need for downstream governance. In a real production environment, source schemas evolve in messy ways:

• columns are added
• types are widened
• nullability changes
• old columns become semantically dead before they are physically dropped

If Databricks is ingesting CDC into Bronze and then promoting into Silver, you need to define what happens when source schemas evolve:

• which changes can auto-flow into Bronze
• which changes require review before promotion into Silver
• how contracts are communicated to downstream consumers
• how backfills are handled when business meaning changes, not just physical structure

The worst approach is pretending that CDC removes schema management as a concern. It does not. It just makes schema drift visible faster.

Operational risks you should plan for early

Debezium is robust, but this architecture still has failure modes that need active ownership.

The main ones are:

• Replication slot lag: if consumers stall, PostgreSQL may retain WAL longer than expected.
• Kafka retention mismatch: if retention is too short, replay windows disappear.
• Checkpoint corruption or drift on the Databricks streaming side.
• Source failover behavior in managed Postgres environments.
• Schema incompatibility between source changes and downstream parsing logic.
• Backpressure when large snapshots or table rewrites collide with regular change volume.

These are not reasons to avoid CDC. They are reasons to treat the pipeline as a production data product rather than a one-off integration.

In other words:

Debezium reduces the complexity of extracting correct change events. It does not remove the need for disciplined platform operations around transport, storage, and consumption.

Security and governance are part of the design, not an add-on

Because the source is a managed operational database and the target is a shared analytics platform, this pattern touches multiple control boundaries. You should think about:

• least-privilege access for replication users
• private networking between Azure Postgres, Kafka infrastructure, and Databricks
• secret management for connector credentials
• encryption in transit and at rest
• topic- and table-level access policies
• data classification and masking for sensitive columns

Many organizations discover too late that the technical CDC flow is easy compared with the governance work around personally identifiable information, residency constraints, and downstream entitlement controls.

If the destination lakehouse will support AI or feature engineering workloads, these controls become even more important because data tends to spread quickly once it becomes easy to consume.

When this pattern is a good fit

Using Debezium from Azure Database for PostgreSQL into Databricks is usually a strong choice when:

• you need near-real-time propagation of operational changes
• deletes and updates must be preserved accurately
• you want an open, replayable architecture rather than a black-box replication service
• multiple downstream systems may eventually consume the same CDC stream
• Databricks is already your main transformation and storage platform

It is less attractive when the workload is tiny, latency does not matter, or your team is unwilling to operate Kafka Connect and streaming infrastructure responsibly. In those cases, a simpler managed replication path may be more appropriate.

The right question is not whether Debezium is fashionable. It is whether your organization benefits from log-based, replayable CDC as a platform capability.

The real value is not just freshness

People often justify CDC by saying they want fresher dashboards. That is true, but it undersells the architectural value.

The deeper benefit is that Debezium turns operational database changes into a governed event stream that Databricks can store, replay, transform, audit, and repurpose. That is much more powerful than simply landing data faster.

Once you have that pattern working well, you can:

• build low-latency analytical views
• maintain current-state Delta tables efficiently
• preserve row-level history for compliance and debugging
• feed feature pipelines without repeated full extraction
• let additional consumers subscribe to the same source-of-truth changes

That is the real reason this architecture matters.

It is not just about moving rows from Postgres to a data lake. It is about turning database mutations into a durable, reusable data product that the rest of the platform can trust.

Bottom line

If you want to propagate changes from Azure Database for PostgreSQL into a Databricks lakehouse, Debezium is one of the cleanest ways to do it.

It gives you log-based CDC from PostgreSQL, durable event transport through Kafka, and a natural ingestion pattern into Delta tables where Databricks can apply replayable, auditable incremental transformations.

The engineering challenge is not whether the pieces can connect. They can. The real challenge is designing the pipeline with the right operational discipline around logical replication, ordering, deletes, schema evolution, retention, and downstream merge semantics.

Teams that get those details right end up with something much better than a sync job. They get a modern data movement backbone.