Five Mistakes We Made Integrating Apache Kafka with Apache Hudi — and How We Fixed Them

By Sinchan Banerjee — Director of Enterprise Data Architecture | Author, Scalable Data Architecture with Java (Packt)

I have built streaming pipelines for most of my career. I am known for building robust ones. But robustness isn’t something you arrive at on day one — it’s something you earn through the mistakes you make, diagnose, and fix in production.

This article is about five of those mistakes.

The context: we built a large-scale medallion lakehouse on AWS S3 using Apache Hudi, ingesting CDC events from two very different source worlds — relational RDBMS systems and hierarchical IMS database segments on mainframe. All events flowed through Apache Kafka into raw Hudi MOR (Merge-on-Read) tables, one Kafka topic per source table, one Hudi table per topic. Ingestion was handled by the Hudi DeltaStreamer utility running on EMR clusters.

I won’t go deep into the architecture — that’s a separate article. What I want to focus on are the five production mistakes we made during this integration, why they happened, and exactly how we fixed them. If you’re building anything similar, I hope this saves you weeks of painful debugging.

Mistake 1: Running All Streamers on a Single EMR Cluster

What we did

In the early days, every DeltaStreamer instance — one per source table — ran on a single shared EMR cluster. It felt operationally clean. One cluster to manage, one place to look when something went wrong.

What actually happened

The cluster was chronically overloaded. Resource contention between streamer instances meant that high-volume tables starved low-volume ones, and low-volume tables wasted allocated resources during quiet periods. Worse, we hadn’t accounted for the fact that many source tables produced burst updates — large volumes of CDC events at irregular intervals — rather than a constant, predictable stream. A cluster sized for average load got hammered during burst windows.

How we fixed it

We separated the workloads by their update pattern:

For burst tables — tables that received large, irregular volumes of CDC events — we moved to HoodieMultiTableStreamer, running on a transient EMR cluster triggered every hour. The cluster spins up, processes the backlog, and shuts down. No idle compute, no wasted cost. This one change delivered significant cost savings.

For constant-stream tables — tables with consistent, predictable CDC volumes — we split the workload across two dedicated EMR clusters, grouped by data volume and load profile. Each cluster was right-sized for its specific workload, and resource contention disappeared.

The lesson: don’t let operational convenience drive infrastructure decisions. Understand your CDC event patterns — burst versus constant — before you design your cluster topology. They are fundamentally different workloads and they need different solutions.

Mistake 2: Two Kafka Topics Writing to the Same Hudi Table Concurrently

What we did

One source table was receiving messages from two different Kafka topics simultaneously — a design decision made for reasons I won’t relitigate here. Both streams were being ingested into the same Hudi table in parallel by two separate DeltaStreamer instances.

What actually happened

Data corruption. Specifically, corruption of the latest version of the .hoodie/metadata.json file due to lock contention during concurrent writes. Hudi's default locking mechanism was not designed for two independent writers hitting the same table simultaneously. The result was a race condition — both writers would read the same metadata state, both would attempt to commit, and the second commit would silently corrupt or overwrite the first.

This was one of the more frustrating bugs to diagnose because it was intermittent. The corruption didn’t happen on every write — only when the timing was unlucky enough for both streamers to be in the commit phase simultaneously.

How we fixed it

We switched the concurrency control mode to Optimistic Concurrency Control (OCC) and replaced the default lock provider with the StorageBasedLockProvider:

hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider

The StorageBasedLockProvider is worth understanding. Unlike lock providers that depend on external infrastructure (ZooKeeper, DynamoDB), it manages concurrency directly on the storage layer — using the .hoodie/ directory on S3 or GCS as the locking mechanism. No external dependencies, no additional infrastructure to manage. For S3-based lakehouses, it's the cleanest solution.

With OCC and StorageBasedLockProvider in place, concurrent writers could proceed optimistically and the lock provider would detect and resolve conflicts at commit time — no more silent metadata corruption.

Mistake 3: Inline Compaction Racing with Active Writes

What we did

We were running MOR tables, which means compaction — the process of merging delta log files into base Parquet files — is a necessary part of the lifecycle. By default, Hudi enables inline compaction, which triggers compaction automatically as part of the write path.

What actually happened

Metadata corruption — again, but this time for a different reason. When a new write began while an inline compaction was already in progress on the same table, the two operations would interfere with each other’s view of the table state. The result was a corrupted metadata state that required manual intervention to recover.

Inline compaction sounds convenient — it keeps tables compact without requiring a separate process. In a high-throughput streaming context with continuous writes, it is a liability.

How we fixed it

We disabled inline compaction entirely and introduced a dedicated compaction window: 12 AM to 4 AM daily. During this window:

1. All DeltaStreamer instances for the affected tables are stopped
2. The Hudi Compactor utility runs as a standalone job
3. Once compaction completes, the streamers restart

We managed this workflow using an EMR Step Function to orchestrate the sequence — stop streamers, trigger compactor, wait for completion, restart streamers. The Step Function gave us visibility, retry logic, and alerting that a purely cron-based approach couldn’t provide.

The lesson: in production streaming systems, separate your write path from your maintenance path. Inline compaction is fine for development. For production high-throughput MOR tables, run compaction as a controlled, isolated operation with a clean window.

Mistake 4: Async Cleaning Failing Silently or Crashing the Driver

What we did

Hudi’s async cleaning runs in the background, removing old file versions that are no longer needed based on your retention configuration. We left it enabled with the default InProcessLockProvider — the assumption being that background cleaning would manage itself.

What actually happened

Two failure modes, both painful:

Silent failures: Async cleaning would silently fail to complete, leaving a growing backlog of old file versions on S3. We didn’t notice until storage costs started climbing and query performance degraded on tables with excessive file versions.

Driver crashes: When a large burst of CDC events had created a massive backlog of history in one go, the async cleaner would attempt to process the entire backlog in a single run — overwhelming the driver and causing it to crash mid-streaming. The InProcessLockProvider compounded this: it would hang during the lock acquisition phase, effectively freezing the streamer.

How we fixed it

We disabled async cleaning entirely and replaced it with a dedicated Cleaning utility, run as a controlled end-of-day job after the day’s streaming workload had completed. The cleaning job runs with its own allocated resources, processes the backlog methodically, and completes before the next streaming window begins.

This gave us three things we didn’t have before: predictable cleaning behavior, clear visibility into what was being cleaned, and no interference with the active write path.

The lesson: async background operations in a high-throughput streaming system are convenient until they aren’t. For operations that touch shared metadata — compaction, cleaning — isolated, scheduled, observable jobs are safer than background threads competing with active writes.

Mistake 5: Precombine Field Timestamp Format Mismatch Between Initial Load and CDC Stream

What we did

For several tables, we used a timestamp column as the precombine field — the field Hudi uses to determine which version of a record is the most recent when deduplicating. This is a common and sensible pattern.

The initial load for these tables came from files — bulk exports from the source systems. The ongoing delta came from Kafka CDC events.

What actually happened

Silent deduplication errors. Records from the initial file load were being incorrectly superseded by — or incorrectly superseding — records from the Kafka CDC stream, even when the CDC event was genuinely newer.

The root cause: the timestamp format in the initial load files was different from the timestamp format in the Kafka CDC events. The file export used one datetime format; the CDC pipeline used another. Hudi’s precombine comparison was effectively comparing apples and oranges — and getting unpredictable results.

Because MOR tables don’t surface these errors immediately (the corruption lives in the delta log files until compaction), this one took a while to diagnose. The symptom was stale data appearing after compaction runs — records that should have been updated were showing old values.

How we fixed it

We added a SQL transformation in the File Streamer configuration for the initial load, normalizing the timestamp format to match the format used by the Kafka CDC events before the data was written to Hudi.

The lesson: when you use a timestamp column as a precombine field, it must be consistent in format across every path that writes to that table. If you have multiple ingestion paths — initial load, CDC stream, backfill jobs — validate the timestamp format on every path before you go live. A format mismatch in a precombine field is one of the hardest bugs to diagnose because the symptom (stale data) appears long after the cause (the initial load) has completed.

What These Five Mistakes Have in Common

Looking back, each of these mistakes shares a root cause: we designed the happy path and underestimated the operational complexity of running multiple concurrent writers against a shared lakehouse at scale.

Hudi is a powerful framework. But it gives you a lot of rope. Inline compaction, async cleaning, default lock providers, and single-cluster deployments all work fine in isolation or at low scale. Under production load, with burst traffic patterns, multiple concurrent writers, and mixed ingestion sources, every one of these defaults becomes a risk.

The five fixes we implemented — transient cluster topology, OCC with StorageBasedLockProvider, isolated compaction windows, scheduled cleaning, and precombine format validation — are now part of our standard Hudi deployment checklist. If you’re building something similar, I’d suggest treating them as defaults, not as lessons you need to learn the hard way.

If you’ve hit similar issues — or different ones — I’d genuinely like to hear about them in the comments. Production war stories are how this community gets better.

Sinchan Banerjee is Director of Enterprise Data Architecture and the author of Scalable Data Architecture with Java (Packt, 2022). He writes about data engineering, AI infrastructure, and enterprise architecture on Medium and LinkedIn.