How OLake Guarantees Exactly-Once Delivery to Apache Iceberg

There is a hidden gap in every data sync, the time between writing data to the destination and saving a checkpoint that records progress. If a process crashes in that gap, pipelines either duplicate rows on retry or silently lose them. And nothing ever throws an error to warn you.

OLake closes this gap for Apache Iceberg by committing data files and a progress marker in a single atomic operation, then comparing that marker to its checkpoint on every restart. If the checkpoint is OK, the sync goes on as it should. If Iceberg is ahead, the data is already there, and OLake skips reprocessing and just fixes the checkpoint without an external coordinator, transaction log, or locks.

This same recovery pattern is consistent across Full Refresh, incremental, and CDC syncs and across every supported driver, including Postgres, MySQL, MongoDB and SQL Server. Each mode has a deterministic identifier (based on chunk bounds, cursor values, or log positions) that can be used to identify exactly what is already committed, so a sync that crashes will always resume without mapping or missing records.

The result is that no matter what interrupts a sync, whether it’s a pod eviction, an OOM kill, or a dropped connection, the Iceberg table will always be correct. No duplicates, no gaps, no manual cleanup.

Read the full technical breakdown, with commit-sequence diagrams and real payload examples for each sync mode: 👉 How OLake Guarantees Exactly-Once Delivery to Apache Iceberg