Two ways to get data in: batch ingestion and Change Data Capture

The choice depends on how you detect what changed, and how fresh the copy must be.

Two ways into Bronze. Batch loads sources on a schedule, CDC reads the write-ahead log

Pisey sells kitchenware in Phnom Penh. Her point-of-sale system records every sale, price change, and refund the instant it happens, into the database that runs the shop. She wants a dashboard, but the analysis cannot run on that database directly, because heavy queries would slow the system that is busy serving customers. The data has to move first, into a raw landing zone (Bronze).

Moving data from a source into the platform is called ingestion, and the hard part is not the first copy, it is that the source never holds still: every minute, the operational database of the shop getting more data further from what you loaded. Ingestion is really the job of keeping the copy in sync with a moving source, and there are two practical ways to do it.

First, clear up a common framing

You often hear that batch is for files and streaming is for databases. That is too neat, and it points people at the wrong tool.

If the source is a published file, like a public dataset released once a year, there is no choice to make: a file has no live change feed to subscribe to, so you load it on a schedule. That is batch.

The real decision appears only when the source is a live database. Both approaches work, and they differ not by the shape of the source but by how they detect change and how fresh they keep the copy of the data.

Batch: retrive from the source on a schedule

A batch job runs on a timer. It wakes, reads a defined set of rows, writes them to the destination, and sleeps until the next run.

The simple form is a full load, rereading everything each time. It is bulletproof but stops scaling once the table is large. The production workhorse is the incremental load, which asks only for what changed since last time using a meta condition: a predicate over a column the source updates on every write. With a timestamp it looks like WHERE updated_at > :last_run. The job records the newest value it saw and resumes from there next time.

A large share of analytics platforms run exactly this way, and for good reason. It is cheap, simple to operate, and easy to re-run after a failure. If the dashboard can be an hour or a day behind, incremental batch is usually the right answer, and anything heavier just adds cost. It pulls from live databases too, not only files. Pisey’s shop could be loaded this way every night.

So why is it not always enough? Asking on a schedule has three honest blind spots.

1. Undetectable deleted records . WHERE updated_at > :last_run can never return a row that was deleted, because the row is gone and left no timestamp behind. The copy keeps a record the source no longer has. The fix is to avoid hard deletes and mark rows with a flag like deleted_at, turning a delete into an update the batch can catch, which works only if you control the source.
2. Intermediate states are lost. If a row changes five times between runs, batch sees only the final state. Fine for a dashboard of current values, useless if you need how the value moved.
3. The marker can lie. Incremental batch is only as correct as updated_at. If application code skips it on some write path, those rows are lost silently and forever, and even a disciplined column can be outrun by a long transaction that commits after the high-water mark has advanced. Every run also queries the live tables, competing with the database's real work and needing the right indexes to stay cheap.

None of this makes batch loading bad. It makes batch the right tool when latency of minutes to hours is fine, the latest state is all you need, and the source can give a reliable marker.

CDC: read from the source’s own log

Change Data Capture reaches the same goal differently. Instead of asking the source what changed, it listens to the record the source already keeps of its own changes.

Every transactional database writes a durable, ordered log of every change before applying it to the tables, for crash recovery and replication. In PostgreSQL this is the write-ahead log (WAL); MySQL has the binlog. CDC reads that log and turns each change into a separate event.

Reading the log instead of querying the tables is what gives CDC its properties:

1. Every change is captured in commit order, including deletes, because a delete is logged like any other operation. Batch loading’s blind spots disappear.
2. Each change arrives as its own event, so the full history is available to any consumer that wants it, not just the latest state.
3. It is light on the source, since reading a sequential log is far cheaper than scanning tables while the database serves customers.
4. It is near real-time, with changes arriving in seconds rather than at the next scheduled run.

This is why Pisey’s live database fits CDC better than nightly batch. A refund reversing a sale, a product pulled from the catalogue, a price corrected twice in one hour: batch would miss or flatten each of these, and CDC captures every one as it happens.

CDC is not free. It needs access to the log, so the database must be configured for it with the right permissions, and some managed services restrict this. It needs streaming infrastructure to carry the events, which is more to run and more to break. The copy is eventually consistent, trailing the source by a small lag. And schema changes must be handled deliberately rather than ignored.

Which one a live source needs

The two techniques are working together

A real platform rarely uses one technique for everything. It matches the tool to each source, and often runs both. Next post, a project for these two loading techniques will be demostrate, as two pipelines into the same Bronze layer:

1. Batch for the Cambodia food price dataset. A published file of historical records, loaded in bulk. No live feed exists to stream, so batch is simply the answer.
2. CDC for an operational database like Pisey’s shop. Incremental batch could load it nightly, but we want every change, including deletes, within seconds, so CDC earns its extra complexity here.

The two can even combine on a single source: CDC for freshness, plus an occasional full reload to reconcile any drift. They are two settings on one dial, not opposing each others.

And every CDC pipeline starts with a batch step. A stream only carries what changes after it begins, so on day one it knows nothing about the rows already there. To close that gap it opens with a one-time snapshot: a full read of the current rows to set a baseline, with the log position recorded, and only then does it switch to streaming forward. Snapshot first, then stream.

Takeaways

1. The choice is not files versus databases. For a file, batch is the only option. For a live database, you choose by how you detect change and how fresh the copy must be.
2. Incremental batch asks on a schedule using a marker like updated_at. Simple and cheap, but it misses deletes, flattens intermediate states, and trusts a marker that can be wrong.
3. CDC reads the database’s own write-ahead log, capturing every change in order, including deletes, with low latency and little load, at the cost of more infrastructure and operational weight.
4. They coexist. Most platforms run both across sources, and a CDC pipeline itself opens with a batch snapshot before it streams.

What’s next

*Next → “Project: I loaded 86k rows of Cambodia food price data into a Lakehouse: here’s the actual process” Implemented a scalable batch data ingestion framework to load structured datasets into the data lakehouse.*

Prev → “Project: Build a Lakehouse storage layer with Iceberg, Parquet, and MinIO AIStor” For a step by step guide of building a data lakehouse storage.

Rathanak SENG . MSc in Data Science and Engineering

Contact: LinkedIn . Github

Email: withrathanak@gmail.com