What Modern Data Ingestion Looks Like Today

How raw operational data becomes analytics ready

Read free for non members

Data ingestion is one of those invisible layers that powers almost everything we do with data from dashboards to deeper analytical work. I’ve been digging into how it actually works under the hood, and one thing is clear.

The way we ingest data has changed a lot over time

Traditionally, most data pipelines followed an ETL approach. Data was extracted from source systems, transformed in a separate processing layer, and only then loaded into a data warehouse. This design made sense at the time it was shaped by limited compute resources and expensive storage. Pre-processing data upfront helped reduce storage costs and improved query performance once the data landed in the warehouse.

Today, the picture is very different. With cloud data warehouses like Snowflake or BigQuery, the model has largely shifted toward ELT. Data is extracted and loaded immediately into a raw or staging layer, and transformations happen afterward directly inside the warehouse, using distributed compute.

This shift comes with a few clear advantages :

• Data fidelity : Raw data is preserved, which makes transformations reproducible and reversible.
• Agility : You can adjust business logic or fix historical assumptions without rebuilding pipelines from scratch.
• Scalability : Modern MPP (massively parallel processing) engines handle transformations at scale, often faster than external processing layers.

So the question becomes : how does modern ingestion actually work in practice ? That’s what I’ve been exploring in this breakdown of the ingestion layer.

Extract : The Entry Point of the Pipeline

Extraction is the first layer of modern ingestion. Before transforming or modeling anything, we first need a reliable way to collect information from the systems that generate it.

And that information can come from almost anywhere :

• Databases such as PostgreSQL or MySQL powering transactional applications
• Files like CSV exports, logs, or JSON dumps generated by internal systems
• IoT devices continuously emitting sensor updates in real time
• SaaS platforms exposing APIs such as Salesforce, Shopify, or Google Analytics

The extraction method itself depends heavily on the source system and on how fresh the data needs to be.

Most ingestion patterns fall into three categories :

• Request based extraction The platform periodically queries an API or database to retrieve new records. This approach is relatively simple and gives good control over scheduling, but frequent queries can put pressure on the source system.
• Event based extraction Instead of being queried, the source actively pushes events through webhooks or streaming systems. This reduces latency significantly and allows near real time ingestion.
• Log based extraction Rather than querying tables directly, the ingestion layer reads database transaction logs such as PostgreSQL WAL files or MySQL Binlogs. This makes it possible to capture inserts, updates, and deletes sequentially while minimizing impact on production systems.

This last approach is where Change Data Capture (CDC) becomes especially interesting. CDC allows systems to replicate changes almost exactly as they happen, preserving both transaction order and historical consistency. In modern architectures, it has become one of the most powerful ways to ingest operational data at scale.

Load : The Raw Data Zone

Before going further, I need to introduce the Medallion architecture, a data design approach popularized by Databricks. The idea is simple : data quality and structure progressively improve as information moves through different processing layers.

The name comes from precious metals, Bronze, Silver, and Gold, representing increasing levels of refinement and business value. You can think of it as raw material being progressively cleaned, standardized, and transformed into something usable for analytics.

Once extracted, data first lands in the Bronze layer, the raw and unprocessed zone. This is the ingestion layer in its purest form. Data is stored almost exactly as it arrived from the source system.

At this stage, the goal is not to clean or model anything yet. The priority is preserving fidelity. Keeping the original records intact makes it possible to replay transformations, debug issues later, or recover historical context when business logic changes.

In practice, this Bronze layer often becomes the foundation of the modern data lake or warehouse.

Transform : Shaping Value

Now comes the part where raw information starts becoming genuinely useful.

This transformation process mainly happens inside the Silver and Gold layers of the Medallion architecture.

• Silver Layer (Cleaned and Enriched) This is where raw records start getting standardized and validated. Duplicate entries are removed, data types normalized, missing values handled, and datasets enriched through joins or reference tables. The goal here is to create reliable and consistent datasets that are still fairly granular and flexible for analysis.
• Gold Layer (Curated Business Data) The Gold layer focuses on business meaning rather than raw operational detail. This is where metrics like monthly revenue, customer retention, or product performance are modeled into curated datasets optimized for dashboards, BI tools, and reporting. In many architectures, this layer relies heavily on dimensional modeling and star schemas.

What I like about this layered approach is that it separates concerns clearly. Raw ingestion remains untouched, while transformations progressively add structure, quality, and business context on top of it.

Beyond Ingestion : Managing Integrity and Reliability

Getting data into a warehouse is only half the job. Modern ELT pipelines also need to stay reliable, reproducible, and secure over time. Once ingestion scales across dozens of systems and teams, the real challenge becomes maintaining trust in the data itself.

Data Contracts

One of the biggest challenges in ingestion is schema evolution. A renamed column, a missing field, or a type change can silently break downstream transformations and dashboards.

This is where data contracts become important. They define expectations between source systems and data consumers : expected schemas, freshness, formats, or quality rules. Instead of discovering issues hours later in a dashboard, pipelines can fail early and surface problems closer to the source.

Orchestration

Modern ingestion pipelines rarely run as isolated jobs anymore. Dependencies between extraction, loading, validation, and transformation steps need to be coordinated carefully.

Orchestration tools like Airflow, Dagster, or Mage help schedule pipelines, manage retries, monitor failures, and keep workflows reproducible. As pipelines grow more complex, orchestration becomes less about scheduling and more about maintaining operational visibility across the entire platform.

Security and Governance

Because raw ingestion often centralizes large amounts of operational data, governance becomes critical. Access control, lineage, encryption, and auditing all play a role in keeping platforms secure and compliant.

The Bronze layer is especially sensitive since it may contain unfiltered or personally identifiable information directly from source systems. Preserving raw fidelity is powerful, but it also means organizations need clear governance rules around who can access what and how long data should be retained.

Conclusion

Modern data ingestion is no longer just about moving data from one system to another. With ELT architectures, cloud warehouses, and layered modeling approaches like Medallion, ingestion has become the foundation of scalable and flexible analytics platforms.

What I find especially interesting is how much this changes the mindset around data work. Instead of forcing structure too early, modern pipelines prioritize preserving raw information first, then progressively refining it as business needs evolve.

For anyone working in analytics and looking to move closer to analytics engineering, understanding ingestion patterns, orchestration, and transformation layers is an excellent place to start. A lot of modern data systems begin here.

Sources

https://aws.amazon.com/compare/the-difference-between-etl-and-elt/

https://aws.amazon.com/compare/the-difference-between-etl-and-elt/

https://aws.amazon.com/compare/the-difference-between-etl-and-elt/

[embed]Streamline data processes with the data engineering lifecycle Understanding the data engineering lifecycle helps every role on the team perform better and get the results they're…www.fivetran.com

[embed]What is Data Ingestion? Data ingestion is thwww.databricks.com