Data Engineering 101 — Part 9 Introduction to Apache Airflow: The Orchestrator of Data Pipelines

Introduction

You have now learned how data moves (Kafka), how it gets processed (Spark), and how pipelines are structured.

But here’s a real problem every data team faces:

A data pipeline is rarely just one step. It’s a chain of steps — extract data, validate it, transform it, load it, send a report, notify the team. Each step depends on the previous one succeeding.

👉 Who makes sure all these steps run in the right order, at the right time, and alerts you when something breaks?

That’s Apache Airflow — the orchestrator that keeps your entire data pipeline running like clockwork.

What is Apache Airflow?

Apache Airflow is an open-source workflow orchestration platform that allows you to programmatically author, schedule, monitor, and manage data pipelines.

In simple terms: 👉 Airflow is like a smart alarm clock for your data pipelines — it knows exactly what needs to run, in what order, at what time, and what to do if something goes wrong.

Originally built by Airbnb in 2014 to manage their growing data pipelines, it was later open-sourced and donated to the Apache Software Foundation. Today it is the industry standard for pipeline orchestration.

What is Orchestration?

Before diving into Airflow, let’s understand orchestration.

Imagine you’re cooking a full meal:

• First you boil water
• Then you cook rice
• While rice cooks, you prepare the curry
• Once both are ready, you serve

Each step has a specific order. Some steps can run in parallel. If you forget one step, the whole meal fails.

👉 Orchestration is the process of managing these dependencies — ensuring each step runs at the right time, in the right order, and handling failures gracefully.

In data engineering, orchestration means managing the execution of your entire pipeline — from ingestion to transformation to loading to alerting.

What is a DAG?

The most important concept in Airflow is the DAG — Directed Acyclic Graph.

A DAG is a collection of tasks organized to show their relationships and dependencies — which task runs first, which runs next, and which can run in parallel.

• Directed — tasks flow in one direction (no going back)
• Acyclic — no circular dependencies (Task A cannot depend on Task B if Task B depends on Task A)
• Graph — a visual representation of all tasks and their connections

Real-World Example — Zomato’s Nightly Pipeline DAG:

Extract orders → Validate data → Transform data
                                      ↓
                          Load to warehouse → Send report → Notify team

Each arrow is a dependency. Airflow executes this automatically every night at midnight.

Core Components of Airflow

1. DAG The blueprint of your pipeline — defines all tasks and their dependencies. Written in Python.

2. Task A single unit of work within a DAG — run a Python script, execute a SQL query, trigger a Spark job, call an API.

3. Operator A template for a task. Airflow has many built-in operators:

• PythonOperator — runs a Python function
• BashOperator — runs a shell command
• SQLExecuteQueryOperator — runs a SQL query
• SparkSubmitOperator — triggers a Spark job
• EmailOperator — sends an email alert

4. Scheduler The brain of Airflow — monitors all DAGs and triggers tasks at the right time based on the schedule.

5. Executor Runs the actual tasks — locally, on a cluster, or in the cloud.

6. Web UI Airflow’s powerful visual interface — shows you the status of every DAG, every task, every run. You can monitor, debug, and manually trigger pipelines from here.

7. Metadata Database Stores the state of all DAG runs, task statuses, logs, and history.

Real-World Example — Zomato’s Nightly Pipeline

Here’s how Airflow orchestrates Zomato’s end-of-day pipeline:

Schedule: Every night at 12:00 AM

DAG Steps:

1. extract_orders → Pull all orders from OLTP database
2. validate_data → Check for nulls, duplicates, schema errors
3. transform_data → Calculate revenue, aggregate by city and restaurant
4. load_to_warehouse → Push clean data to BigQuery
5. update_dashboard → Refresh Looker dashboard
6. send_report → Email summary report to business team
7. notify_success → Send Slack alert — "Pipeline completed successfully"

👉 If step 2 fails, steps 3–7 are automatically skipped and an alert is sent. The team fixes the issue and reruns from step 2 — not from the beginning.

Why Airflow Over Cron Jobs?

Many teams start with simple cron jobs to schedule pipelines. Airflow replaces cron jobs with something far more powerful:

Airflow in the Modern Data Stack

Airflow works as the central coordinator for all the tools you have learned:

ToolHow Airflow uses itApache KafkaAirflow monitors Kafka lag and triggers alertsApache SparkAirflow schedules and triggers Spark jobsdbtAirflow runs dbt transformations on scheduleSnowflake / BigQueryAirflow loads data and runs SQL queriesSlack / EmailAirflow sends success or failure notifications

Quick Summary

• Apache Airflow is a workflow orchestration platform for scheduling and managing data pipelines
• Core concept: DAG — a directed acyclic graph of tasks with dependencies
• Key components: DAG, Task, Operator, Scheduler, Executor, Web UI
• Airflow replaces simple cron jobs with visual monitoring, retries, alerting, and dependency management
• Originally built by Airbnb — now the industry standard for pipeline orchestration
• Works seamlessly with Kafka, Spark, dbt, Snowflake, and all modern data tools

💬 Final Note

If you’re starting your Data Engineering journey, stay tuned for more simple and practical explanations. You can follow me on Medium for upcoming articles in this series.

⬇️ Here’s the full series roadmap:

📚 Data Engineering 101 — Series Guide

✅ Part 1 → [What is Data Engineering?](click here)

✅ Part 2 → [OLTP vs OLAP: Understanding How Data Systems Work](click here)

✅ Part 3 → [ETL vs ELT: How Data Moves Between Systems](click here)

✅ Part 4 → [Data Warehouse vs Data Lake vs Data Lakehouse](click here)

✅ Part 5 → What is a Data Pipeline? (click here)

✅ Part 6 → [Batch Processing vs Stream Processing](click here)

✅ Part 7 → Introduction to Apache Kafka (click here)

✅ Part 8 → Introduction to Apache Spark (click here)

Part 9 → Introduction to Apache Airflow (you are here)

Part 10 → Normalization vs Denormalization (coming soon)

Part 11 → Star Schema vs Snowflake Schema (coming soon)

Part 12 → SCD Types 1, 2 and 3 (coming soon)

Part 13 → Incremental Load vs Full Load (coming soon)

Part 14 → Change Data Capture (CDC) (coming soon)

Part 15 → Partitioning & Indexing in Data Engineering (coming soon)

Part 16 → Data Quality & Validation (coming soon)

Part 17 → Data Governance & Data Catalog (coming soon)

Part 18 → APIs & Webhooks in Data Engineering (coming soon)

Part 19 → Data Observability & Monitoring (coming soon)

Part 20 → What is dbt? Transforming Data the Modern Way (coming soon)

Part 21 → Cloud Platforms — AWS vs GCP vs Azure (coming soon)