DAG: The Backbone of Every ML Pipeline

Airflow, Dagster, and Prefect all rely on the same idea. Here’s the explanation most tutorials skip.

You’re setting up your first real ML pipeline.

Data ingestion. Feature engineering. Model training. Evaluation. Deployment. You’ve got the steps figured out. You start wiring them together with scripts. Then cron jobs.

Then it breaks.

Task 3 depends on Task 2. Task 2 sometimes fails. Task 3 runs anyway — on stale data. Your model quietly trains on garbage. Nobody notices until the metrics look wrong two weeks later.

You start Googling. You look at Airflow. Dagster. Prefect. Metaflow.

Every single one uses the same word.

DAG — Directed Acyclic Graph.

And not one tutorial stops to explain what it actually means before drowning you in YAML configs and Python decorators.

Non-Member’s Link!

This article fixes that.

By the end, you’ll understand:

• What a DAG actually is — in plain English
• Why it exists and what problem it was built to solve
• How it shows up inside every ML pipeline you’ll ever build
• Which tools implement it and when to use each one

Start With the Word Itself

DAG stands for Directed Acyclic Graph.

Three words. Each one matters.

Graph: a collection of nodes connected by edges. That’s it. In a pipeline, nodes are your tasks. Edges are the dependencies between them.

Directed: each edge has a direction. An arrow, not just a line. It says “this task must finish before that task starts.” The flow goes one way.

Acyclic: no cycles. You can never follow the arrows and end up back where you started. There are no loops. A task cannot — directly or indirectly — depend on itself.

Put it together: a DAG is a map of tasks where every arrow points forward and nothing ever circles back.

Why “No Cycles” Is the Whole Point

Here’s the problem a DAG is designed to prevent.

Imagine Task A depends on Task B. Task B depends on Task C. Task C depends on Task A.

You have a loop. The pipeline can never start — each task is waiting for another to finish, and none of them ever do. This is a circular dependency, and it’s one of the most classic ways data pipelines break silently.

The “acyclic” constraint eliminates this entire class of problem by definition. If your graph has no cycles, it has a guaranteed execution order. There is always at least one task that has no dependencies — it runs first. Then the tasks that depend on it. Then the ones that depend on those.

The DAG makes the execution order provable, not guessed.

What a DAG Looks Like in an ML Pipeline

Here’s a concrete example. You’re building a model that predicts customer churn.

[Ingest Raw Data]
       ↓
[Validate Data Quality]
       ↓
[Feature Engineering]
    ↙        ↘
[Train Model]  [Generate Baseline Stats]
    ↓
[Evaluate Model]
       ↓
[Register Model to Registry]
       ↓
[Deploy to Staging]

Every box is a node. Every arrow is a directed edge. You can trace any path from top to bottom — and you can never loop back.

Notice the branch: Feature Engineering feeds into both Train Model and Generate Baseline Stats. Those two can run in parallel — they don’t depend on each other. This is one of the most powerful properties of a DAG. It naturally reveals which tasks can be parallelized and which must be sequential.

Without a DAG, you’d run everything sequentially by default, even when you don’t have to. With a DAG, the parallelism is explicit and automatic.

DAG vs Cyclic Pipeline

The Three Things a DAG Gives You

1. Dependency Management

You never have to manually figure out what runs first. The DAG’s structure encodes that information. If Task B needs Task A’s output, you declare that relationship once. The orchestrator handles the rest — every time, correctly.

2. Parallelism for Free

Any two tasks with no dependency relationship between them can run simultaneously. Your orchestrator sees this from the graph structure and runs them in parallel without you writing threading code or managing queues.

3. Failure Isolation

When one node fails, only the tasks downstream of that node are blocked. Everything else that doesn’t depend on the failed task continues running. You get partial success rather than total failure — and you can re-run just the failed node and its downstream tasks, not the entire pipeline.

DAGs in Practice: A Simple Dagster Example

Here’s what a real DAG looks like in code. This is Dagster, which thinks in terms of assets — the data each step produces — rather than just tasks.

from dagster import asset

@asset
def raw_data():
    # Load data from S3 or database
    return load_from_source()

@asset
def validated_data(raw_data):
    # raw_data is automatically a dependency
    assert raw_data is not None
    return run_validation(raw_data)

@asset
def features(validated_data):
    return engineer_features(validated_data)

@asset
def trained_model(features):
    return train_model(features)

@asset
def evaluation_report(trained_model, features):
    return evaluate(trained_model, features)

Notice what you didn’t write: no explicit dependency declarations, no scheduling logic, no retry configuration at the task level. You just defined what each asset needs as input, and Dagster infers the entire DAG from those function signatures.

The graph is implicit in the code. The orchestrator makes it visual and executable.

DAGs Aren’t Just for ML

Before we go further, it’s worth knowing how widely this pattern shows up — because once you see it, you’ll recognize it everywhere.

CI/CD Pipelines — in GitHub Actions, every job with a needs: declaration is a DAG edge. Tests run after build. Deploy runs after tests. The entire pipeline is a DAG.

dbt (data transformation) — every ref() in a dbt model creates a DAG edge. dbt computes the full dependency graph at compile time and runs models in the correct order.

Apache Spark — Spark’s execution plan is a DAG. When you chain transformations, Spark builds an internal DAG and optimizes it before running a single computation.

Kubernetes — init containers and pod dependencies form a DAG of what starts before what.

Git itself — a Git commit history is a DAG. Commits point to their parents. Branches diverge and merge. But you can never have a commit that is its own ancestor.

The pattern is everywhere because the underlying problem — “how do I run interdependent tasks in the right order without deadlocks” — is universal.

The Tools That Implement DAGs in 2026

Now the practical question: which tool do you actually use?

Here’s an honest breakdown as of 2026. Each tool has a different philosophy.

Apache Airflow

The industry standard. Battle-tested since 2014.

Airflow is the 500-pound gorilla of DAG orchestration. You write DAGs as Python files. The scheduler reads those files, builds the graph, and executes tasks according to the dependency structure you’ve defined.

from airflow.decorators import dag, task
from datetime import datetime

@dag(start_date=datetime(2026, 1, 1), schedule='@daily')
def ml_pipeline():
    @task
    def ingest():
        return load_data()
    @task
    def train(data):
        return train_model(data)
    @task
    def evaluate(model):
        return run_evaluation(model)
    data = ingest()
    model = train(data)
    evaluate(model)

ml_pipeline()

Good for: Teams that need maximum ecosystem breadth, enterprise integrations, and a huge community for support. Airflow has providers for practically everything — AWS, GCP, Snowflake, dbt, Kubernetes, Spark.

Watch out for: The scheduler parses DAG files every 30 seconds. If you write heavy imports or API calls at module level in your DAG file, you’ll tank your scheduler performance. Airflow 3.x (GA 2025, 3.2 in 2026) fixed many of the older architecture problems, but it’s still a system you need to manage.

GitHub stars: 38,000+

Dagster

Asset-first. Built for ML and data-quality-conscious teams.

Dagster flips the mental model. Instead of thinking about tasks, you think about assets — the datasets, tables, and ML models your pipeline produces. Dependencies are declared through function inputs.

Its biggest differentiator: Dagster can skip re-running a step if the upstream data hasn’t changed. If your raw data is the same as yesterday, it won’t re-run feature engineering. Airflow doesn’t have this natively.

Good for: ML pipelines, dbt integration, teams that care deeply about data lineage and observability. If you want to see not just “did this task succeed” but “what data did it produce and is that data fresh,” Dagster is designed for exactly that.

Watch out for: More initial setup than Prefect. Opinionated architecture that requires buying into its mental model. Community is growing fast (12,000 GitHub stars in early 2026) but the operational knowledge base is thinner than Airflow’s.

Prefect

Simplest developer experience. Python-first.

Prefect’s pitch: orchestration should feel like writing normal Python. Flows are just Python functions decorated with @flow. Tasks are decorated with @task. You don't think in DAGs directly — Prefect builds the dependency graph by tracking what data flows between your functions at runtime.

from prefect import flow, task

@task
def ingest_data():
    return load_from_source()

@task
def train_model(data):
    return run_training(data)

@flow
def ml_pipeline():
    data = ingest_data()
    model = train_model(data)
    return model

Good for: Teams moving fast, startups, ML engineers who want minimal ops overhead. Prefect Cloud’s hybrid model keeps orchestration managed (no scheduler to run yourself) while execution stays in your environment.

Watch out for: Enterprise lineage and governance capabilities are less mature than Dagster. If you need strict data asset tracking from day one, Dagster is the stronger choice.

When to Use Which

The Most Common DAG Mistakes

Running heavy imports in Airflow DAG files at module level The Airflow scheduler parses your DAG file every 30 seconds. Any code at the top level — imports, config reads, API calls — runs every parse cycle. With 300 DAG files, this compounds fast.

Creating tasks that are too granular A DAG with 200 tiny tasks is harder to debug than one with 20 well-scoped tasks. The overhead of tracking task state, logging, and retry logic adds up. Group related operations into meaningful units.

Not handling partial failures When a DAG fails mid-run, you want to re-run only the failed node and everything downstream — not the entire pipeline. Make sure your tasks are idempotent: running them twice produces the same result as running them once.

Ignoring task isolation Each task in a DAG should be independently executable. Avoid sharing state between tasks through global variables or in-memory objects. Pass data through return values, not side effects.

Why DAGs Matter More Now Than Ever

In 2026, ML pipelines are no longer “run this script on a schedule.”

They’re multi-step workflows involving data ingestion, validation, feature stores, model training, evaluation, A/B testing, deployment, monitoring, and retraining triggers. Each step has dependencies. Each step can fail. Each step produces data that the next step depends on.

Without a DAG, you’re managing all of that complexity in your head, in documentation, or in fragile cron job chains. With a DAG, the complexity is encoded in the graph. It’s visible. It’s executable. It’s retryable.

<cite index=”5–1">DAGs serve as the backbone of many data engineering workflows, providing clarity, structure, and reliability for processes that must run consistently over time.</cite>

That’s not marketing copy. It’s a description of what happens when you move from “scripts that run in order” to “a dependency graph that runs correctly.”

The Mental Model That Sticks

If you remember one thing from this entire article, make it this:

A DAG is not a tool. It’s a way of thinking about work that has dependencies.

Once you see your ML pipeline as a graph — nodes as tasks, edges as dependencies, arrows pointing forward, no loops — the tools become obvious. Dagster, Airflow, and Prefect are just different ways of expressing and executing that graph in Python.

The concept is what matters. The tool is just syntax.

You were already building DAGs when you were writing scripts that called other scripts in sequence. You just didn’t have the structure to make the dependencies explicit, visible, and recoverable when something broke.

Now you do.

If you want to go hands-on: Dagster has one of the best local development experiences for ML pipelines. Their quickstart at docs.dagster.io has you running a real asset graph in under 10 minutes. For Airflow, the Astronomer tutorials are the best entry point without managing your own infrastructure.