How to Add Data Quality Checks to Your Data Pipelines

A data pipeline can run successfully and still produce bad data.

That is one of the most frustrating parts of working with data systems. The job finishes, the dashboard refreshes, the report goes out, and everything looks fine on the surface. But somewhere inside the pipeline, a column changed format, duplicate records slipped in, a join dropped rows, or a critical field started coming through as null.

By the time someone notices, the issue has already moved downstream. Analysts are questioning the numbers. Product teams are making decisions from unreliable metrics. Engineers are digging through logs trying to understand where the problem started.

This is why data quality checks should not be treated as an extra step at the end of the pipeline. They need to be part of the pipeline itself.

Data quality checks help you catch problems before they reach the people and systems that depend on the data. They give you a way to define what “good data” should look like, test incoming and transformed data against those expectations, and decide what should happen when something fails.

In this article, we will look at how to add practical data quality checks to your pipelines using tools such as Great Expectations and dbt-expectations. We will also look at where these checks should run, how to connect them to workflow tools like Airflow and dbt, and how to monitor failures in a way that is useful rather than noisy.

Why Data Quality Checks Matter

Most data problems do not start as dramatic failures. They usually begin quietly. A source system changes a field name. A customer record is duplicated. A required column starts accepting empty values. A transformation still runs, but the logic no longer matches the shape of the data. Nothing breaks immediately, but the output is no longer trustworthy.

That is the real danger. Bad data often looks normal until someone uses it.

For example, a sales dashboard may still load even if duplicate transactions were introduced upstream. A churn report may still generate even if customer IDs are missing. A machine learning feature table may still be created even if some values are outside the expected range.

The pipeline worked technically, but it failed logically.

Data quality checks help close that gap. Instead of only asking, “Did the pipeline run?” you also ask, “Did the data meet the rules we expected?”

Some of the most common checks include:

• Confirming that important columns are not null
• Checking that IDs are unique
• Making sure values fall within an expected range
• Verifying that dates are valid and not in the future
• Detecting duplicate records
• Checking that joins do not unexpectedly lose rows
• Confirming that accepted values match known categories
• Testing relationships between tables

These checks may look simple, but they can prevent serious issues. A missing customer ID, a duplicated order, or a broken relationship between tables can affect dashboards, financial reporting, product analytics, and downstream machine learning workflows.

The goal is not to make a pipeline perfect. The goal is to make data problems visible early enough that they can be fixed before they cause bigger damage.

Where to Place Data Quality Checks in a Pipeline

A common mistake is to treat data quality as something that happens only after the pipeline has finished. That is too late.

By that point, bad data may already be sitting in a warehouse table, powering a dashboard, feeding a model, or being used by another team. Good data quality design means placing checks at different points in the pipeline, depending on the type of risk you want to catch.

In most pipelines, there are three useful places to add checks: during ingestion, after transformation, and before consumption.

1. Check Data During Ingestion

The first place to validate data is when it enters your system.

At this stage, you are mostly checking whether the raw data is usable. You are not trying to prove that every business rule is correct yet. You are simply asking whether the incoming data has the basic structure and completeness needed for the rest of the pipeline to work.

For example, you may want to check that:

• Required columns are present
• File formats are correct
• Important fields are not empty
• Dates can be parsed correctly
• Numeric fields contain valid numbers
• Incoming records are not completely duplicated

These checks are useful because they catch source-level problems early.

If an API suddenly stops sending a required field, or a CSV file arrives with a missing column, the pipeline should not quietly continue as if everything is fine. It should raise a warning or fail before the bad data moves further downstream.

Ingestion checks are especially important when data comes from external systems that you do not fully control. Source systems can change. Vendors can update exports. Product teams can modify event tracking. Without checks at this stage, those changes may only be discovered after they have already affected reporting.

2. Check Data After Transformation

The second important point is after transformation.

This is where raw data has been cleaned, joined, filtered, aggregated, or reshaped into models that other teams can use. It is also where many hidden errors can be introduced.

A transformation may run without any technical errors, but still produce the wrong result. A join may accidentally remove rows. A filter may exclude more data than expected. A calculation may produce negative values where they should never exist. A model may create duplicate primary keys.

This is why transformation checks should focus on business logic and model correctness.

For example, you may want to check that:

• Primary keys are unique
• Required columns are not null
• Accepted values match known categories
• Order amounts are not negative
• Customer records connect correctly to orders
• Aggregated totals are within a reasonable range
• Row counts do not drop unexpectedly after joins

This is where tools like dbt tests and dbt-expectations are especially useful. Because dbt already owns the transformation layer, it makes sense to keep many of your model-level checks close to the models themselves.

Instead of writing separate validation scripts that live far away from your transformation logic, you can define tests in YAML next to the model. That makes the rules easier to find, review, and maintain.

3. Check Data Before It Is Used

The final place to add checks is before the data is consumed.

This usually means before data is published to production tables, dashboards, reports, reverse ETL tools, or machine learning workflows. At this point, the question is no longer just, “Is this data technically valid?” The question becomes, “Is this data safe to use?”

For high-impact datasets, this stage matters a lot.

For example, a finance dashboard, executive KPI report, customer billing table, or machine learning feature store should not receive unvalidated data. A small error in these areas can create confusion, wrong decisions, or real business impact.

This is where a Write-Audit-Publish pattern can be useful.

First, write the data to a staging or temporary location. Then, audit it using quality checks. Only publish it to the final production table if the checks pass.

This pattern prevents bad data from automatically replacing good production data. It gives the pipeline a safety gate.

Not every dataset needs this level of strictness. A low-risk exploratory table may only need warnings. But for trusted reporting layers and critical business tables, pre-publish checks are one of the best ways to protect downstream users.

Deciding What Should Warn and What Should Fail

Not every data issue should stop the pipeline. This is an important point. If every minor anomaly causes a full pipeline failure, teams quickly start ignoring alerts or disabling checks altogether. Good data quality design is not just about adding more rules. It is about choosing the right response for each rule.

Some checks should fail the pipeline immediately.

For example:

• A primary key is missing
• Required IDs are null
• A production table receives zero rows
• A critical source file is missing
• Data types have changed unexpectedly
• Billing or financial values are clearly invalid

These issues can break downstream systems or lead to incorrect decisions. In those cases, failing fast is better than allowing bad data to spread.

Other checks should only warn.

For example:

• A value is slightly outside the normal range
• A non-critical column has more nulls than usual
• A category appears that has not been seen before
• A row count is lower than expected, but still within a tolerable range

These issues may still need attention, but they do not always justify blocking the entire pipeline.

A useful way to think about this is:

If the data would cause serious damage when used, fail the pipeline. If the data is suspicious but still usable, warn the team. If the data is dirty but can safely be excluded, drop or quarantine the bad records.

This gives you a more realistic data quality system. Instead of treating every issue the same way, you can match the response to the level of risk.

Choosing the Right Tool for Data Quality Checks

Once you know where checks should run in your pipeline, the next question is which tool should handle them.

There is no single correct answer. The right choice depends on where the data is being validated, how your pipeline is built, and which team owns that part of the workflow.

Two common tools for this are Great Expectations and dbt-expectations.

They solve a similar problem, but they fit into different parts of the data stack.

Great Expectations

Great Expectations is useful when you want a dedicated framework for validating data across different stages of a pipeline.

It lets you define expectations, run them against datasets, and generate validation results that show which checks passed or failed. You can use it with data from databases, warehouses, files, Pandas DataFrames, Spark DataFrames, and other sources.

That makes it a good fit when data quality checks need to happen outside a single transformation tool.

For example, you can use Great Expectations to validate:

• Raw files after ingestion
• Tables before they are loaded into a warehouse
• Data pulled from an external database
• DataFrames inside a Python or Spark job
• Staging tables before publishing them to production
• Critical datasets inside an Airflow DAG

The main idea in Great Expectations is simple: you create rules that describe what valid data should look like.

For example:

• This column should not be null
• This column should contain unique values
• These values should be between a minimum and maximum range
• This table should have at least a certain number of rows
• This column should only contain values from an accepted list

These rules are grouped into something called an Expectation Suite. A suite usually belongs to a specific dataset or table. For example, you may have one suite for a customers table, another for orders, and another for transactions.

Great Expectations is especially useful when you want your checks to be more visible to the wider team. Its validation results and Data Docs can make failures easier to review because they show what passed, what failed, and what values were observed during the check.

This matters because data quality is not only an engineering concern. Analysts, analytics engineers, data scientists, and business stakeholders may also need to understand why a dataset is trusted or why it failed validation.

dbt and dbt-expectations

dbt is a better fit when the checks are closely tied to transformation logic.

If your team already uses dbt to build models in the warehouse, then many data quality checks should live beside those models. This keeps the validation rules close to the SQL that creates the data.

For example, if you have a dbt model called fct_orders, you can define tests that check whether:

• order_id is unique
• order_id is not null
• customer_id exists in the customers model
• order_status only contains accepted values
• order_total is greater than or equal to zero

Basic dbt tests already cover many common cases such as uniqueness, non-null values, relationships, and accepted values. dbt-expectations extends this by adding a wider set of expectation-style tests inspired by Great Expectations.

This is useful when you want more expressive checks without writing custom SQL every time.

For example, you may want to check that:

• A numeric column falls within a range
• A date column is not in the future
• A row count falls within an expected range
• A column matches a pattern
• A value in one column makes sense based on another column

The advantage of dbt-based checks is that they become part of the transformation workflow. When you run dbt test, your validation rules run alongside your models. This makes them easy to include in development, CI/CD, and production deployment workflows.

For analytics engineering teams, this is often the most natural place to start.

Great Expectations vs dbt-expectations

The difference between the two tools is not really about which one is better.

It is about where each one fits.

Great Expectations works well when data quality checks need to sit across the broader pipeline. It is useful for ingestion checks, Python-based pipelines, Airflow workflows, Spark jobs, and validation outside the transformation layer.

dbt-expectations works well when the checks belong inside the warehouse transformation layer. It is useful when your models are already built in dbt and you want validation to live close to the SQL logic.

A simple way to decide is this:

1. Use Great Expectations when you need pipeline-level validation.
2. Use dbt tests or dbt-expectations when you need model-level validation. Use both when you want coverage across ingestion, transformation, and publishing.

For example:

During ingestion, Great Expectations checks whether the raw file or source table has the expected structure.

During transformation, dbt tests and dbt-expectations check whether the cleaned models are valid.

Before publishing, Great Expectations or a dbt test step checks whether the final table is safe to expose to dashboards and business users.

This layered approach is usually better than relying on one big validation step at the end. Each tool catches problems where they are easiest to understand and fix.

Start With Simple Checks First

It is tempting to begin with complex validation rules.

That is usually not the best starting point.

Most teams get more value by starting with simple checks on important datasets. A small number of well-placed checks can catch many common issues.

Start with your most trusted tables and ask:

• Which columns must never be null?
• Which IDs must be unique?
• Which relationships must always hold?
• Which values should only come from a known list?
• Which numbers should never be negative?
• Which tables should never be empty after a successful run?

These checks are easy to understand, easy to maintain, and easy to explain when they fail.

Once the basic checks are stable, you can add more advanced rules. These may include distribution checks, freshness checks, anomaly detection, schema drift detection, and cross-table reconciliation.

The point is to build trust gradually.

A data quality system does not need to be complicated on day one. It needs to be useful. Start with the checks that protect the most important data, then expand as your pipeline and team mature.

Setting Up Great Expectations

Great Expectations is a good place to start when you want data quality checks that can run across different parts of a pipeline.

It is especially useful when your pipeline is Python-based, orchestrated with Airflow, or working with data before it reaches the final warehouse models. You can use it to validate raw files, database tables, warehouse tables, Pandas DataFrames, or Spark DataFrames.

The setup does not need to be complicated. At a high level, you need to:

1. Install Great Expectations
2. Create a Data Context
3. Connect to your data
4. Create an Expectation Suite
5. Add expectations
6. Run validations
7. Review the results

Let’s walk through that flow.

1. Install Great Expectations

Start by installing Great Expectations in your project environment.

pip install great_expectations

In a real project, it is better to do this inside a virtual environment instead of installing it globally. That keeps your project dependencies isolated and avoids version conflicts with other Python projects.

For example:

python -m venv .venv
source .venv/bin/activate
pip install great_expectations

On Windows, the activation command is different:

.venv\Scripts\activate

After installation, you can import Great Expectations in a Python script or notebook:

import great_expectations as gx

2. Create a Data Context

The Data Context is the control center of a Great Expectations project.

It keeps track of your data sources, expectation suites, validation results, checkpoints, and documentation settings. In simple terms, it is where Great Expectations stores the configuration it needs to run quality checks consistently.

You can create a file-based context like this:

import great_expectations as gx
context = gx.get_context(mode="file")

A file-based context is useful because it creates project files that can be version-controlled. That means your data quality rules can live alongside your pipeline code, where other engineers can review and update them.

This is important. Data quality checks should not live only in one person’s notebook. If they protect production data, they should be part of the project.

3. Connect Great Expectations to Your Data

Once the context is ready, the next step is connecting Great Expectations to the data you want to validate.

The exact setup depends on where your data lives. You may be validating:

• A local CSV or Parquet file
• A Pandas DataFrame
• A Spark DataFrame
• A PostgreSQL table
• A Snowflake table
• A BigQuery table
• A staging table created by your pipeline

During development, it is often easiest to start with a small file or DataFrame. This lets you test your expectations quickly before connecting to production systems.

For example, you might load a simple customer dataset into a Pandas DataFrame and validate it locally before moving the same style of checks into an Airflow DAG or warehouse workflow.

The point is not to validate everything at once. The point is to prove that the rules make sense on a small dataset first.

4. Create an Expectation Suite

An Expectation Suite is a collection of rules for a specific dataset.

For example, if you have a customers table, you may create a suite called customers_suite. If you have an orders table, you may create another suite called orders_suite.

This keeps the checks organized. Each dataset gets its own set of expectations based on what that dataset should look like.

A simple suite can be created like this:

suite = gx.ExpectationSuite(name="customers_suite")
context.suites.add(suite)

After creating the suite, you can start adding expectations to it.

5. Add Expectations

Expectations are the actual data quality rules.

For a customers table, you might expect that:

• customer_id should exist
• customer_id should never be null
• customer_id should be unique
• email should not be empty
• created_at should contain valid dates
• lifetime_spend should not be negative

In Great Expectations, those rules can be expressed as expectations.

For example:

expectation = gx.expectations.ExpectColumnValuesToNotBeNull(
    column="customer_id"
)
suite.add_expectation(expectation)

You can also check whether numeric values fall inside an expected range:

expectation = gx.expectations.ExpectColumnValuesToBeBetween(
    column="lifetime_spend",
    min_value=0,
    max_value=100000
)

suite.add_expectation(expectation)

These checks are simple, but they protect important assumptions.

If customer_id becomes null, something is wrong. If lifetime_spend is negative, something is wrong. If a column disappears, something is wrong.

Instead of letting those issues move silently through the pipeline, Great Expectations makes them visible.

6. Validate a Batch of Data

After defining expectations, you run them against a batch of data.

A batch is simply the specific slice of data being validated. It could be today’s file, the latest table partition, a DataFrame in memory, or a staging table created by the pipeline.

When the validation runs, Great Expectations compares the actual data against the rules in the Expectation Suite.

The result tells you:

• Which expectations passed
• Which expectations failed
• What values were observed
• How many checks ran
• Whether the overall validation succeeded

This is where data quality becomes operational. You are no longer just hoping that the data is correct. You are checking it directly.

7. Review the Validation Results

Validation results are only useful if people can understand them.

Great Expectations helps here by producing structured validation output and Data Docs. Data Docs give you a readable view of your checks and results, including clear pass and fail indicators.

This makes debugging easier.

Instead of saying, “The pipeline failed,” you can say:

“The customers validation failed because customer_id had null values in today’s batch.”

That is much more actionable.

A good validation result should help the team answer three questions quickly:

1. What failed?
2. Why did it fail?
3. What data caused the failure?

If your quality checks do not help answer those questions, they will become noise. The goal is not just to fail a pipeline. The goal is to make the reason for failure clear enough that someone can fix it.

A Practical Example

Imagine you have a customers table that is used by dashboards and customer reporting.

At minimum, you may want to check that:

• customer_id exists
• customer_id is not null
• customer_id is unique
• email is not null
• created_at is not null
• lifetime_spend is between 0 and 100,000

These checks cover basic structure, identity, completeness, and business logic.

They will not catch every possible problem, but they catch the kinds of issues that can quickly damage trust in the dataset.

That is a good starting point.

Once these basic checks are stable, you can extend them with more advanced rules, such as checking row count changes, detecting unusual spending patterns, validating accepted country codes, or comparing totals against another source system.

Start simple. Make the checks reliable. Then expand.

Setting Up dbt-expectations

Great Expectations is useful across different parts of a pipeline, but if your transformations already live in dbt, then dbt is usually the best place to start adding model-level checks.

That is where dbt-expectations fits in.

dbt already has built-in tests for common rules such as not_null, unique, relationships, and accepted_values. These are enough for many basic checks. But as your models become more important, you may need more expressive tests without writing custom SQL every time.

dbt-expectations gives you that extra layer.

It brings Great Expectations-style tests into dbt, so you can define more detailed validation rules directly inside your model YAML files.

1. Add dbt-expectations to Your Project

Inside your dbt project, open or create a file called packages.yml. For the maintained package, you can add:

packages:
  - package: metaplane/dbt_expectations
    version: [">=0.10.0", "<0.11.0"]

Then install the package:

dbt deps

This downloads the package and makes the tests available inside your dbt project. After that, you can use dbt-expectations tests inside your model YAML files in the same way you use standard dbt tests.

2. Add Basic Tests First

Before adding advanced expectations, start with the simple checks that protect the structure of your model.

For example, imagine you have a model called fct_orders.

A basic YAML file may look like this:

version: 2
models:
  - name: fct_orders
    description: "Fact table containing one row per order."
    columns:
      - name: order_id
        tests:
          - not_null
          - unique
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customers')
              field: customer_id
      - name: order_status
        tests:
          - accepted_values:
              values: ['pending', 'paid', 'cancelled', 'refunded']

These checks are simple, but they catch some of the most damaging problems.

If order_id is null, the model cannot reliably identify an order. If order_id is duplicated, the table no longer has one row per order. If customer_id does not match the customer dimension, the relationship between orders and customers is broken. If order_status contains unexpected values, downstream reports may misclassify orders.

This is why basic dbt tests should not be skipped. They form the first layer of trust.

3. Add dbt-expectations Tests

Once the basic dbt tests are in place, you can add dbt-expectations for more specific rules.

For example, you may want to check that order_total is never negative:

version: 2
models:
  - name: fct_orders
    columns:
      - name: order_total
        tests:
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              row_condition: "order_total is not null"

This check is more expressive than a basic not_null test. It captures a business rule: an order total should not be below zero.

You can also check whether a date column is reasonable:

version: 2
models:
  - name: fct_orders
    columns:
      - name: order_date
        tests:
          - not_null
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: "'2020-01-01'"
              max_value: "current_date"

This kind of rule helps catch strange dates, bad source data, or transformation logic errors.

For example, if a record suddenly appears with an order date from the year 2099, the pipeline may still run, but the data is clearly suspicious. A check like this makes the issue visible.

4. Test Row Counts and Completeness

Column-level checks are useful, but they are not enough.

Sometimes the issue is not inside a single column. Sometimes the issue is the size of the dataset itself.

For example, if your fct_orders model usually receives thousands of records per day and suddenly produces zero rows, that should raise an alert.

You can add a row count expectation like this:

version: 2
models:
  - name: fct_orders
    tests:
      - dbt_expectations.expect_table_row_count_to_be_between:
          min_value: 1

This check protects against empty production tables.

For a development environment, a minimum of one row may be enough. In production, you may want a stronger rule based on expected volume.

For example, if a table should always have at least 10,000 records after a daily run, then your production check should reflect that.

5. Use Severity Levels Carefully

Not every failed check should stop the pipeline.

dbt allows you to configure test severity. This is useful because some checks should fail the job, while others should only warn the team.

For example:

version: 2

models:
  - name: fct_orders
    columns:
      - name: discount_amount
        tests:
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              max_value: 1000
              config:
                severity: warn

A warning means the issue is visible, but it does not necessarily block the pipeline.

This is useful for checks that indicate something unusual but not immediately dangerous. Maybe a discount amount is higher than normal, but the data can still be reviewed later.

For critical checks, use severity: error.

version: 2

models:
  - name: fct_orders
    columns:
      - name: order_id
        tests:
          - not_null:
              config:
                severity: error
          - unique:
              config:
                severity: error

A missing or duplicated order_id should usually fail the pipeline because the table cannot be trusted without a reliable primary key.

The key is to avoid treating every rule the same way.

Critical checks should block bad data. Non-critical checks should create visibility. Noisy checks should be improved or removed.

6. Run the Tests

Once your tests are defined, run:

dbt test

This runs all tests in the project.

You can also run tests for a specific model:

dbt test --select fct_orders

Or run tests as part of a build:

dbt build --select fct_orders

Using dbt build is helpful because it runs models, tests, snapshots, and seeds in dependency order. This makes it useful in CI/CD and production workflows.

7. Keep Tests Close to the Model

One of the best things about dbt testing is that the tests live close to the model definition.

This makes the project easier to understand.

If someone opens the YAML file for fct_orders, they can see not only what the model is, but also what assumptions the model is expected to satisfy.

That is valuable documentation.

A good dbt model should answer more than, “What SQL created this table?”

It should also answer:

• What does one row represent?
• Which columns are required?
• Which columns must be unique?
• Which relationships must hold?
• Which values are acceptable?
• Which business rules should never be broken?

When tests are written this way, they become part of the model contract. They help engineers and analysts understand what the data promises to downstream users.

A Practical dbt Testing Flow

A good dbt quality workflow can be simple:

First, add built-in dbt tests for primary keys, required fields, relationships, and accepted values.

Second, add dbt-expectations tests for ranges, row counts, date checks, pattern checks, and more expressive business rules.

Third, configure severity so critical checks fail and lower-risk checks warn.

Fourth, run tests locally during development before opening a pull request.

Fifth, run tests again in CI/CD before changes are merged or deployed.

Sixth, run tests in production after scheduled transformations.

This gives the team multiple chances to catch bad data before it reaches reports, dashboards, or business users.

The goal is not to add hundreds of tests everywhere. The goal is to protect the models that people depend on most.

Adding Data Quality Checks to Airflow Pipelines

Defining data quality checks is only useful if those checks actually run when the pipeline runs.

This is where orchestration matters.

In many data teams, Airflow is used to schedule and manage pipeline tasks. It controls when data is extracted, transformed, loaded, validated, and published. Because Airflow already manages the pipeline flow, it is a natural place to add data quality checks.

Instead of running validations manually, you can make them part of the DAG.

That means every time the pipeline runs, the data is checked automatically.

Why Add Checks Inside Airflow?

Airflow gives you control over when validation should happen.

For example, you can run checks:

• After extracting data from a source
• Before loading data into the warehouse
• After creating a staging table
• Before replacing a production table
• Before triggering downstream dashboards or reports

This helps you prevent bad data from moving forward.

A pipeline should not simply extract, transform, and load data. It should also pause and ask whether the data is safe enough to continue.

For example, imagine a daily orders pipeline.

The DAG may look like this:

1. Extract orders from the source system
2. Load orders into a staging table
3. Run data quality checks on the staging table
4. Transform the data into analytics models
5. Publish the final table for reporting

The validation step acts as a gate.

If the checks pass, the pipeline continues. If the checks fail, the pipeline stops or alerts the team.

This is much better than discovering the problem after the dashboard has already refreshed.

Using Great Expectations with Airflow

Great Expectations can be connected to Airflow so that validation runs as part of a DAG.

The typical flow looks like this:

1. Define expectations in Great Expectations
2. Save them in an Expectation Suite
3. Create a validation checkpoint
4. Call that checkpoint from an Airflow task
5. Decide what should happen when validation fails

The checkpoint is important because it packages the validation configuration. It tells Great Expectations which data to validate, which expectations to use, and what actions to take after validation.

For example, a checkpoint can validate a staging table and then update Data Docs or send a notification if the validation fails.

In practice, your DAG may include a task like this:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
import great_expectations as gx

def run_quality_check():
    context = gx.get_context()
    checkpoint = context.checkpoints.get("orders_checkpoint")
    result = checkpoint.run()
    if not result.success:
        raise ValueError("Data quality check failed for orders pipeline")
with DAG(
    dag_id="orders_pipeline",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    validate_orders = PythonOperator(
        task_id="validate_orders",
        python_callable=run_quality_check,
    )

This example keeps the logic simple.

The Airflow task runs the Great Expectations checkpoint. If the validation fails, the task raises an error, and the DAG stops at that point.

That is the core idea.

You do not want the rest of the pipeline to continue as if the data is fine when the validation has already failed.

Where the Validation Task Should Sit

The placement of the validation task matters.

If you put checks too early, you may only catch basic file or schema issues. If you put checks too late, bad data may already have reached downstream tables.

A good pattern is to validate data after it lands in a controlled staging area.

For example:

extract_source_data
        ↓
load_to_staging
        ↓
validate_staging_data
        ↓
transform_data
        ↓
publish_to_production

This works well because the staging table gives you a stable place to inspect the data before it is trusted.

If validation passes, the pipeline continues. If validation fails, the pipeline stops before production data is updated.

This pattern is especially useful for pipelines that power executive dashboards, billing tables, customer-facing metrics, or machine learning features.

Using the Write-Audit-Publish Pattern

A strong way to design Airflow validation is to use the Write-Audit-Publish pattern.

The idea is simple.

First, write the new data somewhere temporary or staged. Second, audit the staged data with quality checks. Third, publish it only if the audit passes.

This protects production tables from being overwritten by bad data.

For example, instead of writing directly into orders, the pipeline writes to orders_staging. Then the validation task checks orders_staging.

The checks may confirm that:

• The table is not empty
• order_id is not null
• order_id is unique
• customer_id is present
• order_total is not negative
• order_date is not in the future

If those checks pass, the pipeline can publish the staging data into the final orders table.

If they fail, the previous production table remains untouched.

That is a safer design.

It means a failed pipeline does not automatically become a broken dashboard.

When to Stop the Pipeline

Not every failed check should stop the DAG.

Some checks should block the pipeline because the data is unsafe. Others should only create a warning because the data is unusual but still usable.

For example, the DAG should probably fail if:

• A required source file is missing
• A production table has zero rows
• A primary key is null
• Duplicate IDs appear in a fact table
• A required relationship is broken
• A critical financial field contains invalid values

These failures can damage downstream reporting or business processes.

But the DAG may only warn if:

• A non-critical field has more nulls than usual
• A row count is slightly lower than expected
• A new category appears in a dimension
• A value is outside the normal range but still possible

This is where severity levels are useful.

Critical checks should fail the task. Lower-risk checks should notify the team. Expected dirty records can be dropped or quarantined, depending on the use case.

The worst approach is to make every issue fatal. That creates noisy pipelines that fail too often and eventually get ignored.

Sending Alerts When Checks Fail

A validation failure should not disappear inside Airflow logs.

Someone needs to know what happened.

At minimum, the alert should explain:

• Which pipeline failed
• Which dataset failed validation
• Which check failed
• When it failed
• Where to review the validation result

For example, a useful alert might say:

Data quality check failed in orders_pipeline.

Dataset: orders_staging
Failed check: order_id should not be null
Run date: 2026-02-13
Action: Production orders table was not updated

This is much better than a generic “task failed” message.

The alert should point the team toward the cause of the issue, not just tell them that something went wrong.

Depending on your setup, alerts can be sent through email, Slack, Microsoft Teams, PagerDuty, or another incident tool. The exact tool matters less than the clarity of the message.

A good alert reduces debugging time.

A bad alert creates more confusion.

Keep Airflow Checks Focused

Airflow should not become a dumping ground for every possible validation rule. A good approach is to use Airflow for checks that control pipeline flow.

For example:

• Should the pipeline continue?
• Should this data be published?
• Should downstream tasks run?
• Should the team be alerted?

Detailed model-level tests can live in dbt. Broader pipeline gates can live in Airflow.

This keeps the system easier to maintain.

Airflow controls the workflow. Great Expectations validates data at key gates. dbt tests validate transformation logic. Alerts help the team respond when something goes wrong.

Together, these pieces create a pipeline that does more than move data. It protects trust in the data as it moves.

Reviewing and Monitoring Data Quality Results

Running data quality checks is only the first part.

The real value comes from what happens after the checks run.

A validation system should make failures easy to understand, easy to investigate, and easy to act on. If a check fails but nobody knows what failed or why it failed, then the system is not helping much. It is just adding another task to the pipeline.

Good data quality monitoring should answer a few simple questions:

• Did the data pass or fail?
• Which dataset was checked?
• Which rule failed?
• What value or record caused the failure?
• Should the pipeline stop, warn, or continue?
• Who needs to respond?

Without this clarity, teams end up digging through logs, rerunning jobs, and guessing where the problem started.

Reading Validation Results

When a validation runs, the result should tell you more than just pass or fail.

A useful result shows the expectations that were executed, how many passed, how many failed, and what was observed in the data.

For example, suppose you have a check that says order_total should never be negative.

If the check fails, the result should show that the rule failed because one or more records had an order_total below zero. Ideally, it should also show how many records failed and, where safe, provide examples that help with debugging.

This matters because there is a big difference between these two messages:

Data validation failed.

And:

Data validation failed for fct_orders.
Failed rule: order_total should be greater than or equal to 0
Failed records: 18
Pipeline action: Publishing stopped

The second message is far more useful.

It tells the team what failed, where it failed, and what happened next.

Separating Data Errors from Technical Errors

Not every validation failure means the data is bad.

Sometimes the check fails because of a technical issue.

For example:

• The database connection failed
• The source table was not available
• The query timed out
• The validation configuration was wrong
• A column name changed and the expectation could not run

These are different from actual data quality problems.

A data quality issue means the data did not meet the expected rule. A technical issue means the validation itself could not run properly.

You should treat these differently.

If the data failed a business rule, the team needs to investigate the source data or transformation logic.

If the validation failed for technical reasons, the team may need to fix credentials, table access, timeout settings, package versions, or the validation configuration.

A good monitoring setup should make that distinction clear.

Using Data Docs and Test Reports

Validation results should not only live inside terminal output or Airflow logs.

They should be easy to review later.

This is where reports and documentation views become useful. Great Expectations, for example, can generate Data Docs that show expectation suites, validation runs, and pass/fail results in a more readable format.

This gives the team a history of quality checks.

Instead of asking, “Did this table pass yesterday?” you can review previous validation runs and compare results over time.

For dbt, test results can also be reviewed from command output, CI logs, build artifacts, or documentation tools depending on how your project is configured.

The important thing is to avoid hiding quality results in places nobody checks.

If the team depends on a dataset, the team should also be able to see whether that dataset passed its checks.

Monitoring Trends Over Time

A single failed check is useful, but trends are often more valuable.

For example, imagine a customer table where the percentage of null emails slowly increases over several weeks.

A single run may not look serious. Maybe the null rate moves from 2% to 3%. Then 5%. Then 8%. Eventually, the issue becomes large enough to affect marketing reports, customer segmentation, or product analytics.

If you only look at pass/fail results, you may miss the pattern.

That is why it helps to track metrics such as:

• Row counts over time
• Null rates in important columns
• Duplicate counts
• Failed test counts
• Freshness delays
• Distribution changes
• Number of warnings per run
• Number of critical failures per run

These metrics help the team see whether data quality is improving, getting worse, or becoming unstable.

For important datasets, data quality should be monitored like any other production signal.

You would not ignore rising application errors or slow API response times. Data pipelines deserve the same attention when they support business-critical decisions.

Designing Alerts That People Will Actually Use

Alerts are important, but they can easily become noise.

If every small issue sends a message to the team, people will eventually stop paying attention. A good alerting strategy focuses on impact.

Critical failures should alert immediately.

For example:

• A dashboard-critical table has zero rows
• A primary key is duplicated
• A required column is missing
• A finance metric is outside an acceptable range
• A production model failed its required tests

Warnings can be grouped or sent with lower urgency.

For example:

• A non-critical column has more nulls than usual
• A row count is slightly lower than expected
• A new category appeared in a dimension table
• A distribution changed but is still within a tolerable range

A useful alert should include enough context for someone to act without opening five different tools first.

At minimum, it should include:

• Pipeline name
• Dataset or model name
• Failed check
• Severity
• Run time
• Action taken
• Link or location for deeper investigation

For example:

Data quality failure: orders_pipeline

Dataset: orders_staging
Severity: critical
Failed check: order_id should be unique
Action: Publish step was blocked
Next step: Review duplicate order IDs in the validation report

This kind of alert is direct and useful.

It does not just say something failed. It explains what failed and what the system did in response.

Assigning Ownership

Data quality checks also need owners. If a check fails and nobody knows who should respond, the alert will sit there unresolved.

Ownership does not always mean one person. It can be a team.

For example:

• Source schema issues may belong to the ingestion or platform team
• Transformation logic issues may belong to analytics engineering
• Dashboard-facing metric issues may belong to the analytics team
• Source system changes may require coordination with product or application teams

Every important dataset should have a clear owner or responsible team.

This makes quality issues easier to route. It also helps teams avoid the common situation where everyone sees the alert, but nobody knows who is supposed to act on it.

Reviewing Failed Checks

When a check fails, the response should be structured.

A simple review process can look like this:

1. Confirm whether it is a real data issue or a technical validation issue.
2. Identify the dataset, column, and rule affected.
3. Check when the issue started.
4. Compare the failed batch with previous successful runs.
5. Look upstream for source changes, schema changes, or transformation changes.
6. Decide whether to fix, quarantine, backfill, or accept the anomaly.
7. Update the check if the expectation was too strict or outdated.

That last point matters.

Sometimes a check fails because the data is wrong. Other times, the business has changed and the expectation needs to be updated.

For example, a list of accepted order statuses may need to include a new value after a product change. In that case, the right fix is not to force the data back to the old rule. The right fix is to update the expectation so it matches the new business reality.

Data quality rules should be treated as living documentation.

They need to evolve as the product, data model, and business logic evolve.

Avoiding Alert Fatigue

A data quality system becomes less useful when it produces too much noise.

If checks fail constantly for issues nobody cares about, the team will stop trusting the checks. This is why every rule should have a clear purpose.

Before adding a check, ask:

• What problem does this check prevent?
• Who will respond if it fails?
• Should it fail the pipeline or only warn?
• Is the threshold realistic?
• Is the check still useful as the data changes?

If you cannot answer those questions, the check may not be worth adding yet.

A smaller number of meaningful checks is better than a large number of noisy checks.

The goal is not to prove that you have many tests. The goal is to protect important data and help the team respond quickly when something goes wrong.

Make Quality Results Part of the Workflow

Data quality monitoring should not be separate from the normal engineering workflow.

Validation results should show up where the team already works.

That may be:

• Airflow task status
• dbt test output
• CI/CD checks
• Slack or Teams alerts
• Data documentation
• Incident tools
• Dashboards for pipeline health

When checks are visible in the workflow, they are more likely to be used.

This is how data quality becomes part of everyday engineering practice instead of a separate cleanup activity.

The best systems do not only move data from one place to another. They make it clear when the data can be trusted, when it needs attention, and when it should not move forward.

Best Practices for Building Reliable Data Quality Checks

Adding data quality checks is not just about choosing a tool and writing a few rules.

The checks need to be useful, maintainable, and connected to how the team actually works. A good data quality system should help teams catch real issues early without creating unnecessary noise.

Here are some practical best practices to follow.

Start With the Most Important Datasets

Do not begin by trying to test every table in the warehouse.

That usually leads to too many checks, too much noise, and not enough impact.

Start with the datasets that matter most to the business. These are usually the tables that support dashboards, executive reports, customer-facing features, billing, financial reporting, machine learning models, or operational workflows.

For example, you may begin with:

• Customer tables
• Orders or transactions tables
• Revenue models
• Product usage tables
• Core reporting models
• Machine learning feature tables

These datasets deserve stronger validation because more people depend on them.

Once the important tables are covered, you can gradually expand quality checks to less critical datasets.

Protect the Basic Assumptions First

The most valuable checks are often the simplest.

Before adding advanced statistical tests, make sure the basic assumptions are protected.

For each important table, ask:

• What makes one row unique?
• Which columns must never be null?
• Which fields must follow a specific format?
• Which values should only come from a known list?
• Which relationships must exist between tables?
• Which numbers should never be negative?
• Which tables should never be empty?

These questions help you design checks that protect the meaning of the data.

For example, an orders table should probably have a unique order_id. A customers table should not have null customer IDs. A revenue table should not contain negative amounts unless the business logic explicitly allows refunds or adjustments.

Simple checks like this catch many of the issues that damage trust in data.

Keep Checks Close to the Logic They Protect

Data quality rules are easier to maintain when they live close to the pipeline logic.

If the check protects a dbt model, define it near the model in the YAML file. If the check protects a staging table in an Airflow workflow, place it in the DAG at the point where that table is created or published.

This makes the system easier to understand.

When someone changes a model, they should be able to see the tests that protect it. When someone reviews a pipeline, they should be able to see where validation happens before data moves forward.

Checks that live far away from the pipeline often become stale. People forget they exist, do not update them when business logic changes, and only notice them when they fail unexpectedly.

Quality rules should feel like part of the pipeline, not a separate side project.

Use Severity Levels Intentionally

Every check should have a clear response.

Some checks should fail the pipeline. Some should only warn the team. Some should quarantine bad records while allowing clean records to continue.

A critical check should fail when bad data would cause serious downstream damage.

For example:

• A primary key is missing
• A fact table has duplicate IDs
• A production table has zero rows
• A required source file is missing
• A financial metric is outside an acceptable range
• A key relationship between tables is broken

A warning is better when the issue is worth reviewing but does not make the dataset completely unusable.

For example:

• A non-critical column has more nulls than usual
• A value appears outside the normal range
• A new category appears in a dimension table
• Row count is lower than expected, but not dangerously low

The important thing is to avoid making everything fatal.

If every small issue breaks the pipeline, people will eventually start ignoring the checks or disabling them. A reliable quality system should match the response to the level of risk.

Make Checks Easy to Debug

A check that fails without context is frustrating.

A useful check should make the problem easier to investigate. It should tell the team what failed, where it failed, and why it matters.

For example, this is not very helpful:

Validation failed.

This is much better:

Validation failed for fct_orders.

Failed check: order_id should be unique
Failed records: 42
Action: Publish step was blocked

The second message gives the team enough information to begin debugging immediately.

When possible, validation results should include:

• Dataset name
• Column name
• Failed rule
• Number of failed records
• Observed value
• Expected value
• Severity
• Pipeline action taken

This turns a failed check into a useful signal instead of a vague error.

Avoid Hardcoding Rules That Change Often

Some rules are stable.

For example, a primary key should be unique. A required ID should not be null. A transaction date should be a valid date.

Other rules change as the business changes.

For example, accepted order statuses may change when a new product flow is introduced. Valid country codes may expand when the business enters a new market. A maximum allowed transaction amount may increase as the company grows.

For rules that change often, avoid hardcoding values in too many places.

Instead, consider storing reference values in lookup tables, configuration files, seeds, or central model definitions. This makes the rules easier to update when the business changes.

A good data quality check should protect the business logic, not freeze the business in its old state.

Track Quality Over Time

A single validation result tells you what happened in one run.

A trend tells you whether the data is getting better or worse.

This is why teams should track quality metrics over time, especially for critical datasets.

Useful metrics include:

• Number of failed checks per run
• Number of warnings per run
• Null rates for important columns
• Duplicate counts
• Row counts
• Freshness delays
• Schema changes
• Percentage of records passing validation

These trends help you spot slow-moving problems.

For example, a column may not suddenly fail overnight, but its null rate may slowly increase over several weeks. If nobody is tracking that trend, the issue may only become visible after it affects a report or business decision.

Data quality should be monitored like any other production concern.

Review and Remove Bad Checks

Not every check remains useful forever.

Some checks become outdated. Some are too strict. Some are too loose. Some create noise but do not prevent real problems.

That is why data quality checks should be reviewed regularly.

A good review asks:

• Is this check still relevant?
• Has the business logic changed?
• Does this check catch real issues?
• Does it fail too often for acceptable reasons?
• Should the severity be changed?
• Should the threshold be adjusted?
• Does anyone respond when it fails?

If the answer is no, update the check or remove it.

A smaller set of useful checks is better than a large set of ignored checks.

The goal is not to collect as many tests as possible. The goal is to protect trust in the data.

Add Checks to Development and CI/CD

Data quality should not only run in production.

Some checks should run earlier, while engineers and analytics engineers are still developing changes.

For dbt projects, this means running tests locally before opening a pull request and again in CI/CD before merging. For Python or Airflow pipelines, this may mean running validation on sample data during development and adding quality checks to automated deployment workflows.

This catches issues before they reach scheduled production jobs.

For example, if a model change introduces duplicate IDs, it is better to catch that in a pull request than after the production dashboard has refreshed.

The earlier a data issue is caught, the easier it is to fix.

Document What Each Check Protects

Data quality checks are also a form of documentation.

They explain what the team expects from the data.

For example, a test that says order_id must be unique is also saying something important about the model: one row should represent one order.

A test that says order_status must be one of five accepted values documents the valid lifecycle states of an order.

A test that says customer_id must exist in the customers table documents the relationship between two models.

This is why checks should be named clearly and written in a way that other people can understand.

Good checks help future team members understand the data model faster. They also reduce confusion during reviews, debugging, and handovers.

Treat Data Quality as an Ongoing Practice

Data quality is not something you set up once and forget. Pipelines change. Source systems change. Product behavior changes. Business rules change. New tables are added. Old assumptions become outdated.

Your quality checks need to evolve with the system.

The best teams treat data quality as part of normal pipeline development. When a new model is created, basic checks are added. When a bug is found, a new check is added to prevent the same issue from happening again. When a business rule changes, the related expectations are updated.

This turns data quality from a cleanup activity into a continuous engineering practice.

Reliable pipelines are not built only by moving data successfully. They are built by checking that the data still means what the business thinks it means.

Till next time

Thank you for reading !