Your Model is Fine. Your Data Isn’t. DataFrame Validation with Pandera

Hello everyone! In my previous articles, I talked about LLMs. Now it is time to talk about data. Have you ever had a bug caused by your data after spending a lot of time working on your model?

You spend days improving your model. You explore and analyze your data. Maybe you even fine-tune your model. Then one day, something changes. For example, a date value becomes empty. Your model starts getting new data, not because the real world changed, but because your data changed.

The model didn't break. The data did.

This is more common than you think. In most real-world ML pipelines, the silent killer isn't a bad model. It is bad data. The bad data comes in quietly. There are no errors and no warnings, but the results are wrong.

This is where Pandera helps. Pandera is a open-source Python library. It helps you check your data. You can set rules for your DataFrames and check them before the data goes to your model. In this post, we will go from basic column checks all the way to integrating Pandera with Pydantic and Hypothesis for stronger data checks in real projects.

Let's stop blaming the model.

✨ What is Pandera?

Pandera is an open-source Python library for data validation. It works with Pandas, Polars, Spark DataFrames and more. Basically, we define a schema then pandera checks if our DataFrame follows those rules or not. If something is wrong it raises an error.

You can think it is like a type checker for our data. For example, we can say this column must be an integer or this column cannot be null.

Before using Pandera, we need to install it:

uv install pandera
uv install pandera[polars]
uv install pandera[pandas]

✨Pandera Basics

Let’s imagine, we have a simple ordering dataset and we want to validate it.

import polars as pl

df = pl.DataFrame({
    "order_id": [1, 2, 3],
    "customer_name": ["Alice", "Bob", "Charlie"],
    "amount": [10.5, 20.0, 15.75],
    "order_date": ["2024-01-01", "2024-01-02", "2024-01-03"]
})

We can define a schema and tell Pandera what to expect from each column. For example order_id must be an integer.

import pandera.polars as pa

schema = pa.DataFrameSchema({
    "order_id": pa.Column(int),
    "customer_name": pa.Column(str),
    "amount": pa.Column(float),
    "order_date": pa.Column(str)
})

schema.validate(df)

If something is wrong, you will see errors in your terminal.

df_broken = pl.DataFrame({
    "order_id": [1, 2, 3],
    "customer_name": ["Alice", "Bob", "Charlie"],
    "amount": [10.5, None, 15.75],
    "order_date": ["2024-01-01", "2024-01-02", "2024-01-03"]
})

try:
    schema.validate(df_broken)
except pa.errors.SchemaError as e:
    print("Data is invalid!")
    print(e)

You can also define a class-based schema to validate your data.

import pandera.polars as pa
import polars as pl

df_broken = pl.DataFrame({
    "order_id": [1, 2, 3],
    "customer_name": ["Alice", "Bob", "Charlie"],
    "amount": [10.5, None, None],  # null değerler
    "order_date": ["2024-01-01", None, "2024-01-03"]  # null değerler
})

class OrderSchema(pa.DataFrameModel):
    order_id: int
    customer_name: str
    amount: float
    order_date: str

try:
    OrderSchema.validate(df_broken)
except pa.errors.SchemaError as e:
    print("Data is invalid!")
    print(e)

In this example we have 2 errors. One in the amount column and one in the order_date column. But we see only the first error when we run the code. Pandera stops at the first error and shows it. If we want to see all errors, we can use the lazy option.

try:
    OrderSchema.validate(df_broken, lazy=True)
except pa.errors.SchemaErrors as e:
    print("Data is invalid!")
    print(e)

✨ DataFrame Schema Column Constraints

When we define a schema then we need to tell Pandera what to expect from each column. Can this column be empty? Should Pandera change the data type? Is this column required? These are all column constraints, and Pandera gives us many built-in options to handle them.

nullable: This lets a column have empty (null) values.

import pandera.polars as pa
import polars as pl

schema = pa.DataFrameSchema({
    "name": pa.Column(str),
    "age": pa.Column(int, nullable=True)
})

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25, None, 30]
})

schema.validate(df)
print("1. DataFrame is valid!")

coerce: This changes the column data to the correct type automatically.

schema = pa.DataFrameSchema({
    "age": pa.Column(int, coerce=True)
})

df = pl.DataFrame({
    "age": ["25", "30", "35"]  # strings, not integers
})

schema.validate(df)
print("2. DataFrame is valid!")

required: This means the column must be in the DataFrame. (Default: yes)

schema = pa.DataFrameSchema({
    "name": pa.Column(str),
    "age": pa.Column(int, required=False)  # age is optional
})

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"]
    # age column is missing, but that is okay
})

schema.validate(df)
print("3. DataFrame is valid!")

strict: This checks for extra columns. (Default: extra columns are okay)

schema = pa.DataFrameSchema({
    "name": pa.Column(str),
    "age": pa.Column(int)
}, strict=True)

df = pl.DataFrame({
    "name": ["Alice", "Bob"],
    "age": [25, 30],
    "extra_column": ["x", "y"]  # this column is not in the schema
})

try:
    schema.validate(df)
except pa.errors.SchemaError as e:
    print("4. DataFrame Validation failed!")
    print(e)

unique: This makes sure all values in a column are different.

schema = pa.DataFrameSchema({
    "order_id": pa.Column(int),
    "name": pa.Column(str)
}, unique=["order_id"])

df = pl.DataFrame({
    "order_id": [1, 1, 3],  # order_id 1 is duplicated
    "name": ["Alice", "Bob", "Charlie"]
})

try:
    schema.validate(df)
except pa.errors.SchemaError as e:
    print("5. DataFrame Validation failed!")
    print(e)

ordered: This checks if columns are in the correct order.

schema = pa.DataFrameSchema({
    "order_id": pa.Column(int),
    "name": pa.Column(str),
    "amount": pa.Column(float)
}, ordered=True)

df = pl.DataFrame({
    "name": ["Alice", "Bob"],       # wrong order!
    "order_id": [1, 2],
    "amount": [10.5, 20.0]
})

try:
    schema.validate(df)
except pa.errors.SchemaError as e:
    print("6. DataFrame Validation failed!")
    print(e)

add_missing_columns: If a column is missing, Pandera adds it automatically when it checks the data.

schema = pa.DataFrameSchema(
{
    "name": pa.Column(str),
    "age": pa.Column(int, default=0),
    "score": pa.Column(float, nullable=True)
}, 
add_missing_columns=True,
coerce=True,
)

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"]
    # age and score columns are missing
})

validated_df = schema.validate(df)
print("7. Validated DataFrame with missing columns added:")
print(validated_df)

✨DataFrame Schema Column Checks

Pandera provides many built-in checks for columns. You can use them to validate values without writing any custom code. For example, you can check if a value is greater than a number, if it is in a list, or if it contains a specific string.

greater_than_or_equal_to: This checks if a value is greater than or equal to a given number.

less_than_or_equal_to: This checks if a value is less than or equal to a given number.

schema = pa.DataFrameSchema({
    "age": pa.Column(int, pa.Check.greater_than_or_equal_to(18)),
    "score": pa.Column(float, pa.Check.less_than_or_equal_to(100.0))
})

df = pl.DataFrame({
    "age": [25, 17, 30],  # 17 is less than 18!
    "score": [85.0, 90.0, 105.0]  # 105 is greater than 100!

})

try:
    schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as e:
    print("Validation failed!")
    print(e)

isin: This checks if a value is one of the allowed values in a given list.

# isin
schema = pa.DataFrameSchema({
    "status": pa.Column(str, pa.Check.isin(["active", "inactive", "pending"]))
})

df = pl.DataFrame({
    "status": ["active", "unknown", "pending"]  # unknown is not allowed!
})

try:
    schema.validate(df)
except pa.errors.SchemaError as e:
    print("Validation failed!")
    print(e)

str_contains: This checks if a value contains a specific text.

schema = pa.DataFrameSchema({
    "email": pa.Column(str, pa.Check.str_contains("@"))
})

df = pl.DataFrame({
    "email": ["alice@example.com", "notanemail", "bob@example.com"]
})

try:
    schema.validate(df)
except pa.errors.SchemaError as e:
    print("Validation failed!")
    print(e)

You can also write your own check function to validate your data.

import pandera.polars as pa
import polars as pl

def is_divisible_by_3(x):
    return x % 3 == 0

def mean_greater_than_10(polars_data):
    series = polars_data.lazyframe.collect()[polars_data.key]
    return series.mean() > 10

schema = pa.DataFrameSchema({
    "number": pa.Column(
        int,
        pa.Check(is_divisible_by_3, name="divisible_by_3", element_wise=True)
    ),
    "salary": pa.Column(
        int, 
        pa.Check(mean_greater_than_10, name="mean_must_be_greater_than_10")
    )

})

df = pl.DataFrame({
    "number": [3, 6, 8],  # 8 is not divisible by 3!
    "salary": [5, 6, 7]  # mean is 6, not greater than 10!

})                                                                                                                          

try:
    schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as e:
    print("Validation failed!")
    print(e)

We can set the element_wise argument to control how Pandera checks the data.

• If element_wise = True, Pandera checks each value one by one. We use True to check if every value is greater than 0.
• If element_wise = False, Pandera checks the whole column together. The default value is False. It is really useful for statistical calculations such as mean, sum, and standard deviation. In our example we use False to check if the average of the column is greater than 0.

Also, we can combine multiple checks for a single column.

import pandera.polars as pa
from pandera.polars import DataFrameSchema, Column, Check
import polars as pl

def is_divisible_by_3(x: pl.Series) -> pl.Series:
    return x % 3 == 0

schema = DataFrameSchema({
    "number": Column(
        int,
        checks=Check(is_divisible_by_3, element_wise=True, name="divisible_by_3"),
    ),
    "salary": Column(
        int,
        checks=Check(lambda s: s.mean() > 10,
                     element_wise=False,
                     name="mean_must_be_greater_than_10"
        ),
    ),
})

df = pl.DataFrame({
    "number": [3, 6, 8],
    "salary": [5, 6, 7]
})

try:
    schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as e:
    print("Validation failed!")
    print(e)

✨DataFrame Models Column Constraints & Checks

I talked about DataFrame Model before. Now we can learn how to use checks and rules inside a model. In this example, we define columns using the Field class and set their type. We use Field class to set rules for each column. For example, the age column can be null.

# nullable and coerce with Field
class UserSchema(pa.DataFrameModel):
    name: str
    age: int = pa.Field(nullable=True, coerce=True)
    score: float = pa.Field(ge=0.0, le=100.0)

df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": ["25", None, "30"],
    "score": [85.0, 90.0, 95.0]
})

schema = UserSchema.validate(df)
print("Data is valid!")
print(schema)

Pandera can change the data type automatically. So even if we write age values as strings, Pandera converts them to integers. That is why the schema is still valid.

We can use other schemas and create a new schema from them.

class BaseSchema(pa.DataFrameModel):
    order_id: int = pa.Field(gt=0)
    customer_name: str

class ExtendedSchema(BaseSchema):
    amount: float = pa.Field(ge=0.0)
    status: str = pa.Field(isin=["active", "inactive", "pending"])

df = pl.DataFrame({
    "order_id": [1, 2, 3],
    "customer_name": ["Alice", "Bob", "Charlie"],
    "amount": [10.5, 20.0, 15.75],
    "status": ["active", "inactive", "pending"]
})

ExtendedSchema.validate(df)
print("Extended Schema Data is valid!")

We can also create our own checks for each column in the dataframe model.

class OrderSchema(pa.DataFrameModel):
    order_id: int
    amount: float

    @pa.check("amount", name="positive_amount", element_wise=True)
    def amount_must_be_positive(cls, amount):
        return amount > 0

df = pl.DataFrame({
    "order_id": [1, 2, 3],
    "amount": [10.5, -5.0, 15.75]  # -5.0 is not valid!
})

try:
    OrderSchema.validate(df)
except pa.errors.SchemaError as e:
    print("Validation failed!")
    print(e)

Pandera supports Polars native data types. This means you can use types like pl.Int8, pl.Float32, pl.Date instead of generic Python types. Polars also supports nested data types like pl.List and pl.Struct. Also, Pandera can validate these as well.

import pandera.polars as pa
import polars as pl

schema = pa.DataFrameSchema({
    "int8_col": pa.Column(pl.Int8),
    "int16_col": pa.Column(pl.Int16),
    "uint8_col": pa.Column(pl.UInt8),
    "float32_col": pa.Column(pl.Float32),
})

df = pl.DataFrame({
    "int8_col": pl.Series([1, 2, 3], dtype=pl.Int8),
    "int16_col": pl.Series([1, 2, 3], dtype=pl.Int16),
    "uint8_col": pl.Series([1, 2, 3], dtype=pl.UInt8),
    "float32_col": pl.Series([1.0, 2.0, 3.0], dtype=pl.Float32),
})

schema.validate(df)
print("Data is valid!")

import pandera.polars as pa
import polars as pl

class NestedSchema(pa.DataFrameModel):
    list_col: pl.List(pl.Int64)
    struct_col: pl.Struct({"name": pl.String, "age": pl.Int64})

df = pl.DataFrame({
    "list_col": [[1, 2], [3, 4], [5, 6]],
    "struct_col": [
        {"name": "Alice", "age": 25},
        {"name": "Bob", "age": 30},
        {"name": "Charlie", "age": 35}
    ],
})

NestedSchema.validate(df)
print("Data is valid!")

✨Pandera with Pandas

Until now, we used Polars in our examples. However, Pandera has some features that only work with Pandas. Let’s look at them.

SeriesSchema A Series is a single column of data. SeriesSchema allows us to validate it directly without using a DataFrame. However, Pandera doesn’t support validating a Series in Polars. Instead, you need to use a DataFrame with one column.

import pandera.pandas as pa
import pandas as pd

# SeriesSchema
schema = pa.SeriesSchema(
    str,
    checks=[
        pa.Check(lambda s: s.str.startswith("foo")),
        pa.Check(lambda s: s.str.endswith("bar")),
        pa.Check(lambda x: len(x) > 3, element_wise=True)
    ],
    nullable=False,
    unique=False,
    name="my_series"
)

validated_series = schema.validate(
    pd.Series(["foobar", "fooexamplebar", "foolorembar"], name="my_series")
)
print(validated_series)

MultiIndex A DataFrame has an index, which is like a label for each row. By default, it is a sequence starting from 0, but you can also choose a column to use as the index.

df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie"]
})
print(df)

#    name
# 0  Alice       <- 0 index
# 1  Bob         <- 1 index
# 2  Charlie     <- 2 index

A MultiIndex means a DataFrame has more than one index level For example, one level can be a id and another can be a name.

df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie"]
})
print(df)

#         name
# foo  0  Alice      <- "foo" and 0 together as the index.
# bar  1  Bob        <- "bar" and 1 together as the index.
# foo  2  Charlie    <- "foo" and 2 together as the index.

In the example above, the schema uses two index levels: city and year. The city index must be one of “New York”, “Los Angeles”, or “Chicago”, and the year index must be one of 2022, 2023, or 2024. Pandera checks both before the data is validated.

This feature is specific to pandas because Polars doesn’t have an index system.

import pandera.pandas as pa
import pandas as pd

schema = pa.DataFrameSchema(
    columns={"revenue": pa.Column(float)},
    index=pa.MultiIndex([
        pa.Index(str,
            pa.Check(lambda s: s.isin(["New York", "Los Angeles", "Chicago"])),
            name="city"),
        pa.Index(int,
            pa.Check(lambda s: s.isin([2022, 2023, 2024])),
            name="year"),
    ])
)

df = pd.DataFrame(
    data={"revenue": [100.0, 200.0, 150.0]},
    index=pd.MultiIndex.from_arrays(
        [["New York", "Los Angeles", "Chicago"], [2022, 2023, 2024]],
        names=["city", "year"]
    )
)

schema.validate(df)
print("Data is valid!")

Alias

Variable names cannot start with a number in Python. However, our DataFrame can have column names like 2020 or my-column, which are not valid Python variable names. In these cases, you can use an alias to map the column name to a valid Python variable name.

Polars column names must be strings, so they already follow a consistent format. Because of this, we usually don’t need to use aliases when working with Polars.

import pandas as pd
import polars as pl

df = pd.DataFrame({2020: [99, 50, 75]})
print("Pandas DataFrame:")
print(df)

try:
    df = pl.DataFrame({2020: [99, 50, 75]})  # this will fail!
except TypeError as e:
    print(f"\nPolars error: {e}")

In this example, our DataFrame has a column named 2020. It isn’t a valid variable name so we define it as col_2020 in the schema and set alias to 2020. This tells Pandera to look for the column 2020 in the DataFrame.

We also add a custom check with @pa.check. Here, we use the alias name. When using @pa.check, we need to give the column name. If the column uses an alias we must use the alias not the class attribute name.

# Alias
class Schema(pa.DataFrameModel):
    col_2020: pa.typing.Series[int] = pa.Field(alias=2020)

    @pa.check(2020)
    def int_column_lt_100(cls, series):
        return series < 100

df = pd.DataFrame({2020: [99]})
print(Schema.validate(df))

✨Pandera with Pydantic

Pydantic is a open source Python library that helps us validate data. We can use Pandera and Pydantic together. There are two ways to do this:

• We can use DataFrame inside a Pydantic model. Pandera validates the Dataframe when the object is created.
• We can validate each row of a DataFrame as a Pydantic model.

import pandas as pd
import pandera.pandas as pa
import pydantic
from pandera.typing import DataFrame, Series

class OrderSchema(pa.DataFrameModel):
    order_id: Series[int] = pa.Field(unique=True, gt=0)
    amount: Series[float] = pa.Field(gt=0)
    status: Series[str] = pa.Field(isin=["pending", "completed", "cancelled"])

class PipelineConfig(pydantic.BaseModel):
    pipeline_name: str
    version: int
    orders: DataFrame[OrderSchema]

try:
    invalid_df = pd.DataFrame({
        "order_id": [1, 1, 3],       # duplicate order_id!
        "amount": [100.0, -50.0, 150.0],  # negative amount!
        "status": ["pending", "unknown", "cancelled"]  # unknown status!
    })
    PipelineConfig(pipeline_name="order_pipeline",
                   version=1,
                   orders=invalid_df)
except pydantic.ValidationError as e:
    print("Validation failed!")
    print(e)

from pydantic import BaseModel
from pandera.engines.pandas_engine import PydanticModel

class Transaction(BaseModel):
    transaction_id: int
    amount: float
    currency: str

class TransactionSchema(pa.DataFrameModel):
    class Config:
        dtype = PydanticModel(Transaction)
        coerce = True

try:
    invalid_df = pd.DataFrame({
        "transaction_id": [1, 2, 3],
        "amount": [100.0, "invalid", 150.0],  # invalid amount!
        "currency": ["USD", "EUR", "GBP"]
    })
    TransactionSchema.validate(invalid_df)
except pa.errors.SchemaErrors as e:
    print("Validation failed!")
    print(e)

✨Property Based Testing for DataFrames

In normal unit test we provide specific inputs and outputs pair. However, we are human and we can miss important cases or hidden bugs. In property based testing we just provide strategies and test generates its own data automatically. Hypothesis is an open source Python library that helps us write property-based tests and Pandera works with Hypothesis to do this. First, we need to install Hypothesis.

uv add pandera[hypotheses] hypothesis

We can generate an example DataFrame using example() method.

import pandera.pandas as pa
import pandas as pd
from hypothesis import given, settings, HealthCheck

class OrderSchema(pa.DataFrameModel):
    order_id: int = pa.Field(gt=0, lt=100)
    amount: float = pa.Field(gt=0.0, lt=1000.0)
    status: str = pa.Field(isin=["pending", "completed", "cancelled"])

df = OrderSchema.example()
print("Generated example:")
print(df)

We can also use the strategy() method with Hypothesis to automatically generate random DataFrames that match our schema.

@given(OrderSchema.strategy(size=3))
@settings(
    max_examples=5,
    suppress_health_check=[HealthCheck.filter_too_much, HealthCheck.too_slow]
)
def test_order_schema(df):
    # validate the generated DataFrame
    OrderSchema.validate(df)
    # add your own assertions here
    assert (df["order_id"] > 0).all()
    assert (df["amount"] > 0.0).all()

test_order_schema()
print("All tests passed!")

At the end of the day our goal is to make sure the data we send to our model is clean and correct. Bad data can cause hidden bugs, wrong predictions and errors that are hard to find. We use Pandera to check our data before we use it. Pandera helps us find problems early so bad data doesn’t reach the model. It checks things like data types, missing values, and rules we define.

Remember: your model is usually not the problem. The real problem is often the data until you validate it.

You can find the source codes here. I hope this article is helpful for you. See you in the next article!

✨ Resources

[1]https://pandera.readthedocs.io/en/stable/ [2]https://pandera.readthedocs.io/en/stable/pydantic_integration.html#pydantic-integration [3]https://pandera.readthedocs.io/en/stable/data_synthesis_strategies.html#data-synthesis-strategies [4]How to Use Pandas With Pandera to Validate Your Data in Python by ArjanCodes

Grammar checked by Gemini.