Data validation with Pandera

In 2021, Zillow, a major real estate marketplace in the USA, shocked the tech and real-estate worlds when it abruptly shut down its highly publicized home-flipping business, after its AI-driven pricing model made hundreds of millions of dollars’ worth of bad bets. The culprit wasn’t a fancy algorithm gone rogue. It was something far more fundamental: bad data. Inaccurate, inconsistent, and poorly validated housing inputs fed directly into Zillow’s models, causing wildly overconfident predictions and ultimately a massive financial failure.

Stories like Zillow’s are becoming increasingly common as organizations rush to build data-powered products and AI systems. But beneath many of these failures lies the same hidden flaw: a lack of rigorous data validation. Data validation, the process of ensuring that data is accurate, consistent, and reliable before it’s used, may not sound glamorous, but it’s one of the most critical steps in any data science or AI project. When it’s ignored, even the most sophisticated models can crumble. When it’s done well, it becomes a silent guardian that keeps prediction pipelines trustworthy, stable, and safe.

In a world where data is the fuel of AI, validation is the quality check that ensures the engine runs smoothly. And as the stakes grow, whether in finance, healthcare, hiring, or autonomous systems, so does the importance of getting it right.

Introducing Pandera

With the importance of data validation established, the natural next step is to look at how it can be woven into everyday data pipelines. For Python-based workflows, Pandera is a particularly compelling choice.

Pandera is a Python library that brings data validation to the world of data science and analytics by letting you define schemas for your data, much like Pydantic does for Python objects, but tailored specifically for Pandas DataFrames and Series. With Pandera, you can declaratively specify column types, allowed ranges, categories, custom checks, and statistical properties, to then automatically validate your datasets before they’re used in analysis or machine-learning pipelines.

In short, Pandera helps you catch data issues early, enforce data quality standards, and build more reliable, reproducible pipelines, all while fitting naturally into the workflows of Pandas, Polars, and PySpark users.

When Data Meets Its Contract

To illustrate the workings of Pandera let’s go through some of its major features by applying them to a sample set of data. Sticking to the topic of house prices, we will be using house pricing data made available by Kaggle.

At the core of Pandera’s validation system lies the DataFrameSchema: an object-based representation of a DataFrame that acts as a blueprint for how your data is expected to look. It defines column names, data types, and validation rules, and serves as the foundation for all schema-based validation in Pandera.

In this post, however, we will work with the DataFrameModel, a higher-level abstraction built on top of DataFrameSchema. DataFrameModel is heavily inspired by Pydantic and allows you to define DataFrame schemas using familiar Python class syntax and type annotations. For anyone accustomed to working with Pydantic models, this approach feels natural and expressive, while still offering the full power of Pandera’s validation engine under the hood.

Let’s see what this looks like in practice.

from pandera import DataFrameModel

class HousingSchema(DataFrameModel):
    price: float
    area: int
    bedrooms: int
    stories: int
    guestroom: str
    basement: str
    parking: int
    furnishingstatus: str

In the example above, we have defined HousingSchema, a Pandera DataFrameModel, in which all fields of the dataset are defined as attributes of the schema. Behind each attribute you find a type annotation to specify the expected data type of the values in each respective column. This already gives us our first validation, namely that all values in the corresponding DataFrame column conform to the declared type. Let’s look at how one can use this to validate a dataset.

import pandas as pd
from pandera.pandas import typing as pa_typing
from pathlib import Path

def read_housing_data(path: Path) -> pa_typing.DataFrame[HousingSchema]:
    return pa_typing.DataFrame[HousingSchema](
        pd.read_csv(path)
    )

df = read_housing_data(path=Path() / "path" / "to" / "housing.csv")

In the above code snippet, we import the necessary libraries, we define a function to read the housing .csv file, and on the last line the output of the function gets assigned to a variable. The validation is performed by feeding the output of the Pandas read_csv function to the call parameters of the Pandera DataFrame[HousingSchema] class.

Note that the input and output variables have been annotated with types. This is another benefit of using Pandera DataFrameModels, as it works nicely together with type checking tools like MyPy, thereby giving developers the possibility to be very precise in what input and output types functions should have.

Executing the above gives us a clear Pandera schema error:

pandera.errors.SchemaError: expected series 'price' to have type float64, got int64

Apparently, the content of the price column does not match with what the schema prescribes, which is actually desirable in this stage as it clearly exposes the mismatch between expectations and reality. That said, we ultimately want the prices to be represented as floats in our DataFrame. For that we can use Pandera’s coercion functionality, which can be set per column, or, and that is what will do here, be activated Schema wide by modifying the configuration of our schema. Below we have defined a small BaseSchema that functions as a parent schema which our HousingSchema will be inheriting from. By setting coerce = True, Pandera will try to coerce all input columns to the attributed type.

class BaseSchema(DataFrameModel):
    class Config:
        coerce = True

class HousingSchema(BaseSchema):
    price: float
    area: int
    ...

With this setting, our read_housing_data function now returns a proper DataFrame.

>>> print(df)

        price  area  bedrooms  ...  basement parking furnishingstatus
0  13300000.0  7420         4  ...        no       2        furnished
1  12250000.0  8960         4  ...        no       3        furnished
2  12250000.0  9960         3  ...       yes       2   semi-furnished
3  12215000.0  7500         4  ...       yes       3        furnished
4  11410000.0  7420         4  ...       yes       2        furnished

From Types to Domain Constraints

Basic type validation is rarely sufficient in real-world datasets, so Pandera allows you to encode richer domain constraints directly into your schema. To expand the validation rules, we can make use of another Pydantic inspired feature which is Pandera’s Field object. With a Field object we can extend the definition of a column to add more validation rules. For example, we want the columns like bedrooms and stories to carry values that are at least greater or equal to 0.

from pandera import Field

class HousingSchema(BaseSchema):
    price: float = Field(ge=0)
    area: int = Field(ge=0)
    ...

In addition, one can constrain the present categorical fields by using the Pandera Category type which limits the allowed values to those specified in the categories parameter.

from pandera import Field
from pandera.pandas import typing as pa_typing

class HousingSchema(BaseSchema):
    price: float = Field(ge=0)
    ...
    guestroom: pa_typing.Category = Field(
        dtype_kwargs={"categories": ["yes", "no"]}
    )
    basement: pa_typing.Category = Field(
        dtype_kwargs={"categories": ["yes", "no"]}
    )
    furnishingstatus: pa_typing.Category = Field(
        dtype_kwargs={
            "categories": ["furnished", "unfurnished", "semi-furnished"],
        }
    )

And when built-in constraints fall short, Pandera lets you encode arbitrary business logic as executable validation rules at both the field and dataframe level.

For the field specific case, one can simply define a method to the DataFrameModel class and decorate it with acheck decorator that is marked to be operated on the desired field.

from pandera import check

class HousingSchema(BaseSchema):

    price: float
    ...

    @check("price")
    @classmethod
    def custom_validation(cls, series: pa_typing.Series) -> bool:
        return series.mean() > 1000000

In above code snippet, the price field enforces that the mean of the corresponding column exceeds 1,000,000.

Lastly, for custom dataframe wide validations there is the dataframe_check decorator. In below example the decorated method ensures that at least one of the 3 corresponding columns has a value greater than 1.

from pandera import dataframe_check

class HousingSchema(BaseSchema):
    price: float
    ...

    @dataframe_check
    @classmethod
    def custom_df_validation(cls, df: pd.DataFrame) -> pa_typing.Series[bool]:
        return df[["bedrooms", "parking", "stories"]].sum(axis=1) > 0

Schema-Guided Data Transformation

In addition to validating data, Pandera can safely transform it as part of the schema enforcement process. To illustrate the parsing capabilities, we will zoom into the contents of the guestroom and basement columns which contain only the two values yes and no. To make those values more compatible with any potential downstream process it makes more sense to convert them to boolean values. For this, we can apply another Pandera feature which is called dataframe parsing. Below a method to convert all yes and no to their boolean counterparts has been added to our HousingSchema class.

class HousingSchema(BaseSchema):
    guestroom: pa_typing.Series[bool]
    basement: pa_typing.Series[bool]
    ...

    @pandera.dataframe_parser
    def convert_to_bool(self, df: pd.DataFrame) -> pd.DataFrame:
        df[["guestroom", "basement"]] = df[["guestroom", "basement"]].replace(
            {"yes": True, "no": False}
        )
        return df

Note that the types of the guestroom and basement attributes have been changed fromstr to bool. And the added convert_to_bool method is decorated with the Pandera dataframe_parser function, turning this method into a dataframe operation that is executed before the validation rules get applied. So even though the incoming column values are strings, the conversion ensures the final values in those columns become of type bool and hence the type validation passes successfully.

Like with the check decorator for field validation, there is a parser decorator that enables parsing logic to be applied on a specific field. However, in above example this is not applicable, as the type coercion of the values happens before the field parser operation and hence all string values get cast toTrue (not an empty string).

Stress-Testing Your Data Contracts

At this point, we have spent quite some time carefully unraveling the content of our input data and expanding a DataFrameModel to configure the identified constraints accordingly. This greatly simplifies the construction of DataFrame processing pipelines by allowing each step to explicitly annotate its inputs and outputs, making it clear how the data is expected to look at every stage. However, despite elaborate field constraints, certain edge cases may remain unaccounted for, allowing unexpected value combinations to propagate through the pipeline and result in failures or silent incorrect behavior.

For such scenarios, Pandera’s compatibility with Hypothesis could offer the helping hand. Hypothesis is a library to synthesize test data based on the input types of a function, enabling a testing method called property-based testing. What this means in the context of Pandera, is that you can:

1. Synthesize multiple sample dataframes with data that falls within the constraints of your Pandera schema
2. Feed the synthesized dataframes to a processing function
3. Test if the processing function fails for certain input dataframes.

Any failure exposes a potential weakness of the implementation.

To illustrate this process, we will return to our Housing dataset. As an example processing step we will introduce a function to compute the price per area. This requires us to define a new HousingProcessedSchema as well as to make slight adjustments to our input HousingSchema.

from pandera import Field
from pandera.pandas import typing as pa_typing
from pandera.pandas import check_types

class HousingSchema(BaseSchema):
    price: float = Field(in_range={"min_value": 0, "max_value": 10 * 8})
    area: int = Field(in_range={"min_value": 0, "max_value": 10 * 5})
    bedrooms: int = Field(in_range={"min_value": 0, "max_value": 100})
    stories: int = Field(in_range={"min_value": 0, "max_value": 100})
    guestroom: str = Field(isin=["yes", "no"])
    basement: str = Field(isin=["yes", "no"])
    furnishingstatus: str = Field(isin=["furnished", "unfurnished", "semi-furnished"])
    parking: int = Field(in_range={"min_value": 0, "max_value": 100})

class HousingProcessedSchema(HousingSchema):
    price_per_area: float

@check_types
def calc_price_per_area(
    df: pa_typing.DataFrame[HousingSchema]
) -> pa_typing.DataFrame[HousingProcessedSchema]:
    df[HousingProcessedSchema] = df[HousingSchema.price].div(
        df[HousingSchema.area]
    )
    return pa_typing.DataFrame[HousingProcessedSchema](df)

Three things are worth noting:

• The guestroom, basement and furnishingstatus fields have been changed back to str and constrained by the isin check. This is because the synthesis capability does not (yet) work with the Pandera Category type.
• The other columns have been further constrained by using the in_range check.
• A check_types decorator has been added which validates both the input and output at runtime.

The extra constraints limit the space in which Hypothesis will try to synthesize data thereby greatly enhancing the efficiency of the synthesis process. Each Pandera built-in check, like isin or ge, has an associated synthesis strategy to generate data. If you have multiple checks, the data synthesis strategies get chained, which improves expressiveness, but can also significantly increase the synthesis time.

For example, if you would combine gt=0 and lt=100 in this order, Hypothesis will first generate any number that is greater than 0 and then check if it fulfills the second check. As the possible range of numbers above 0 is basically endless you can imagine this can take a while until it has come to a set of numbers that meet both conditions. Therefore, for this example, it’s much more efficient to use the in_range check instead. Long story short, one has to think carefully about their arrangement of checks before letting Hypothesis synthesize data for you.

To put the property based testing in practice, we can run below test function. The hypothesis.given decorator enables the generation of example dataframes using a specified strategy, here a Pandera schema, with a specified number of rows. Each sample dataframe will be fed to the calc_price_per_area function after which the output dataframe is tested by asserting that it contains the expected price_per_area column. By default hypothesis will generate 100 samples.

import hypothesis

@hypothesis.given(HousingSchema.strategy(size=10))
def test_calc_price_per_area(dataframe: pd.DataFrame):
    df_output = calc_price_per_area(dataframe)
    assert "price_per_area" in df_output

When running above test with for example pytest, immediately a Pandera SchemaError will be thrown. This is because the first dataframe it will generate is filled with single unique values taken from one end of the given constraints, which results in the area column being filled with zeros. Computing the price per area than results in NaN, while null values are not permitted by the price field in the HousingProcessedSchema.

    price  area  bedrooms  stories guestroom basement  parking furnishingstatus
0    0.0     0         0        0       yes      yes        0        furnished
1    0.0     0         0        0       yes      yes        0        furnished
2    0.0     0         0        0       yes      yes        0        furnished

This is exactly the kind of failure we want to see: it uncovers a blind spot in the schema. Tightening the constraint to require "min_value": 1 ensures that zero-area values are rejected, causing invalid data to fail loudly rather than propagate silently through the pipeline.

Conclusion

What the Zillow story ultimately illustrates is not a failure of AI, but a failure of discipline around data. Pandera offers a pragmatic way to bring that discipline into everyday data science workflows. By turning assumptions about data into explicit, executable schemas, it shifts validation from an afterthought to a first-class concern.

Throughout this post, we’ve seen how Pandera enables more than just basic type checking. It allows you to express domain constraints, enforce categorical values, apply custom logic, transform data safely through parsing, and even stress-test pipelines using property-based testing. Together, these features create a powerful feedback loop: expectations are clearly documented, violations are caught early, and hidden edge cases are surfaced before they can cause real-world damage.

Perhaps most importantly, Pandera encourages a mindset change. Instead of trusting that data is “probably fine,” schemas force you to confront what must be true for your code and models to behave correctly. Combined with type annotations and testing tools like Hypothesis, this results in pipelines that are not only more robust, but also easier to understand, maintain, and extend.