Solving Schema Evolution in Data Pipelines: Standards Every Data Engineer Should Follow

Most data pipelines don’t fail because of infrastructure. They fail because somebody upstream changed the data.

• New column appears.
• Datatype changes.
• Nested JSON structure evolves.
• Producer silently renames a field.

And suddenly:

• DBT models fail
• MERGE jobs break (In my case BigQuery)
• Dashboards show incorrect numbers
• Streaming pipelines start lagging

If you’ve worked on production-grade data platforms long enough, you realize something important:

Schema evolution is not a rare problem. It is a certainty.

The real challenge is not preventing schema changes. The challenge is designing pipelines that survive schema evolution gracefully.

In this article, I’ll explain:

• Why schema evolution breaks pipelines
• Common mistakes data engineers make
• Engineering standards that actually work
• Real-world examples using modern data platforms
• How to design robust and scalable pipelines

What is Schema Evolution?

In simple terms, Schema Evolution is the ability of a data system to change its structure (adding, renaming, or modifying columns) over time without breaking the entire pipeline or losing historical data.

A Real-World Production Example: The “E-Commerce Discount” Disaster

Imagine you work for a major e-commerce platform. Your production database has a Sales table that tracks every purchase.

The Starting Point (Version 1)

Your schema is simple and strict:

order_id : INT64
price : FLOAT64
discount_code : STRING

The Change (The Evolution)

The Marketing team launches a new “Loyalty Program.” Instead of one discount code, they now allow users to stack multiple discounts.

The Upstream Change: The backend developers change the discount_code field from a single String to an Array of Strings (e.g., ['SUMMER24', 'WELCOME10']).

The “Production Crisis”

If you don’t have a schema evolution strategy, here is what happens:

• In Batch Pipelines: Your nightly load job from the app database to BigQuery hits that first array. BigQuery says: “Wait, I expected a String, but you gave me a List.” The job fails. The dashboard for the stakeholders is empty the next morning.
• In Real-Time Pipelines: Your streaming connector (like Datastream) keeps trying to push the array into the string column. It gets rejected. The “backlog” of messages grows, the system slows down, and eventually, the whole pipeline crashes.

Common Mistakes Data Engineers Make

Mistake 1 — Tight Coupling Between Source and Curated Tables

Why this fails:

• Any schema change directly impacts analytics
• No protection layer exists
• Downstream systems break immediately

Mistake 2 — Assuming Datatypes Never Change

Real-world example:

Original:

"discount_code": "WELCOME-10"

Later:

"discount_code": ["WELCOME-10", "NEW_YEAR-10"]

This breaks:

• JOIN logic
• MERGE operations
• Incremental pipelines

Mistake 3— Using SELECT *

This is one of the most dangerous patterns.

Example:

SELECT *
FROM raw_orders

Problems:

• Unexpected columns propagate downstream
• Renamed fields silently break transformations
• Analytics becomes unpredictable

Mistake 4: No Schema Ownership Standards

Many teams:

• Change APIs freely
• Don’t communicate schema updates
• Don’t maintain contracts

Result: downstream chaos.

Standards That Actually Make Pipelines Robust

Now let’s discuss what works in real production systems.

Standard 1: Separate Raw and Curated Layers

This is foundational. The raw layer should be schema flexible. Store data as is and preserve original payload. One can relate this with the Medallion Architecture (Bronze/Silver/Gold)

Standard 2: Auto-Evolution (I use this most often in BigQuery)

When using BigQuery for batch or streaming ingestion, you can programmatically allow the schema to evolve.

bq load \
--source_format=CSV \
--autodetect \
--schema_update_option=ALLOW_FIELD_ADDITION \
--schema_update_option=ALLOW_FIELD_RELAXATION \
your_dataset.your_table \
gs://your-bucket/data.csv

Key Configurations Explained

• **ALLOW_FIELD_ADDITION**: This is what enables "Schema Evolution." If your source file has a column new_marketing_tag that doesn't exist in your BigQuery table, BigQuery will add it to the schema automatically instead of failing the job.
• **ALLOW_FIELD_RELAXATION**: Very useful for production. If your BigQuery table has a column marked as REQUIRED, but a new batch of data is missing that column, this option "relaxes" the constraint to NULLABLE so the pipeline doesn't break. Warning — schema relaxation is a one-way street. You can’t easily go back to REQUIRED without a table recreate.
• **autodetect=True**: Essential for discovery. It tells BigQuery to scan the source file and compare its structure to the existing table. Works well for evolving schemas, but for production pipelines, explicit schema management is usually safer.

Important Limitation

BigQuery cannot automatically change a data type of an existing column (e.g., changing STRING to INT64). If a source field has a conflicting type compared to the existing table, the job will still fail. Pro-Tip — This is why using the JSON Data Type for volatile fields remains the "Best Practice" for high-frequency evolution.

Standard 3: Use Backward-Compatible Changes Whenever Possible

In production environments, Backward Compatibility means your schema changes should never break existing “consumers” — whether those are dbt models, Looker dashboards, or Python scripts.

Good schema changes: ✅ Add nullable fields ✅ Add optional attributes

Dangerous changes: ❌ Rename columns ❌ Remove fields ❌ Change datatypes

Standard 4: Quarantine Invalid Records

Do not fail entire pipelines because of few bad records. Instead isolate problematic events. Use DLQ if using Dataflow or separate out the bad records into different table in DBT before applying the transformation rules.

Standard 5: Add Data Validation Layer

Your stakeholders don’t care about your pipeline; they care about their Looker or Tableau dashboards. Add a validation layer and data contract before making the data available for end users.

Validation layer should check:

• Required fields
• Datatypes
• Value ranges
• Null spikes

Standard 6: Schema Contracts Between Producers and Consumers

One of the biggest operational improvements — explicit contracts. Instead of ‘Teams will coordinate manually’

Define:

• Required fields
• Allowed datatypes
• Backward compatibility rule

Example contract. This creates accountability.

{
  "event_name": "order_created",
  "schema_version": 2,
  "required_fields": [
    "order_id",
    "customer_id",
    "order_create_datetime"
  ]
}

Use tools like **Confluent Schema Registry or [AWS Glue Schema Registry](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html) **to enforce these json contacts.

Realistic Production Scenario

Imagine an e-commerce platform add new field to the existing data stream. This should NOT: ❌ break ingestion ❌ fail analytics ❌ crash streaming jobs

A robust pipeline should: ✅ ingest successfully ✅ preserve raw payload ✅ validate schema ✅ update curated models safely

Robust Pipelines Actually Optimize For Not perfection. Robust pipelines optimize for:

• Recoverability
• Flexibility
• Observability
• Controlled evolution

Modern data engineering is less about preventing change and more about surviving change safely.