Building Self-Adaptive Data Pipelines That Do Not Break

A discussion based on research paper “Evolving Gracefully” by Kramer, Restat and Storl

VLDB 2025 Workshop QDB25

Signifigant paper

This blog discusses the research paper “Evolving Gracefully” by Kramer, Restat and Storl, presented at the VLDB 2025 Workshop QDB25. If you have ever woken up at 2 AM because a pipeline crashed and nobody warned you that someone upstream renamed a column, this paper is written for exactly that situation.

Why Data Pipelines Break in the Real World?

Most data pipelines do not fail because of logic errors. They fail because the data changes. In real systems upstream teams modify data structures without informing downstream consumers. A simple change like renaming a column or adding a field can break an entire pipeline overnight. The team that built the cleaning layer usually finds out not from a notification but from a crash report or an angry colleague.

This leads to three recurring problems. The pipeline crashes and stops processing. Outputs that do make it through may be silently wrong. And the team responsible has to drop everything and fix it before anyone downstream can continue their work.

This problem is not an edge case. It is extremely common in long-running data systems where no formal communication channel exists between the team writing data and the team cleaning it.

Figure 1: How a schema change cascades into downstream failure across a typical data pipeline

What is a Schema Evolution?

A schema defines the structure of data. It describes what fields exist, what they are called, and what type of values they hold. Schema evolution refers to changes in this structure over time. The most common types of schema changes are adding a new field, deleting an existing field, and renaming a field. Each of these is simple on its own. But even small changes can break tightly coupled pipelines that were written assuming the schema would stay fixed.

The challenge is not just that changes happen. It is that they happen without warning, without documentation, and without any mechanism for the downstream team to prepare.

The three most common schema change types are:

• Adding a new field that downstream operators were not built to handle
• Deleting an existing field that operators depend on for cleaning
• Renaming a field that operators reference by name in their logic

The Hidden Cost of Manual Data Cleaning

Data cleaning is not a small step in the pipeline. It is the majority of the work. Close to sixty percent of a data scientist’s time goes toward cleaning and preparing data rather than on analysis or modeling. When schema changes are handled manually that number gets worse. Every time a field is renamed or a column disappears someone has to diagnose the failure, understand what changed, update the operator or the configuration, test the fix, and restart the pipeline.

The paper frames this as a structural problem and not a discipline problem. Teams are not careless. They simply lack the tools to handle change gracefully.

What is a Data cleaning pipeline looks like ?

A data cleaning pipeline processes raw data into usable form using a sequence of operators. Each operator performs a specific task. One operator might fill in missing values. Another might remove duplicate records. A third might detect and correct outliers.

These operators are chained together so that data flows from one step to the next. The pipeline is created once based on the initial schema of the data. That is where the fragility begins. The pipeline is a snapshot of assumptions about the data at a particular moment. When the data changes those assumptions can become quietly wrong.

Classification of Cleaning Operators

To reason about which pipelines are most vulnerable to schema changes the paper introduces a classification of data cleaning operators based on how they interact with the schema. This classification is the foundation for everything that follows.

Figure 2: The four operator types and their relative sensitivity to schema changes

This classification matters because each type responds differently to schema changes. An operator that only knows about one column behaves very differently from one that relies on the relationship between multiple columns.

What does Robustness really mean?

This is where the paper makes its most important conceptual contribution. A pipeline is not robust just because it keeps running.

The authors define robustness more carefully than that. A data cleaning operator is robust if it continues to function after a schema change and it still produces the highest possible quality results. Both conditions must hold. An operator that survives a schema change but produces worse data than it did before is not truly robust. It is just failing quietly.

This definition connects two things that are usually treated separately. Functional correctness is about whether the system runs without crashing. Data quality is about whether the output is actually good. The paper argues that robustness requires both at once, and that designing for one without the other is not enough.

Why Context makes such a difference?

Data is not isolated. Values in one column often depend on values in another. Pollution readings differ by location. Temperature readings vary by time of year. A good cleaning operator knows this and uses it.

Without context a missing pollution value gets replaced with the overall average across all readings. With context the operator groups readings by location first and uses the average for that specific location. The result is meaningfully more accurate.

The paper calls this mechanism contextualization. It is the use of additional context properties in the cleaning process to achieve higher data quality. And it is particularly important because it is the place where schema changes have the most subtle impact. If a context property is deleted the operator may keep running but produce worse results without any visible error.

The Decision Tree for schema changes

The paper introduces a decision tree that takes any combination of operator type and schema change and produces a clear outcome. Either the operator is robust and nothing needs to change or it needs to be adapted. The logic works through a series of questions about what changed and how the operator depends on the affected parts of the schema.

Figure 3: Decision tree for determining whether and how to adapt an operator after a schema change

The tree turns what was previously a judgment call into a systematic decision. It does not remove the need for engineering work but it makes clear exactly when that work is necessary and what kind of change is required.

Workflow for handling schema evolution

Rather than reacting to failures as they happen the paper proposes a structured workflow for handling schema changes before they cause problems. The workflow has three steps.

First detect which schema changes occurred between the last batch and the current one. Second pair each change with every operator in the pipeline. Third run each pair through the decision tree to determine what action to take.

This approach transforms uncertainty into a manageable checklist. Instead of discovering a problem at runtime the system can reason about schema changes in advance and respond appropriately.

Moving toward self-adaptive pipelines

The ultimate goal the paper works toward is automation. Instead of humans fixing pipelines every time the data structure changes the system should be able to detect what changed, analyze the impact, and adapt on its own.

This vision is grounded in the well-established MAPE-K framework from self-adaptive systems research. MAPE-K stands for monitor, analyze, plan, and execute with a shared knowledge base supporting all four steps.

Figure 4: The MAPE-K loop applied to self-adaptive data cleaning pipelines

The key insight is that self-adaptation is not an all-or-nothing proposition. There is a spectrum of automation and different parts of a pipeline can sit at different points on that spectrum depending on how much certainty the system has about what a change means and what the right response is.

Techniques That Exist Today

The paper organizes adaptation techniques from simple and broadly applicable to complex and genuinely dynamic.

• Fallback logic provides a safe default when a dependency disappears. If the context property used for grouping is deleted the operator falls back to a simpler imputation method rather than crashing.
• Schema abstraction avoids hardcoding property names into operator source code. Names are injected at runtime from a configuration layer. When a property is renamed only the configuration needs to change.
• Meta programming through decorators or metaclasses allows operators to validate their own assumptions about the schema at runtime and react when those assumptions are violated.
• Schema and pipeline versioning stores a history of schema states and their associated pipeline configurations. When a batch arrives with a schema that has been seen before the system can restore the known-good pipeline for that schema rather than adapting from scratch.
• Contextualization of new properties is where the system becomes genuinely dynamic. When a new column appears the system evaluates whether it could serve as useful context for any existing operator and adapts accordingly.
• Automatic code generation sits at the most ambitious end. Using semantic patching or model-driven development the system generates new code in response to schema changes. This is the furthest from production-ready but it points in the right direction.

How This Work Can Be Extended

The paper defines the problem well and gives a solid framework for thinking about it. But several directions remain open and worth pursuing.

• LLMs for semantic schema mapping. When a column is renamed from user_id to customer_id a large language model could infer the mapping based on naming patterns and sample values rather than requiring a human to make that connection manually. This could automate the rename case almost entirely.
• Automatic semantic matching of renamed columns. Building a lightweight embedding model trained on column names and value distributions could flag likely renames with high confidence and handle them without human intervention.
• Integration with streaming systems. The paper focuses on batch pipelines. Extending this framework to streaming environments like Apache Spark or Flink is a natural and important next step given how much real-time data processing happens in production today.
• Feedback loops using data quality metrics. If the system tracks quality scores before and after adaptation it can learn over time which adaptation strategies work best for which types of changes. That feedback loop does not currently exist in the framework.
• Real-time schema drift alerts. Even before full automation is possible a monitoring layer that detects schema drift and notifies downstream teams immediately would reduce the damage significantly. The current state of most systems is that teams find out through failures.

Tradeoff between simplicity and quality

There is always a balance in how a pipeline is designed. A simple and generic pipeline will survive most schema changes because it makes few assumptions. But it will never produce the best possible output because it ignores the structure and context of the data.

A specialized pipeline that uses all available context and encodes deep knowledge about the data will produce excellent results under stable conditions. But it will break more often because every assumption it makes is a potential point of failure.

The paper is explicit that this tradeoff is real and unavoidable. The goal is not to eliminate it but to manage it deliberately. Robustness properly defined is about maintaining quality as conditions change rather than sacrificing quality to survive change.

Key Takeaways

• Schema evolution is unavoidable in any long-running data system. Designing as though the schema will stay fixed is not a plan. It is a debt.
• Pipelines must be designed for change from the start. This means thinking about how each operator interacts with the schema when it is first built.
• Context-aware cleaning improves quality significantly. The difference between a naive imputation and a context-grouped one is the difference between average data and accurate data.
• Decision-based adaptation is more reliable than intuition. The decision tree gives a systematic way to know when adaptation is required and what kind.
• Automation is the right long-term direction. The techniques exist today at the lower end of the spectrum. The harder dynamic approaches remain open problems worth solving.

Final Thoughts

Data pipelines should not break every time the data changes. They should evolve with it.

The paper does not claim to have solved this problem. It claims to have defined it precisely, which is the necessary first step. Before you can build a system that handles schema evolution gracefully you need to know what graceful actually means. Kramer, Restat and Storl give you that vocabulary: a classification of operators, a decision framework, and a spectrum of adaptation approaches to work from.

The future is not just pipelines that keep running. It is pipelines that adapt on their own, maintain data quality without constant human intervention, and treat schema evolution as a routine operating condition rather than an emergency. This paper is a reasonable and well-grounded starting point for getting there.

Based on Evolving Gracefully | VLDB 2025 Workshop QDB25 | Kramer, Restat and Storl