Governed Schema Evolution on Databricks

DevOps-powered Version Control for Unity Catalog with Liquibase

Author: Manuel Beuttler, Delivery Solutions Architect @ Databricks

Introduction: Schema Drift and Version Control Challenges

In modern data platforms like Databricks, managing schema changes can be a significant headache, often leading to schema drift and broken downstream pipelines. This challenge is a core problem faced by many data teams that seek a robust solution for version control and Data Product lifecycle management.

Without a disciplined approach, changes made manually in the UI or via ad hoc scripts create a “wild west” environment. A simple column rename in a development environment may go undocumented, only to cause a critical pipeline failure when the code is moved to production.

The solution lies in establishing a DevOps pipeline where every schema modification, from a simple column comment update to a major table alteration, is version-controlled, automated, and subject to a formal review and approval process.

While our previous articles (Database Change Management and Advanced Schema Management) focused on what Liquibase can do and how to execute commands, in this one, we will walk through an example architecture that leverages Liquibase within an Azure DevOps CI/CD pipeline to govern schema changes in a Databricks Unity Catalog (UC) environment and build production-ready data products.

About Liquibase

Liquibase is an open source database change management tool that helps development teams track, version, and deploy database schema changes in a controlled and automated way. It works by storing all database modifications (called “changesets”) in text-based files that are version-controlled alongside application code. Liquibase automatically translates these platform-independent change definitions (in SQL, XML, YAML, or JSON) into database-specific SQL. It supports various database systems as targets and offers additional features and capabilities, including integration with CI/CD pipelines for automated deployments across different stages, rollback support, drift detection to identify unauthorized database changes, and comprehensive tracking tables that ensure each change is applied only once.

Solution Architecture: The key components

Architecture sketch

The goal is to build an architecture that establishes a bridge between code repositories and the data warehouse, treating database schemas with the same rigor as application code. Our solution relies on four key components:

1. Source Control Repository (e.g., Git): Stores the “source of truth” for the database, specifically the Liquibase change log files that define the schema state.
2. CI/CD Pipeline (e.g., Azure DevOps): Our orchestration component and approval gateway. Including the binaries (Liquibase Docker Container) and execution environment that runs the schema migration logic, ensuring consistency across deployments.
3. Databricks Unity Catalog (UC): The central governance layer that serves as the target for all schema changes, managing tables, permissions, and metadata.
4. Databricks SQL Warehouse: Acts as the compute engine and connection endpoint, allowing Liquibase to execute JDBC commands against the Databricks environment.

Next, let’s look deeper into the individual components and how to tie them together.

Repository Structure and Organization

Our starting point is a well-organized repository as a foundation for maintainable schema management. The example structure below separates concerns: binaries for dependencies, templates for reusability, pipeline definitions for orchestration, and data products for actual schema changes.

Example Folder Structure

repository/
├── bin/
│   ├── DatabricksJDBC42.jar
│   └── liquibase-databricks-1.4.2.jar
│
├── templates/
│   └── liquibase-template.yml
│
├── pipelines/
│   └── azure-pipelines.yml
│
└── data_products/
    ├── changelogs/
    │   ├── fleet/
    │   │   ├── 20241215-01-create-fleet-table.sql
    │   │   ├── 20241220-01-add-mileage-column.sql
    │   │   └── 20241230-01-add-vehicle-type-comment.sql
    │   │
    │   ├── trips/
    │   │   ├── 20241215-01-create-trips-table.sql
    │   │   └── 20241222-01-add-driver-id-column.sql
    │
    └── root_changelog.xml

Folder Breakdown

bin/: Contains the necessary JDBC drivers and Liquibase extensions.

• DatabricksJDBC42.jar: The official Databricks JDBC driver that enables Liquibase to connect to the SQL Warehouse.
• liquibase-databricks-1.4.2.jar: The Liquibase extension specifically designed for Databricks, which handles Unity Catalog-specific SQL syntax.

templates/: Contains reusable YAML templates that abstract the Liquibase Docker execution logic. This keeps the main pipeline file clean and promotes consistency across multiple pipelines if you manage multiple catalogs or schemas.

pipelines/: Contains the Azure DevOps pipeline definition(s). This is where the orchestration logic lives, defining triggers, stages, variables, and approval gates.

data_products/: The heart of the repository. Each subfolder represents a logical data product (typically corresponding to a table or a related group of tables). Within each product folder we have:

• root_changelog.xml: The master changelog file that Liquibase reads. It references all individual SQL files in the changelogs/ folder, maintaining the correct execution order.
• changelogs/: By using this extra level in the folder hierarchy, we only have to reference it once in the root_changelog.xml and every new subfolder (f.e. fleet or trip) will be automatically added. If we don’t want to have this redundant level, we will have to reference each of them individually.
• [fleet|trip|{table}]/: Contains individual SQL files with the actual schema definitions, so called changelogs (we will cover them in detail later). Each file is named with a timestamp and sequence number for easy ordering (e.g., YYYYMMDD-NN-description.sql).

This structure easily scales: as new data products are added, simply create a new folder under changelogs/ and start committing SQL files.

CI/CD Pipeline Definition

For better readability and maintainability, we split the pipeline definition into two files, encapsulating the logic for running the Liquibase Docker container and other reusable parts into a YAML template.

The template mounts the necessary volumes (change logs and drivers), uses the master changelog as the entry point and constructs the JDBC connection string dynamically:

parameters:
 - name: action
   type: string
 - name: displayName
   type: string

steps:
 - script: |
     # Use latest docker liquibase image
     docker run \
       -v $(Build.SourcesDirectory)/data_products:/liquibase/changelog \
       -v $(Build.SourcesDirectory)/bin:/liquibase/lib \
       liquibase \
       --classpath="/liquibase/changelog:/liquibase/lib/DatabricksJDBC42.jar:/liquibase/lib/liquibase-databricks-1.4.2.jar" \
       --url="$(DATABRICKS_SQL_ENDPOINT)ConnCatalog=$(CATALOG);ConnSchema=$(SCHEMA);UID=token;PWD=$(PAT);" \
       --changeLogFile=root_changelog.xml \
       ${{ parameters.action }}
   displayName: ${{ parameters.displayName }}

After defining the template, we can reference and use it in our main pipeline definition: the azure-pipelines.yml file, which orchestrates the entire workflow. It is triggered on commits to the main branch and executes a two-stage process: validation followed by deployment with an approval gate.

trigger:
  - main

stages:
  - stage: validation
    displayName: 'Validate Schema Changes'
    dependsOn: []
    jobs:
      - job: reviewChanges
        displayName: 'Review Changes'
        pool:
          name: ${AGENT_POOL}

        variables:
          - name: CATALOG
            value: ${TARGET_CATALOG}
          - name: SCHEMA
            value: ${TARGET_SCHEMA}
          - name: SQL_ENDPOINT
            value: ${DATABRICKS_SQL_ENDPOINT_DEV}
          - name: DATABRICKS_CLIENT_ID
            value: $(DATABRICKS_CLIENT_ID_DEV)
          - name: DATABRICKS_CLIENT_SECRET
            value: $(DATABRICKS_CLIENT_SECRET_DEV)

        steps:
          - checkout: self

          - template: templates/template_liquibase.yml
            parameters:
              action: validate
              displayName: 'Validating changelog files'

          - template: templates/template_liquibase.yml
            parameters:
              action: status
              displayName: 'Show status on current and pending changelogs'

          - template: templates/template_liquibase.yml
            parameters:
              action: updateSQL
              displayName: 'Show SQL before it will be executed'

  - stage: deployment
    displayName: 'Deploy Schema Changes'
    dependsOn: [validation]
    jobs:
      - deployment: deployProd
        displayName: 'Deploy to Production'
        pool:
          name: ${AGENT_POOL}

        variables:
          - name: CATALOG
            value: ${TARGET_CATALOG}
          - name: SCHEMA
            value: ${TARGET_SCHEMA}
          - name: SQL_ENDPOINT
            value: ${DATABRICKS_SQL_ENDPOINT_DEV}
          - name: DATABRICKS_CLIENT_ID
            value: $(DATABRICKS_CLIENT_ID_DEV)
          - name: DATABRICKS_CLIENT_SECRET
            value: $(DATABRICKS_CLIENT_SECRET_DEV)

        environment: demo-prod-env

        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self

                - template: templates/template_liquibase.yml
                  parameters:
                    action: update
                    displayName: 'Executing changelog files'

Trigger: For our example, the pipeline automatically runs whenever changes are committed to the main branch, ensuring that every schema modification undergoes validation before deployment. For real-world production scenarios, we will likely want to implement more advanced trigger logic, depending on the branch used for staging and releases.

Variables: Each stage defines environment-specific variables for the target catalog, schema, and Databricks connection details. Using template variables (e.g., ${TARGET_CATALOG}) allows the same pipeline to be reused for different environments by simply overriding values at pipeline execution time.

Stage 1: The Validation Stage

The first stage of the pipeline is read-only and designed purely for validation/review. It executes several Liquibase commands to ensure the proposed changes are safe and correct, without modifying the target database.

1. validate: Scans the change logs for syntax errors, missing files, or malformed XML/SQL structures.
2. status: Reports which changesets have not yet been applied to the target database.
3. updateSQL: This is the most critical step for reviewers. It generates and prints the exact SQL statements that Liquibase intends to execute.

By inspecting the output of updateSQL, the data owners can verify exactly what will happen (e.g., “Ah, this script drops a column I didn’t expect”). This is vital for preventing accidental data loss.

Screenshot of job run output

The screenshot above shows an example of the updateSQL stage’s output. It is intended for users who are familiar with SQL. It not only shows which statements are executed by the changelog file, but also lists database locks and some additional information about the changeset.

Screenshot of deployment stage waiting for approval

If the validation state succeeds without errors, the deployment state will be triggered next. However, to enforce a manual check before deployment, the pipeline pauses. This is where the Approval Gate comes into play.

Screenshot of an example environment setup

Stage 2: The Deployment Stage

In Azure DevOps, this is implemented using Environments. A specific environment (e.g., demo-prod-env) is configured to require approval from a designated Data Steward or Data Owner. The pipeline will not proceed to the Deployment stage until this person reviews the Validation output and explicitly clicks “Approve.”

Screenshot of the approval dialog

Deployment Stage:

• dependsOn: [validation]: Only runs after Validation succeeds.
• environment: demo-prod-env: Configures the Approval Gate
• update: Executes the pending changesets, applying the schema changes to the target catalog and schema.

Once approved, the Deployment Stage triggers. It runs the update action, which:

1. Connects to the Databricks SQL Warehouse.
2. Acquires a lock on the database to prevent concurrent updates.
3. Executes the pending changesets.
4. Updates the tracking table to indicate that these changes are complete.

Screenshot of the job output after deployment

With this separation of concerns — validation without side effects, followed by human review and deployment — we ensure that only reviewed and approved changes reach production.

Change Logs: The core of our schema definitions

In this example, we define schema changes as a series of “changesets.” While Liquibase supports XML, JSON, and YAML, using Formatted SQL is the suggested best practice for data teams. We assume that data engineers are familiar with standard SQL, which lowers the entry barrier while still benefiting from Liquibase’s tracking capabilities. Liquibase, in general, supports various systems (not only Databricks), which allows us to use the same changesets for deployments on PostgreSQL (the core of Databricks Lakebase) or other databases.

Our recommended best practice is to use one changeset file for each version or logical change. This modular approach makes code reviews easier and simplifies the tracking of history.

The examples below illustrate this approach, showing two types of changelog files:

1. Master Changelog (XML): This file references all individual changeset files located under a specified path. We defined a filter to only consider for SQL files.
2. Individual SQL File: This is the actual SQL code being tracked and executed by the Liquibase command.

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
   xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
       http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">

   <includeAll path="changelogs" relativeToChangelogFile="true" endsWithFilter=".sql"/>
</databaseChangeLog>

- liquibase formatted sql
 - changeset Max.Mustermann:20251230–001
 - comment: Updating comments for vehicle_type to include examples.
ALTER TABLE fleet ALTER COLUMN vehicle_type COMMENT 'The type of the vehicle for example TRUCK or CAR';

Treating schema changes as modular, version-controlled artifacts brings the same discipline to data infrastructure that software engineering has long relied on. With formatted SQL and Liquibase, we can confidently evolve databases while preserving consistency and accountability across deployments.

Advanced Usage: Ideas for using each component to its full potential

Implementing this architecture requires us to consider certain nuances regarding how Liquibase interacts with Databricks Unity Catalog.

Tracking State with Governance

Liquibase maintains its own internal state using a DATABASECHANGELOG table created within the target schema. This table acts as the ledger of history, tracking:

• ID: The unique identifier of the changeset.
• Author: Who created the change.
• Filename: Where the change was defined.
• Checksum: A hash of the file content to detect if a deployed script has been retroactively modified (which throws an error to prevent inconsistencies).

Because this table resides within Unity Catalog, it is governed in the same manner as any other table in Unity Catalog. We can, and should, restrict access so that only the service principal running the CI/CD pipeline has write access to it, ensuring the audit trail remains tamper-proof.

Handling Existing Tables (Brownfield Projects)

In a real-world example, we would most likely not start from a blank slate; instead, we would have existing tables filled with valuable data. Luckily, bringing these under version control is straightforward.

When initializing the Liquibase project for an existing schema, we create a “baseline” changeset. Crucially, if the table already exists in the target environment, we can omit the CREATE TABLE statement or use Liquibase “preconditions” to check for existence. This allows Liquibase to “sync up” its tracking table without failing because the object already exists.

More information on diff-changelog can be found in the Liquibase documentation.

Managing Rollbacks

All quality gates, code reviews, or other safety measures will fail sometimes. So, what can we do if our changesets include errors and we need to revert them? Fortunately, Liquibase also offers advanced options to support you:

• Use the built-in rollback behavior for default operations, such as CREATE TABLE.
• Define custom undo logic directly in the change-log to make rollbacks more testable and predictable
• Tag important versions of our deployments to later roll back to a specific point in time

Before deploying such logic into production, it is essential to verify and test our rollback strategy. So we know that we are prepared when the time comes.

For more details and additional implementation examples, refer to the Liquibase documentation.

The Risk of Automated Change Generation

Liquibase offers a powerful feature called diff, which can compare two databases (e.g., Dev vs. Prod) and automatically generate a changelog to synchronize them. While tempting, this feature introduces significant risk, particularly regarding Column Renames.

The Issue: If a developer renames a column from customer_id to client_id in the development environment, Liquibase’s diff tool does not recognize this as a rename operation. Instead, it interprets the change as two separate actions:

1. DROP column customer_id
2. ADD column client_id

The Implication: If this auto-generated script is deployed to production, it will delete the column and all its data, then create a new, empty column.

Because of this risk, the best practice is to avoid automated changelog generation for production deployments. Instead, engineers should manually author SQL changesets. This ensures that a rename is explicitly written as ALTER TABLE … RENAME COLUMN, preserving the data.

Tool Selection: Why Liquibase and not Databricks Asset Bundles (DABs)

If you have already implemented Infrastructure as Code using Terraform or Databricks Asset Bundles, you might wonder why you should introduce a new technology and tool into your ecosystem, just for the sake of Schema Management.

While both tools manage Databricks resources, they serve fundamentally different purposes based on the lifecycle of the object and the consequences of replacing it.

1. The Case for Databricks Asset Bundles (DABs): Resource Definition DABs follow an “Infrastructure as Code” (IaC) philosophy. They are designed for resources where the definition is the master, and the actual object can be updated, restarted, or replaced to match that definition.

Best For:

• Spark™ Declarative Pipelines with Materialized Views.
• Lakeflow Jobs and Pipelines: Jobs, tasks, and pipeline schedules.
• Compute Infrastructure: Job Clusters, SQL Warehouses, and Instance Pools.

Why: These resources are declarative.

• If you update a Job schedule in a DAB, the platform simply overwrites the old schedule.
• If you change a Compute configuration, the cluster restarts with the new settings.
• If you modify logic in a Spark Declarative Pipeline, the system handles the update automatically, often by refreshing the flow or recomputing the data. The state is either ephemeral (compute) or fully derived from code (pipelines).

2. The Case for Liquibase: Persistent State Evolution Liquibase follows an “Evolutionary Database” philosophy. It is designed for resources where the data is the primary asset, and the structure must be carefully mutated around it.

Best For: Standard Managed Tables (Delta tables).

Why: These tables maintain a persistent state that cannot be simply recomputed or replaced. You cannot “re-deploy” a standard table containing 5 years of historical customer data just because you want to add a middle_name column; replacing the table would delete the history. Liquibase excels here because it executes granular, imperative modifications (ALTER TABLE…, UPDATE…) that evolve the schema version by version, while strictly preserving the underlying data.

The Rule of Thumb:

• If the resource describes how code runs (Jobs, Compute) or data derived from code (Declarative Pipelines) → Use DABs.
• If the resource describes stored data that must be preserved and evolved over time (Standard Tables) → Use Liquibase.

Conclusion

By combining the governance of Databricks Unity Catalog with Liquibase’s version control capabilities, we showed how data teams can bring software engineering rigor to their data platforms. Our example architecture replaces anxiety-inducing manual updates with a transparent, automated process where every change is tracked, reviewed, and reversible.

While it requires an initial investment in setup, creating the repository structure, configuring the Docker container, and defining the pipelines, the result is a stable, audit-ready data environment that scales with your team.

Liquibase offers a wide range of capabilities, allowing you to quickly define your production-ready Data Products. Some automations, however, require careful evaluation, and manual definitions and steps should be preferred.

Final remark

Tools like Liquibase for database change management, CI/CD platforms for automation, and unified governance layers such as Unity Catalog provide the technical foundation. However, sustainable success comes from pairing these tools with a culture of domain ownership, thorough documentation, rigorous testing, and accountability for data quality and accessibility. Without data owners and stewards living up to these expectations and managers supporting the organizational change to develop data products ready for production, it is hard to achieve, and the potential for shortening the path from idea to insight and from insight to action remains unutilized.

To successfully implement production-ready data products, you will need both people and technology.

About the author

Manuel Beuttler is a Delivery Solutions Architect at Databricks, helping customers across Central EMEA advance their data journey. Drawing on his experience in building data pipelines on Azure Databricks and establishing data-driven products and services in the sheet metal manufacturing industry, he brings a strong mix of hands-on engineering and practical governance expertise to every engagement.

Additional Resources

Documentation:

• Liquibase Official Documentation
• Databricks Unity Catalog Best Practices
• Azure DevOps Pipeline Templates
• Azure DevOps Environments and Approvals

Community Posts:

• Advanced Schema Management on Databricks with Liquibase
• Databricks Schema Versioning with Flyway and Liquibase: A Step-by-Step Guide