Getting Started with Databricks Spark Declarative Pipelines: Building a Simple Volume-to-Delta Ingestion Pipeline

Understanding Databricks Spark Declarative Pipelines Through a Simple Ingestion Pipeline

For years, data engineers have built ETL pipelines by explicitly defining every step of data processing. We read data, transform it, handle checkpoints, manage dependencies, write outputs, monitor failures, and maintain orchestration logic. While this approach provides flexibility, it also introduces operational complexity as the number of pipelines grows.

Databricks Spark Declarative Pipelines (SDP) take a different approach.

Instead of telling Spark how to execute a pipeline, engineers define what the desired outcome should be. The Databricks runtime then determines the optimal execution plan, dependency management, and orchestration behind the scenes.

In this article, we’ll explore the fundamentals of Spark Declarative Pipelines and build a simple ingestion pipeline that reads files from Databricks Volumes and writes them into Delta tables in append mode.

What Are Spark Declarative Pipelines?

Spark Declarative Pipelines are a framework introduced by Databricks that allows data engineers to define datasets, transformations, and dependencies using a declarative programming model.

Traditional Spark development typically looks like this:

df = spark.read.json("/landing/orders")

transformed_df = (
    df
    .filter("status = 'ACTIVE'")
    .withColumn("load_date", current_timestamp())
)

transformed_df.write \
    .format("delta") \
    .mode("append") \
    .saveAsTable("silver.orders")

As pipelines become larger, engineers must manually handle:

• Job orchestration
• Table dependencies
• Incremental processing
• Monitoring
• Recovery logic
• Schema evolution
• Data quality validations

Spark Declarative Pipelines reduce this operational burden by allowing engineers to describe datasets as reusable pipeline components.

The framework then manages execution planning and dependency resolution automatically.

Declarative vs Imperative Thinking

One of the biggest mindset shifts is understanding the difference between imperative and declarative programming.

Imperative Approach

The engineer specifies every step:

Read File
↓
Transform Data
↓
Write Table
↓
Manage Checkpoint
↓
Schedule Job
↓
Monitor Execution

Declarative Approach

The engineer specifies the desired state:

This dataset comes from Volume X
This table should contain these records
Apply these transformations

Databricks handles the execution strategy.

This allows engineers to focus more on business logic and less on infrastructure management.

Simple Use Case

Let’s assume JSON files are arriving in a Databricks Volume:

/Volumes/raw/orders/

Our objective is simple:

• Read new JSON files
• Process incrementally
• Write records into a Delta table
• Append only

This is a very common Bronze layer ingestion pattern.

Reading Data from a Volume

In Spark Declarative Pipelines, we can define a dataset as follows:

from pyspark import pipelines as dp

@dp.table(
    name="bronze_orders"
)
def bronze_orders():

    return (
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/Volumes/raw/orders/")
    )

At first glance, this looks similar to Structured Streaming.

However, there is an important difference.

The function does not execute immediately.

Instead, it declares a dataset that becomes part of the pipeline graph.

Databricks analyzes these declarations and builds the execution plan automatically.

What Happens Behind the Scenes?

When the pipeline runs:

1. Databricks discovers new files in the Volume.
2. Auto Loader processes only new files.
3. Schema information is maintained automatically.
4. Checkpoints are managed by the pipeline.
5. Records are written to the target Delta table.

As engineers, we don’t need to manually create streaming queries or manage long-running jobs.

Writing to a Delta Table in Append Mode

The previous example automatically creates and maintains a Delta table.

Conceptually, it behaves similarly to:

df.writeStream \
  .format("delta") \
  .outputMode("append") \
  .toTable("bronze_orders")

The key difference is that the pipeline framework manages this behavior declaratively.

The engineer defines the table.

The platform manages the execution.

Adding Simple Transformations

Suppose we want to enrich the data before storing it.

from pyspark.sql.functions import current_timestamp

@dp.table(
    name="bronze_orders"
)
def bronze_orders():

    return (
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .load("/Volumes/raw/orders/")
        .withColumn("ingestion_time", current_timestamp())
    )

The transformation becomes part of the dataset definition.

No separate orchestration logic is required.

Automatic Dependency Management

Now suppose we create a Silver table.

@dp.table(
    name="silver_orders"
)
def silver_orders():

    return (
        spark.read.table("bronze_orders")
        .filter("status = 'ACTIVE'")
    )

Notice something interesting.

There is no explicit dependency configuration.

Databricks automatically understands:

Volume Files
     ↓
bronze_orders
     ↓
silver_orders

When Bronze updates, Silver updates automatically according to the pipeline definition.

This significantly simplifies pipeline maintenance.

Why This Matters for Data Engineering Teams

As organizations scale, pipelines often become difficult to maintain because engineers spend substantial time managing operational concerns.

Common challenges include:

• Dependency tracking
• Incremental load handling
• Monitoring failures
• Schema evolution
• Checkpoint management
• Job orchestration

Spark Declarative Pipelines address these challenges by moving infrastructure concerns into the platform.

The result is:

• Less boilerplate code
• Simpler pipeline definitions
• Easier maintenance
• Better scalability
• Faster development cycles

A Practical Way to Think About It

When I explain Spark Declarative Pipelines to engineers coming from traditional Spark development, I use this analogy:

Traditional Spark

“I will tell Spark every step required to move the data.”

Spark Declarative Pipelines

“I will tell Databricks what the final dataset should look like, and Databricks will figure out how to build and maintain it.”

This shift may seem small, but it fundamentally changes how data pipelines are designed and operated.

Final Thoughts

Spark Declarative Pipelines represent a move toward higher-level data engineering abstractions. Rather than spending time managing execution mechanics, engineers can focus on defining datasets and business logic.

For simple ingestion use cases — such as reading files from Databricks Volumes and writing them to Delta tables in append mode — the benefits are immediately visible. The code becomes shorter, dependencies become easier to manage, and operational overhead is significantly reduced.

As data platforms continue to evolve, declarative pipeline frameworks are likely to become the standard approach for building scalable, maintainable data engineering solutions.