Delta Live Tables (DLT) Is Powerful. These Are Its Real Constraints.

What a Medallion pipeline on Databricks Free Edition revealed about declarative frameworks, production trade offs, and where architectural judgement still matters

Every declarative framework promises you’ll write less code. What the documentation doesn’t tell you is where the abstraction breaks — and what you’re left holding when it does.

I validated the Medallion architecture end-to-end on Databricks Free Edition — five data streams, full CDC handling, Auto Loader, Delta Live Tables, Unity Catalog. The pipeline works. But building it revealed something more useful than a working prototype: a precise picture of where DLT’s declarative model buys you velocity, and where it quietly hands control back to the engineer.

If you’re evaluating Databricks for a streaming data platform, or deciding whether DLT belongs in your production stack, this is the post I wish I’d read before I started.

The Architecture Problem Worth Understanding

Most organisations adopting a Lakehouse platform face the same structural challenge: raw operational data arrives from multiple heterogeneous sources — databases, event streams, mobile apps, IoT devices — and needs to be collected, cleansed, and served to analytical consumers without coupling any of those concerns together. Kafka breaks. Schema changes happen. CDC events arrive out of order. The pipeline needs to absorb all of it without propagating noise downstream.

The Medallion architecture is Databricks’ answer: Bronze for raw ingestion, Silver for cleansed and typed data, Gold for joined and aggregated analytical outputs. Decoupled layers, progressively refined quality. In theory, clean. In practice, the interesting question is what happens at the boundaries — and specifically, what happens when the transformation you need doesn’t fit the framework’s model.

That boundary question is what this project was designed to stress-test.

What I Validated — and What It Was Designed to Prove

The project: a wearable health data pipeline for a fictional device company. Five source streams — device registrations, user profile CDC events, continuous heart rate readings, gym login/logout events, and workout session start/stop events — ingested through Bronze, transformed in Silver, and aggregated into two Gold outputs: a per-session Workout BPM Summary and a per-gym-visit Gym Summary.

The architectural choice I was validating: decouple ingestion from transformation by using Databricks Workflows for the Bronze layer and Delta Live Tables for Silver and Gold. The two layers communicate only through Delta tables — no code dependency between them. Either layer can be re-run, modified, or extended independently.

What I wanted to know: does DLT’s declarative model hold up across all five transformations? Where does it deliver on its promise, and where does it require you to step outside it?

The Architecture — and the Decisions Behind It

End-to-end Medallion pipeline on Databricks. The annotated BATCH on workout_session_paired is the architectural decision this post is about.

Four design decisions shaped the whole system:

1. Workflows for Bronze, DLT for Silver/Gold. Ingestion and transformation are decoupled at the Delta table boundary. The Bronze Workflow can be re-run, re-sequenced, or replaced without touching the DLT pipeline. This is a production engineering principle, not a convenience — it means the two layers have independent failure modes and independent recovery paths.
2. Five sequential Bronze tasks, not parallel. Databricks Free Edition caps concurrent job tasks at five. Running all five ingestion tasks in parallel would have exhausted the limit and blocked the DLT trigger task. Sequential execution was a platform constraint, not a preference — and that distinction matters when you’re sizing a production deployment. On a paid tier with no concurrency ceiling, parallel ingestion is the right default.
3. Single DLT pipeline for Silver and Gold. DLT resolves intra-pipeline dependencies automatically. A Gold table reading from a Silver table will always execute after that Silver table is updated, within the same pipeline run. Splitting into two pipelines adds orchestration complexity with no benefit at this scale — but it also means Silver and Gold cannot run at different cadences in production without a pipeline split.
4. CDC via apply_changes — SCD Type 1 only. The User Profile stream sends new, update, and delete events. DLT’s apply_changes API handles ordering by sequence key, deduplication, and merge into a current-state Silver table automatically. What it doesn’t do: preserve history. Every update overwrites the previous record. This is sufficient for the Gold outputs this pipeline serves, but it permanently forecloses “what was the user’s profile at time T” queries without replaying the Bronze log. SCD Type 2 is available via the same API — with a more complex schema and different downstream join semantics.

Databricks Workflow for ingesting into Bronze layer

The DLT pipeline after a full run on Databricks Free Edition.

Where the Declarative Model Hands Back Control to the Engineer

Declarative frameworks buy you velocity in the common case and cost you control in the edge case. The question worth asking before you commit is: how common is your edge case?

The most instructive moment in this build wasn’t a failure. It was a constraint — one that forced a design decision with direct implications for production scale.

The Workout Session Bronze table contains two row types: action=’start’ and action=’stop’ , each tagged with a session_id . The Silver output needs one row per session — start timestamp, stop timestamp, computed duration. A join on session_id between the two filtered subsets.

DLT does not support this. You cannot filter a streaming DataFrame for starts, filter the same streaming DataFrame for stops, and join them within a single @dlt.table function. The framework prohibits joining two streaming DataFrames derived from the same source table.

The resolution: read Bronze as a batch — dlt.read() instead of dlt.read_stream() . The join works. The Silver table is produced correctly. But the table is now fully recomputed on every DLT pipeline run, not incrementally. At a hundred sessions, that’s negligible. At ten million sessions, that’s a full table scan on every trigger.

This is not a DLT bug. It’s a boundary condition of the declarative model. Declarative streaming frameworks are designed around the common transformation patterns: append, filter, aggregate, join across different sources. The moment you need to join a source to itself — correlating two event types from the same stream — you’re outside the model. The framework has no mechanism for it. Your options are: batch-read and recompute (simple, expensive at scale), or write stateful streaming logic outside DLT (complex, correctly incremental).

The pattern itself is common. Start/stop events. Login/logout pairs. Request/response matching. Order placement and order fulfilment. Any domain with correlated event pairs from the same stream will encounter this boundary. The architectural implication: if your production pipeline has significant volume of correlated event pairs, plan for stateful stream processing outside DLT from the start — don’t discover this constraint mid-project.

ARCHITECTURAL PRINCIPLE

DLT’s declarative model eliminates orchestration boilerplate, dependency management, and checkpointing overhead for the common transformation case. The cost is that non-standard patterns — stream self-joins, complex stateful aggregations, multi-source correlation — require stepping outside the model, which means writing the imperative Spark code the framework was supposed to replace. This is not a reason to avoid DLT. It is a reason to map your transformation patterns against the framework’s model before committing to it

What This Means for Your Platform Decision

DLT delivers on its promise for the majority of Medallion pipeline transformations. Incremental ingestion, CDC merge, deduplication, cross stream joins, data quality expectations, automatic lineage — all of it works correctly and with significantly less boilerplate than the equivalent PySpark. For a team building a greenfield data platform on Databricks, DLT should be the default choice for Silver and Gold. The velocity gain is real.

The constraint worth knowing before you commit: if your pipeline requires correlated event pairing from a single stream at scale — start/stop, login/logout, request/response — DLT will require either a batch recompute workaround or stateful streaming logic outside the declarative framework. For moderate data volumes, the batch recompute is acceptable. For production pipelines processing millions of paired events, design for stateful streaming from the start. Retrofitting this later is expensive.

Prototype on the free tier before committing to the paid architecture. The concurrency limits, single-workspace constraint, and restricted networking on Free Edition surface platform boundaries fast. Every constraint I hit on Free Edition maps to a real production design decision — sequential vs parallel tasks, single vs multi-environment, file simulation vs live Kafka. Understanding where the boundaries are before you’re billing for compute is worth the two weeks it takes to build something real.

CDC via apply_changes is the right default — but decide on SCD Type 1 vs Type 2 before you write a line of Silver code. The API supports both. The schema, the downstream joins, and the storage cost are fundamentally different. Changing from Type 1 to Type 2 after your Gold layer is built means rebuilding both Silver and Gold.

The declarative model is the right architectural default for Medallion pipelines on Databricks. The question isn’t whether DLT can handle your pipeline — it almost certainly can. The question is whether you’ve mapped your edge cases before you commit. The ones that fall outside the model don’t announce themselves in the documentation. They announce themselves at 2am when your production pipeline runs out of memory recomputing ten million paired events.

If you’ve hit the stream self-join boundary in a production context and found a cleaner solution than batch recompute, I’d genuinely like to know how you solved it.