Databricks Lakeflow: A unified data engineering solution (Part1)

Photo by Adam Khasbulatov on Unsplash

This article is divided into three parts due to its length and the complexity of the topic.

• Part 1 covers the introduction, key components, and foundational concepts of Databricks Lakeflow.
• Part 2 focuses on the initial hands-on implementation
• Part 3 continues with the remaining implementation steps, key takeaways, and references.

TL;DR — Databricks Lakeflow in a Nutshell:

Databricks Lakeflow is a unified data engineering solution that simplifies the ingestion, transformation, and orchestration of both batch and streaming data. This article walks through:

• Core Components: Lakeflow Connect, Declarative Pipelines, and Jobs.
• Foundational Concepts: Batch vs. streaming, Medallion Architecture, CDC/SCD, and joins.
• Hands-On Demo: A full pipeline using config and transaction datasets, showcasing full and incremental loads, CDC/SCD logic, and stream-static joins.
• Key Benefits:
• a) Declarative, scalable pipeline development
• b) Built-in support for CDC and SCD
• c) Seamless orchestration across Medallion layers
• d) Production-ready scheduling with Lakeflow Jobs
• e) Config-Driven Enhancement (Advanced Use Case)

Whether you’re building your first pipeline or scaling enterprise workflows, Lakeflow offers a powerful, streamlined approach to modern data engineering.

Introduction:

In June 2025, at the Databricks Data + AI Summit, Databricks officially announced the General Availability (GA) of Lakeflow — a powerful new offering designed to unify and simplify data engineering workflows.

This article explores what Databricks Lakeflow brings to the table and how it empowers data teams to build scalable, end-to-end pipelines with ease. Whether you’re a data engineer, architect, or tech leader, this guide will help you understand the core components, foundational concepts, and practical implementation of Lakeflow. This article covers the following key areas:

• Key Components: A breakdown of the main building blocks of Databricks Lakeflow.
• Foundational Concepts: Essential concepts that support a unified data engineering solution.
• Hands-On Implementation: A practical walkthrough to solidify your understanding with real-world examples.
• Key Takeaways: Simplifying Data Engineering with Lakeflow
• References: Supporting Docs and Implementation Assets

To maintain focus on Lakeflow, this article does not delve into each related Databricks topic in detail. It assumes that the reader has a basic understanding of the relevant Databricks concepts.

Databricks Lakeflow — Key Components

Databricks provides Lakeflow, an end-to-end data engineering solution to deliver high-quality data for downstream analytics, AI, and operational applications. Lakeflow is a unified solution for ingestion, transformation, and orchestration of your data, and includes Lakeflow Connect, Lakeflow Declarative Pipelines, and Lakeflow Jobs.

Photo by Wai Siew on Unsplash

Lakeflow Connect simplifies data ingestion with connectors to popular enterprise applications, databases, cloud storage, message buses, and local files. There are two flavours of connectors: Managed connectors, Standard connectors. Managed connectors provide simple UI and configuration-based ingestion service with minimal overhead, without requiring you to use the underlying lakeflow declarative pipeline APIs and infrastructures. Whereas, standard connectors provide the ability to access the data from a wide range of data sources from within your lakeflow declarative pipelines and other queries.

Photo by Jens Freudenau on Unsplash

Lakeflow Declarative Pipelines is a declarative framework that lowers the complexity of building and managing the efficient batch and streaming data pipelines. Lakeflow declarative pipelines run on the performance-optimized Databricks runtime. In addition, it automatically orchestrates the execution of flows, sinks, streaming tables and materialized views by encapsulating and running them as a pipeline.

Photo by Waldemar on Unsplash

Lakeflow Jobs provide reliable orchestration and production monitoring for any data and AI workload. A job can consist of one or more tasks that run notebooks, pipelines, managed connectors, SQL queries, machine learning training, and model deployment and inference. Jobs also support custom control flow logic, such as branching with if / else statements, and looping with for each statements.

These definitions are taken from official documentations of the databricks.

Foundational Concepts

In order to implement a unified data engineering solution using the lakeflow offering, we also need to understand some other basic concepts of databricks that are explained below.

Photo by Adrien on Unsplash

Batch vs Stream data processing:

In data engineering, batch and streaming are two distinct processing paradigms used for tasks such as ingestion and transformation.

• Streaming is typically associated with low-latency, continuous data processing, often sourced from message buses like Apache Kafka. It enables near real-time insights and is ideal for time-sensitive applications.
• Batch processing, on the other hand, involves scheduled or delayed execution, such as daily or hourly jobs. It often handles large volumes of data in one go — either as a full load or through incremental updates.

Both batch and streaming data processing can be implemented in either a stateful or stateless manner, depending on the nature of the transformation logic.

• Stateful processing is more common in streaming workloads, where the system maintains context or “state” across events. This is essential for operations like windowed aggregations, joins, or deduplication, where the logic needs to remember previously seen data to compute correct results over time.
• Stateless processing, on the other hand, is typically associated with batch workloads, where each run processes data independently without retaining any memory of past executions. For example, a batch job that simply appends new records from a source to a target table without checking for duplicates or maintaining history is stateless.

What sets Databricks Lakeflow apart is its unified engine that seamlessly supports both batch and streaming workloads. This architecture allows data teams to ingest data from cloud object storage in an efficient, incremental fashion using streaming semantics. These streaming jobs can be executed in either triggered or continuous modes, offering flexibility based on latency and throughput requirements.

Later in this article, we’ll walk through a hands-on batch processing use case, demonstrating how to ingest data using both full load and incremental load strategies from a source system — showcasing Lakeflow’s versatility in handling diverse data engineering scenarios.

Photo by Girish Sangammanavar on Unsplash

Medallion Architecture:

The Medallion Architecture is a foundational design pattern in Databricks for organizing data into progressively refined layers:

• Bronze Layer: Raw, unfiltered data ingested from source systems. Often includes duplicates, schema drift, and minimal transformation.
• Silver Layer: Cleaned and enriched data. This layer applies transformations like joins, filters, and deduplication to make data analytics-ready.
• Gold Layer: Business-level aggregates and curated datasets used for reporting, dashboards, and machine learning.

This layered approach promotes data quality, reusability, and governance — and aligns naturally with Lakeflow’s unified orchestration of batch and streaming pipelines across these layers.

Photo by Sheila C on Unsplash

Tables and Views:

Understanding how data is represented and accessed is essential when working with Lakeflow. Here’s a quick overview of the core table and view types in Databricks:

• Table: A persistent, managed dataset stored in Delta Lake format — supports ACID transactions and schema evolution.
• View: A virtual table defined by a SQL query — does not store data but provides a logical abstraction over one or more tables.
• Materialized View: A precomputed view that stores the results of a query — offers faster performance for repeated access and can be refreshed on a schedule.
• Streaming Table: A special type of table that includes the processing logic using flows to define it and continuously ingests and processes streaming data.

Photo by Max Kukurudziak on Unsplash

Types of Joins:

Joins are essential for combining datasets in both batch and streaming pipelines. Here’s a quick breakdown of the key join types:

• Batch Join: A traditional join between two static (batch) datasets — executed once, typically as part of a scheduled job.
• Stream-Stream Join: A real-time join between two continuously updating streaming sources — requires watermarking and state management to handle late or out-of-order data.
• Stream-Static Join: A join between a streaming source and a static (batch) dataset — commonly used to enrich real-time data with reference or lookup tables.

Photo by John Wiesenfeld on Unsplash

Change Data Capture (CDC) and Slowly Changing Dimensions (SCD):

Handling data changes over time is a key aspect of building reliable pipelines. Here’s how Databricks Lakeflow supports it:

• Change Data Capture (CDC): A technique to identify and process only the changed records (inserts, updates, deletes) from a source system — enabling efficient, incremental data ingestion.
• Slowly Changing Dimensions (SCD): A strategy for managing historical changes in dimensional data:
• SCD Type 1: Overwrites old data with new values — no history is preserved.
• SCD Type 2: Preserves history by creating a new record for each change, often with effective dates or versioning.

Lakeflow makes it easy to implement both CDC and SCD patterns using Delta Lake features.

To be continued …

I hope this Part 1 article provided valuable insights into the Databricks Lakeflow offering. To explore the hands-on implementation in detail, please refer to Part 2 of the article.