#delta-lake

Advanced security alerting with DuckDB and Delta Lake | Clickdetect

Hey, do you like DuckDB and want to generate security alerts using it? I have a solution for you.

Your Spark Job Is Running. Then a Server Dies. Here’s Exactly What Happens Next.

What Apache Spark does when a Worker Node completely crashes — and why it’s smarter than you think.

Parquet vs. Delta Lake: Why Modern Data Engineering Prefer Delta Lake

If you are building data pipelines, you will inevitably choose between Parquet and Delta Lake. The foundational difference is simple…

Optimize Storage Based on Ingestion Strategy: The Fabric Lakehouse Maintenance Guide

In modern data lakehouse design, mastering the core lifecycle elements — partitioning, data compaction (OPTIMIZE), and storage pruning…

The PySpark Interview Question That Trips Up 90% of Senior Data Engineers (And How to Nail It)

Why Delta Lake MERGE Gets Slow on Wide Tables, and How I Actually Fix It

The cluster was never the problem. Write amplification on a wide table is, and throwing bigger nodes at it just burns money slower.

14,000 Files, 40GB: Fixing a Slow Delta Table in Azure Synapse

What OPTIMIZE, Z-ORDER, partitioning, and VACUUM actually changed

Lakebase CDF: The Simplest Way to Stream Postgres Changes into Your Lakehouse

Databricks Native change data capture from Postgres into Delta — no connectors, no staging pipelines, no operational overhead.

Delta Lake & the Lakehouse: What the Transaction Log Actually Does

DataForge Series: Databricks Data Engineer Associate - Part 1 of 16

Open Table Format Interoperability Using Apache XTable (Incubating)

Breaking Lakehouse Silos: Open Table Format Interoperability with Apache XTable

The State of Apache Iceberg Catalogs in June 2026

Delta Lake & Change Data Feed: The Modern Data Engineer’s Secret Weapon

How Delta Lake solves real-world data reliability problems — and how Change Data Feed takes it to the next level

The Triad (Hudi, Iceberg, Delta) of Open Table Formats [3/n] in 2026

In the last 2 blogs we explored the origins, motivations, design philosophies and architecture of the triad of Open Table Formats:

Deep Dive into Delta Lake Transaction Log (_delta_log)

Part of series: Databricks Complete Guide (Beginner to Advanced)

The Triad (Hudi, Iceberg, Delta) of Open Table Formats [2/n] — Architecture Deep Dive

In the previous blog in the series, we explored the history and design philosophies behind Hudi, Iceberg and Delta Lake. In this post…

Databricks Sink: The Easiest Way to Stream Data Out to Event Hubs(and Beyond) from Delta Tables

Why this article?

Understanding Open Table Formats in the Modern Data Lakehouse

Data lakehouses have become the foundation of modern analytics platforms, combining the scalability of data lakes with the reliability and…

Why Lakehouse Architecture Feels Like the Future of Data Engineering

Recently, while learning Databricks, I came across the term “Lakehouse Architecture” almost everywhere. Initially, it sounded like another…

Delta Sharing Explained: The End of “Can You Send Me That CSV?”

How Databricks kicked out the data export — and why the old way of sharing data was secretly costing your company a fortune.

Delta Lake Liquid Clustering: The Internal Mechanics Every Data Engineer Should Know

Originally published in full on Substack — @nishankmahore

The Undo Button for Big Data: An Intro to Delta Lake Time Travel

Every Data Engineer will eventually experience “The Moment.”

Building a Governed Data Lakehouse

Introduction

I Built a trading analytics pipeline on Databricks — without writing a single line of code

The approach I took

The Triad of Open Table Formats [1/n] — History, Motivation & Design Philosophy

Context

Building a Production-Grade Data Lakehouse on Labour Force Data — Without a Cloud Bill

A walkthrough of how modern data pipelines actually work, built end-to-end on a laptop.

What Happens If Two Jobs Run a Delta MERGE on the Same Partition? (Concurrency Deep Dive)

Imagine this: You’re orchestrating two scheduled ETL jobs that both do MERGE operations into the same Delta Lake table. They happen to hit…

Your Data Lakehouse Has a Memory Problem — And Your AI Is Inheriting It

How historical biases, zombie tables, and undocumented schema evolution in lakehouses silently corrupt the AI systems built on top of them.

Databricks Autoloader: Schema Evolution vs Schema Inference

A practical guide for Spark engineers stepping into streaming pipelines

SAP’s Dremio acquisition gets me confused

Since 2025, I have been warning about SAP Catalogs' problem”, Dremio acquisition by SAP has made it even worse. I did not see this coming…

How the OPTIMIZE Command Supercharges Delta Table Performance in Azure Synapse and Databricks

Published on Medium · Data Engineering · ~6 min read

Data Engineer Interview Question : How Does Databricks Handle Job Failures?

You hit “Run” on your Databricks job, lean back, and… it fails. Now what? Does the whole run die? Do you lose days of work? Or does…

How Databricks Handles Metadata Management: Unity Catalog Secrets Revealed (2026 Guide)

Imagine losing track of your data’s origin, permissions, and lineage in a massive lakehouse — chaos ensues, compliance fails, and teams…

Delta Table Explosion: How to Shrink a Monster Table Before It Crushes Your Spark Jobs

Imagine your Delta table starts as a lean, mean data machine — but months of streaming inserts later, it’s a bloated beast slowing queries…

How I Cracked the Databricks Certified Data Engineer Associate Exam in Just One Week

A practical, no-fluff guide from a seasoned software engineer who went from zero Databricks knowledge to certified in seven days.

Embracing the Chaos: A Pragmatic Guide to Schema Evolution in Delta Lake

Every data engineer has lived through this exact scenario: It is 2:00 PM on a Friday. The upstream software engineering team deploys a new…

How Databricks Executes a Job Internally? (From Button Click to Result)

What Really Happens When You Click “Run”?

Databricks Interview Question : How Databricks Handles Data Updates (MERGE INTO)

Imagine this: you have a Delta Lake table with millions of rows in Databricks, and every hour you receive new data. Some records are new…

Simplifying SCD type tables

Data doesn’t stay still. People change addresses, accounts change status, products change prices. SCD (Slowly Changing Dimensions) is your…

【硬核大師班】PARTITION BY 的末日！深入拆解 Liquid Clustering 與 UniForm 的底層儲存演算法

前言：那個被「分區策略」詛咒的資料工程師

Databricks Lakebase: Rethinking OLTP for the AI Era

The Evolution of OLTP Databases

Implementing Change Data Feed for Incremental Processing in Microsoft Fabric Lakehouses

Most lakehouse pipelines are still doing one expensive thing repeatedly: reprocessing entire tables, even when only a tiny fraction of rows…

Liquid Clustering + Deletion Vectors — what they actually are, how they work under the hood, and…

Most data engineers still reach for Hive partitioning + ZORDER by default. That’s an outdated mental model. But before you enable Liquid…

Using Merge to create a append/historical table with Delta lake

Yea yea, I know that sounds a bit unusual but below is why using merge to create a table that requires history which usually means append…

Under the Hood of the Lakehouse (Part 1): Overview and Apache Iceberg Internals

If you’ve worked with big data platforms over the last decade, you’ve probably felt the pain of just files on S3 at scale. Pipelines step…

Time Travel with Delta Lake

Modern data analysis rarely happens on static datasets. Data is constantly updated, corrected, and enriched as new information becomes…

Databricks series and production stories : 3

Databricks is evolving fast, make sure your platform understanding does too!

Mastering Delta Table Management: Essential Operations to Prevent Slow Queries

Maintaining your Delta tables is not just a best practice; it is a necessity to avoid spending your entire week investigating why…

The Engine of the Lakehouse: A Deep Dive into Delta Lake

In my previous article, we explored how the Data Lakehouse emerged as the “Holy Grail” of data architecture, blending the best of…

How does Delta Lake improve data reliability in Azure?

In the modern data landscape, organizations are rapidly moving away from costly and rigid data warehouses toward the scalability and…

Improving Lakehouse Storage Cost Visibility to Reduce Total Cost of Ownership (TCO)

Written By: Saul Tawil