Author

Hubert Dudek

@databrickster

articles: 17
total claps: 553
top claps: 118
avg read: 4.0 min
paywalled: 17
active: 2025-05-27 → 2026-06-25

Wiki topics

🔧 · Data Engineering · 17 ☁️ · DevOps & Cloud · 3 CRY · Crypto & Web3 · 2 AGT · AI Agents · 2 🔭 · Astronomy & Space · 1 DSN · Design · General · 1 🥊 · Combat Sports · 1 🎬 · Film & Television · 1

Top tags

#databricks ·16 #spark ·5 #dab ·4 #sql ·4 #cicd ·3 #lakeflow ·3 #genie ·2 #databricks-news ·2 #databricks-sql ·2 #data-ingestion ·2

TCH

HU Hubert Dudek · Databricks Community Articles

Databricks Docker: from runtime CI/CD to compliance

So far, we’ve had ML and non-ML for classic dedicated compute, but Databricks dropped 12! different Docker images. Most importantly, it is…

Jun 25, 2026 · 4 min read · 118 claps paywalled #databricks #docker #cicd

Databricks Docker: from runtime CI/CD to compliance

TCH

HU Hubert Dudek · DBSQL SME Engineering

Temp Tables Are Here, and They’re Going to Change How You Use SQL

Temporary tables have arrived in Databricks SQL warehouses, bringing a powerful new approach to data engineering that bridges the gap…

Jan 30, 2026 · 4 min read · 77 claps paywalled #databricks #databricks-sql #sql

Temp Tables Are Here, and They’re Going to Change How You Use SQL

AI TCH

HU Hubert Dudek

Databricks News: watermark-based incremental ingestion, MCP in AI gateway, Genie, Vector Search

My favorite update is watermark-based incremental ingestion. I will record a video during the weekend and update the article content.

Apr 21, 2026 · 3 min read · 65 claps paywalled #databricks #databricks-news #void

Databricks News: watermark-based incremental ingestion, MCP in AI gateway, Genie, Vector Search

TCH

HU Hubert Dudek · DBSQL SME Engineering

Deduplicating Data on the Databricks Lakehouse (five major strategies)

Deduplication is a core topic in a Lakehouse because data is stored in files (Delta tables). Unlike many OLTP systems, we do not have…

Feb 28, 2026 · 8 min read · 56 claps paywalled #databricks #deduplication #primary-keys

Deduplicating Data on the Databricks Lakehouse (five major strategies)

TCH

HU Hubert Dudek

Just because you can do it in Databricks doesn’t mean you should. My favourite five bad practices.

Here are a few of the worst common issues I see in production projects that give me the biggest headaches.

Mar 26, 2026 · 3 min read · 47 claps paywalled #databricks #databricks-sql #dab

Just because you can do it in Databricks doesn’t mean you should. My favourite five bad practices.

TCH

HU Hubert Dudek

Stored Procedures in databricks

Databricks Runtime 17.0 introduced support for SQL stored procedures. For example, the procedure below moves orders to demonstrate that…

May 27, 2025 · 4 min read · 39 claps paywalled #databricks #sql #spark

SPT TCH

HU Hubert Dudek

Change Data Feed in Databricks Delta — How to Process It the Most Efficient Way

Recently, the AUTO CDC pipeline command gained a new option, WITH (readChangeFeed=true), which enables out-of-the-box processing of Change…

Apr 11, 2026 · 5 min read · 30 claps paywalled #databricks #cdf #change-data-feed

Change Data Feed in Databricks Delta — How to Process It the Most Efficient Way

TCH DSN

HU Hubert Dudek

Databricks News: Lakeflow Designer, UV package manager, DABs templates, Genie scheduled tasks

Platform development is not slowing down, and we have some exciting news. Databricks Summit is approaching, and in one month, we will have…

May 12, 2026 · 3 min read · 25 claps paywalled #databricks-news #genie #lakeflow

Databricks News: Lakeflow Designer, UV package manager, DABs templates, Genie scheduled tasks

TCH

HU Hubert Dudek

Watermark-Based Incremental Ingestion (Lakeflow Connect query-based capture)

Recently, I wrote a lot about change data capture and the amazing AUTO CDC functionality. But what if we want to ingest data incrementally…

May 05, 2026 · 5 min read · 22 claps paywalled #databricks #data-ingestion #incremental-load

Watermark-Based Incremental Ingestion (Lakeflow Connect query-based capture)

AI SCI TCH

HU Hubert Dudek

Implementing Enterprise Naming Convention, Agentic Way

Recently, Databricks added Workspace skills, which are used by Genie Code

Apr 21, 2026 · 5 min read · 22 claps paywalled #databricks #agents #genie

Implementing Enterprise Naming Convention, Agentic Way

TCH

HU Hubert Dudek

Databricks Lakeflow Jobs Workflow Backfill

In modern data pipelines, incremental processing is the standard approach for efficiency. However, when issues arise (whether from system…

Jan 03, 2026 · 4 min read · 15 claps paywalled #databricks #databricks-lakeflow #backfill

Databricks Lakeflow Jobs Workflow Backfill

TCH

HU Hubert Dudek

The Nightmare of Initial Load (And How to Tame It)

Initial loads can be a total nightmare. Imagine that every day you ingest 1 TB of data, but for the initial load, you need to ingest the…

Feb 07, 2026 · 4 min read · 11 claps paywalled #databricks-lakeflow #data-ingestion #pipeline

The Nightmare of Initial Load (And How to Tame It)

TCH MDA

HU Hubert Dudek · DBSQL SME Engineering

Catalog Commits: make your Managed Delta layer safer and more performant

I see confusion about what catalog commits mean and about catalog-commit enabled tables. It also looks really boring, but it is actually…

Jun 05, 2026 · 4 min read · 7 claps paywalled #databricks #delta #unity-catalog

Catalog Commits: make your Managed Delta layer safer and more performant

ECO TCH

HU Hubert Dudek

Global Job Parameters, Thanks To DABs Mutators

Aren’t you tired of defining the same parameters in every Databricks job? The same catalog variable, the same schema, the same environment…

May 19, 2026 · 4 min read · 5 claps paywalled #databricks #dab #ci-cd-pipeline

Global Job Parameters, Thanks To DABs Mutators

ECO TCH

HU Hubert Dudek

Pass data from Terraform Infrastructure to DABS variables

There are more and more resources available in DABS, and I have to say, defining them is much nicer and easier to manage than in Terraform…

Apr 02, 2026 · 3 min read · 5 claps paywalled #databricks #dab #terraform

Pass data from Terraform Infrastructure to DABS variables

TCH

HU Hubert Dudek

Databricks News: CLI v 1.0.0, AI-tools [LAST UPDATED 25th May]

Content will be updated frequently until the video is released.

May 25, 2026 · 1 min read · 5 claps paywalled #databricks #sql #cli

Databricks News: CLI v 1.0.0, AI-tools [LAST UPDATED 25th May]

TCH MDA

HU Hubert Dudek

Dashboards for Nerds

I don’t like BI tools. I use Databricks AI/BI, and I stopped using Power BI and Qlik a long time ago. However, I always feel like something…

Aug 10, 2025 · 2 min read · 4 claps paywalled #databricks #spark #plotly