Publication

Data Engineer Things

@data-engineer-things

articles: 74
total claps: 4,183
top claps: 1,119
avg read: 7.5 min
paywalled: 28
active: 2023-09-11 → 2026-08-19

Wiki topics

🔧 · Data Engineering · 66 GRW · Growth & Analytics · 11 AI · AI · General · 11 🏛️ · Architecture · 9 ML · Machine Learning · 9 AGT · AI Agents · 9 ☁️ · DevOps & Cloud · 9 🎬 · Film & Television · 7

Top tags

#data-engineering ·56 #python ·11 #databricks ·10 #airflow ·9 #dbt ·9 #sql ·7 #data-warehouse ·7 #snowflake ·6 #data-science ·6 #analytics ·5

TCH MDA

HE Henry · Data Engineer Things

5 Data Ingestion Methods Every Data Engineer Should Know

How Data Actually Gets Into Your Pipeline- Full load, incremental, CDC, streaming, and API with Python code.

Apr 22, 2026 · 11 min read · 1,119 claps paywalled #data-engineering #data-pipeline #python

5 Data Ingestion Methods Every Data Engineer Should Know

AI TCH HUM SCI LIF

HE Henry · Data Engineer Things

How I Integrated AI Into My Data Journey — And Stopped Worrying It Would Replace Me

The Tool Has Changed. The Motivation Hasn’t. Curiosity is still the only thing that matters

May 18, 2026 · 7 min read · 479 claps paywalled #data-engineering #data-science #ai

How I Integrated AI Into My Data Journey — And Stopped Worrying It Would Replace Me

TCH

HE Henry · Data Engineer Things

Data Modeling for Data Engineers: OLTP, OLAP, Inmon, and Kimball Explained

Understanding OLTP, OLAP, and the two roads your data can take after the transaction is recorded

Apr 27, 2026 · 8 min read · 414 claps paywalled #data-engineering #data-modeling #data-warehouse

Data Modeling for Data Engineers: OLTP, OLAP, Inmon, and Kimball Explained

TCH AI SCI

HA Hang Nguyen · Data Engineer Things

Web Scraping with APIs: A Complete Tutorial with Real Examples

A Hands-On Python Guide to Extracting Clean Data Efficiently

Jun 22, 2026 · 10 min read · 154 claps paywalled #data-engineering #data-science #python

Web Scraping with APIs: A Complete Tutorial with Real Examples

ART TCH AI

SA Satyam Sahu · Data Engineer Things

How to Design a Data Pipeline for a National Event (System Design Guide)

Real-time ingestion, distributed aggregation, and fault-tolerant serving at national scale

Jun 17, 2026 · 17 min read · 152 claps paywalled #data-engineering #system-design-interview #kafka

How to Design a Data Pipeline for a National Event (System Design Guide)

TCH MDA MKT

SA Santosh Joshi · Data Engineer Things

Consume Data with Structured Streaming

Real-time Data Processing with Azure Event Hub and PySpark Structured Streaming on Azure Databricks

Apr 18, 2025 · 4 min read · 128 claps paywalled #structured-streaming #pyspark #event-hub

TCH

AN Ana Escobar · Data Engineer Things

A Data Platform Engineer’s Guide to DBT Fusion in 2026

How the DBT Fusion Engine & State-Aware Orchestration can slash your warehouse bill by 40%

Jan 13, 2026 · 6 min read · 114 claps #dbt #dbt-fusion #data-platforms

A Data Platform Engineer’s Guide to DBT Fusion in 2026

TCH

SM Smit Shilu · Data Engineer Things

SQL-only DLT pipelines in Databricks: what the docs don’t tell you

A practitioner’s guide to building production-grade medallion pipelines — without writing a single line of transformation Python. Yes…

Jul 01, 2026 · 5 min read · 110 claps #databricks #sql #dlt

SQL-only DLT pipelines in Databricks: what the docs don’t tell you

ART TCH MDA

VU Vu Trinh · Data Engineer Things

Everything you need to know about Spark Structured Streaming

From its architecture, event-time processing, stateful processing to how it achieves fault tolerance.

Jun 08, 2026 · 16 min read · 104 claps paywalled #data-engineering #apache-spark

Everything you need to know about Spark Structured Streaming

AI TCH SOC

AJ Ajith Shetty · Data Engineer Things

Coding to CLAWing: AI-Native Data Engineering using OpenClaw and MCP

Integration of AI Assistant with Data

Jul 02, 2026 · 8 min read · 101 claps #data-engineering #openclaw #agentic-ai

Coding to CLAWing: AI-Native Data Engineering using OpenClaw and MCP

TCH

JO Joachim Hodana · Data Engineer Things

Writing Airflow DAGs that don’t fight you in production

Most Airflow performance problems are not infrastructure problems. Adding more workers or tuning the scheduler helps, but the biggest gains…

Jun 23, 2026 · 6 min read · 100 claps #airflow #data-engineering #python

Writing Airflow DAGs that don’t fight you in production

TCH MDA MKT

FE Felix Gutierrez · Data Engineer Things

Unlocking Data Insights with Rill: A Comprehensive Guide to Streamlined Data Analytics

Harness the Power of Rill for Real-time Data Processing and Analytics

Sep 11, 2023 · 7 min read · 76 claps #analytics #rill #business-intelligence

Unlocking Data Insights with Rill: A Comprehensive Guide to Streamlined Data Analytics

TCH

VI Vishal Chandra · Data Engineer Things

Setting Up SMTP Email Alerts for DAG Failures in Airflow

Step-by-Step Guide to Setting Up SMTP Email Alerts for DAG Failures in Airflow

Mar 01, 2025 · 3 min read · 74 claps paywalled #smtp-server #smtp #airflow

TCH

HE Henry · Data Engineer Things

dbt Snapshots Explained: Track How Your Data Changes Over Time [dbt Series #6]

A hands-on guide to building Type 2 slowly changing dimensions with dbt and DuckDB

Jul 10, 2026 · 10 min read · 71 claps paywalled #data-engineering #sql #data-modeling

dbt Snapshots Explained: Track How Your Data Changes Over Time [dbt Series #6]

TCH

JO Joachim Hodana · Data Engineer Things

6 tools that keep your dbt project clean

There is more to dbt linting than SQLFluff

Jun 07, 2026 · 6 min read · 70 claps #dbt #data-engineering #code-quality

6 tools that keep your dbt project clean

TCH LIF

SM Smit Shilu · Data Engineer Things

Passing custom variables to Databricks DLT pipelines using Asset Bundles

How to manage catalog names, schema names, and environment-specific config without hardcoding a single value — and actually sleep at night…

Jul 02, 2026 · 5 min read · 65 claps #databricks #databricks-asset-bundles

Passing custom variables to Databricks DLT pipelines using Asset Bundles

AI TCH

AD Adediwura Boluro-Ajayi · Data Engineer Things

CI/CD for On-Premises SQL Server: The Guide I Wish Existed Part II

Automating SQL Server database deployment with TeamCity Server and Agent.

Aug 19, 2026 · 12 min read · 55 claps paywalled #data-engineering #teamcity #ci-cd-pipeline

CI/CD for On-Premises SQL Server: The Guide I Wish Existed Part II

TCH

VI Vishal Chandra · Data Engineer Things

Setting Up Databricks Connection in Airflow: A Step-by-Step Guide

Learn how to set up a Databricks connection in Airflow to seamlessly orchestrate and manage your Databricks workflows. 🔄

Mar 04, 2025 · 2 min read · 54 claps paywalled #airflow #databricks #data-engineering

Setting Up Databricks Connection in Airflow: A Step-by-Step Guide

TCH

SM Smit Shilu · Data Engineer Things

Deploying multi-pipeline Databricks bundles: keeping everything in sync

How to structure, correlate, and deploy a complete Databricks project — pipelines, jobs, workflows, and CI/CD — as one coherent unit. No…

Jul 02, 2026 · 6 min read · 51 claps #databricks #deployment #databricks-asset-bundles

Deploying multi-pipeline Databricks bundles: keeping everything in sync

TCH AI

VI Vishal Chandra · Data Engineer Things

Setting Up Azure Connections in Airflow (Event Hub, IoT Hub, Blob Storage)

Learn how to set up Azure connections in Airflow to interact with Event Hub, IoT Hub, and Blob Storage effortlessly. ☁️

Feb 20, 2025 · 3 min read · 50 claps paywalled #azure-event-hub #blob-storage #azure-iot-hub

Setting Up Azure Connections in Airflow (Event Hub, IoT Hub, Blob Storage)

TCH

VI Vishal Chandra · Data Engineer Things

Managing Airflow Secrets Securely with Azure Key Vault

Learn how to securely manage your Airflow secrets using Azure Key Vault for enhanced protection of sensitive credentials. 🔐

Feb 28, 2025 · 3 min read · 50 claps paywalled #azure-key-vault-secrets #azure-key-vault #airflow

Managing Airflow Secrets Securely with Azure Key Vault

TCH MDA

AL Alvaro Leandro Cavalcante Carneiro · Data Engineer Things

Stop Staring at Cron Expressions: Airflow Just Got a Google Calendar Upgrade

Discover the missing visual calendar that finally lets you manage cron and timedelta DAGs like a human.

Jun 15, 2026 · 6 min read · 40 claps #airflow #python #data-engineering

Stop Staring at Cron Expressions: Airflow Just Got a Google Calendar Upgrade

GA Gaurav Goel · Data Engineer Things

Forward Deployed Engineers: What Pharma needs to know about Tech’s fastest growing role

AI tools are getting better but getting them to work inside pharma’s regulated systems is a different problem entirely

May 30, 2026 · 12 min read · 37 claps paywalled #artificial-intelligence #pharmacovigilance #automation

Forward Deployed Engineers: What Pharma needs to know about Tech’s fastest growing role

MA Marin Aglić · Data Engineer Things

Using Marquez as a lineage tool for Celery — adding the parent-run facet

This story is the second one about integrating Celery with Marquez using the OpenLineage Python package. In this story, we take the first…

Sep 22, 2024 · 7 min read · 33 claps #celery #marquez #data-lineage

Using Marquez as a lineage tool for Celery — adding the parent-run facet

AI ART TCH SCI

RE René Luijk · Data Engineer Things

Three Lessons Learned Building Data Platforms From Scratch

The case for boring tools, smaller ambitions, and building for the team you have

Jun 15, 2026 · 4 min read · 30 claps #data-engineering #data-science #ai