Articles

AI TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

🤖 SDA 05: Profiling Source Data — Distributions, Nulls, Formats, and Outliers

SDA 04 gave the Synthetic Data Agent a governed map of the data estate. It can see tables, columns, types, tags, comments, and declared…

Jul 31, 2026 · 19 min read · 0 claps #databricks #synthetic-data #data-engineering

ART TCH MDA MKT

BR Bran Kop, Engineer @Conformal, Founder of aiHQ · Towards Data Engineering

Batch and Streaming: Two Sides of the Data Architecture Coin

Every system that processes data eventually confronts one fundamental question: When is the data complete enough to be moved to the compute…

Jul 17, 2026 · 5 min read · 28 claps #data-engineering #stream-processing #data-architecture

TCH SPT

KU Kushalvishwa · Towards Data Engineering

How to Handle Long-Running Jobs in Databricks: A Practical PySpark Playbook

By Kushal Vishwakarma

Jul 16, 2026 · 3 min read · 1 claps #data-engineering #databricks #pyspark

TCH

NE Nevenka Lukic · Towards Data Engineering

Designing Idempotent Data Pipelines

Article 4 in the single-engineer data stack series. Article 1 swapped Spark for DuckDB. Article 2 swapped Pandas for Polars. Article 3…

Jul 16, 2026 · 8 min read · 4 claps #data-engineering #python #data-pipeline

AI TCH SCI

PE Peggie Mishra · Towards Data Engineering

Let’s explore different types of API methods

Dear Readers,

Jul 15, 2026 · 4 min read · 0 claps paywalled #python #python-programming #data-science

ART TCH AI SCI

RU Rutvij Acharya · Towards Data Engineering

The Data Engineering Landscape Is Shifting Fast. Here’s What Actually Matters.

By Rutvij Acharya — Lead Data & Integration Engineer | SnapLogic Architect

Jul 15, 2026 · 8 min read · 54 claps #data-science #data-architecture #data-engineering

TCH

FE Feroz Mulla · Towards Data Engineering

BigQuery Cost Optimization: A Comprehensive Guide

BigQuery is a powerful Data platform that enables you to create peta-byte scale Data Warehouses, Performant Data Marts, Open format…

Jul 15, 2026 · 9 min read · 0 claps #bigquery #google-cloud-platform #data-warehouse

ART TCH AI

KU Kushalvishwa · Towards Data Engineering

When Should You Use Liquid Clustering? A Practitioner’s Benchmark of Lakehouse Layout Strategies”

Liquid Clustering vs ZOrder Benchmark

Jul 15, 2026 · 8 min read · 5 claps #data-engineering #data-lakehouse #programming

AI TCH DSN

PA paul copper · Towards Data Engineering

Data Engineering for AI Agents: What Actually Changes in Your Pipeline Design!

Nearly every data engineering team today is hearing the same request: “Can we add an AI agent to our pipeline?”

Jul 14, 2026 · 6 min read · 12 claps #data-engineering #ai-agent #data-analysis

TCH MDA AI SCI

SA Saurav Singh · Towards Data Engineering

How Data Actually Flows Through a Modern Data Platform

Four systems failed quietly for three weeks. Here’s what actually broke, from the log to the dashboard.

Jul 13, 2026 · 7 min read · 53 claps paywalled #data-science #data-engineering #dbt

ART TCH

JA Jaco van der Laan · Towards Data Engineering

Source Modeling: The Missing Workflow

Every data platform assumes the source model is a given. In banks, it almost never is — here’s the workflow nobody named and the role…

Jul 13, 2026 · 8 min read · 1 claps paywalled #data-engineering #data-modeling #data-architecture

TCH MKT

FA Fahad Shamim · Towards Data Engineering

Mastering CI/CD in Microsoft Fabric: The Complete Dev/Test/Prod Playbook for Enterprise Analytics

How Deployment Pipelines, Variable Libraries, and Git Integration Work Together to Deliver a Modern, Scalable DevOps Lifecycle

Jul 12, 2026 · 9 min read · 0 claps #microsoft-fabric #ci-cd-pipeline #deployment-pipelines

TCH

VA VaishaliChoudhary@Work · Towards Data Engineering

The Silent Failures Your Data Tests Aren’t Catching

Building a trustworthy semantic layer with dbt, Snowflake & Airflow

Jul 12, 2026 · 8 min read · 2 claps #semantic-layer #silent-failures #error-handling

AI ART TCH MDA

NA Naveena · Towards Data Engineering

How We Replaced Text-to-SQL Wrappers with a Warehouse-Native Cortex Layer for ITAM Compliance

A compliance manager asks: “Show me unattested IBM DB2 installations.”

Jul 12, 2026 · 5 min read · 0 claps #snowflake #data-engineering #generative-ai

TCH MDA GEN

IS Ishwarya Modika · Towards Data Engineering

Beyond Secret Scopes: How Unity Catalog Brings Real Governance to Databricks Secrets

Fine-grained, auditable access control finally comes to Databricks secrets — here’s what changed and how to use it.

Jul 12, 2026 · 4 min read · 3 claps #databricks #databricks-unity-catalog #data-engineering

TCH ECO HUM

IT IT Training Institute · Towards Data Engineering

How VPNs Improve Privacy and Security Online — Why Learning Cyber Security Matters in 2026

In today’s digital world, online privacy has become more important than ever. From online banking and shopping to social media and remote…

Jul 12, 2026 · 4 min read · 0 claps #cybersecurity #cyber-security-awareness #cyber-security-solutions

TCH

BR BricksNotes for Data Engineers · Towards Data Engineering

Databricks Is Moving Fast, and the Data + AI World Should Pay Attention.

If you missed the Summit, the bigger story is not just the number of announcements. It is the direction behind them.

Jul 12, 2026 · 4 min read · 25 claps #databricks #data-engineering #big-data

TCH

TH Think Data · Towards Data Engineering

Why Senior Data Engineers Ask Questions Before They Write SQL

Most candidates treat silence as failure. In a Data Engineering interview, it’s often the opposite.

Jul 12, 2026 · 4 min read · 1 claps paywalled #data-engineering #sql-interview #data-engineer-interview

TCH

MA Maximilian Oliver · Towards Data Engineering

Building My First Modern Data Pipeline Completely Changed How I Think About Data Engineering

I used to believe data engineering was mostly about moving data from one database to another. After building real-time pipelines, I…

Jul 11, 2026 · 4 min read · 2 claps paywalled #data-engineering #data-engineer #data-pipeline

TCH

GE G e o r g i a n · Towards Data Engineering

Managing Network Security on Databricks SQL Serverless

What Regulated Industries Need to Know and Implement

Jul 11, 2026 · 8 min read · 0 claps #databricks #azure #aws

AI HUM

PE Peggie Mishra · Towards Data Engineering

How Temperature Impacts AI Responses

Dear Readers,

Jul 10, 2026 · 2 min read · 71 claps paywalled #artificial-intelligence #machine-learning #llm

TCH MDA SPT

AL Alireza Khorami · Towards Data Engineering

Building a Production-Grade Apache Spark Platform on Kubernetes

Deploying the Spark Operator, Building Custom Images, Running Structured Streaming through Helm, Integrating Airflow, and Enforcing…

Jul 10, 2026 · 29 min read · 63 claps #spark #kubeflow #kubernetes

AI GEN HUM

AK Akshat Jain · Towards Data Engineering

Should Artificial Intelligence Be Held Responsible for Its Actions?

As artificial intelligence makes real-world decisions, who is responsible when things go wrong?

Jul 09, 2026 · 5 min read · 0 claps #artificial-intelligence #ai-ethics #technology

TCH

MU Muthupalaniappan Ramanathan · Towards Data Engineering

Building Trust in Data: Why Data Reliability Engineering Is More Than Just Monitoring

“The most dangerous data issue isn’t the one that fails loudly — it’s the one that quietly changes a business decision.”

Jul 08, 2026 · 3 min read · 0 claps #data #data-reliability #anomaly-detection

TCH HUM MKT AI

YI Yingzhao Ouyang · Towards Data Engineering

Teaching Graphify to Understand Databricks Notebooks

A side quest from my “Build Your Own LinkedIn Analytics” project

Jul 08, 2026 · 9 min read · 7 claps #databricks #ai-coding #claude

TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

✨Spark UI Trek: The Klingon Attack

Spark UI: The Last Frontier. A chaotic telemetry matrix where cheerful progress bars pretend everything is fine, while red error logs…

Jul 07, 2026 · 54 min read · 0 claps #apache-spark #pyspark #data-engineering

TCH MDA

AB Abhirup Pal · Towards Data Engineering

Databricks Unity Catalog ↔ Snowflake Iceberg Integration — Full Setup Guide

Bidirectional access between Snowflake and a Databricks Unity Catalog managed Iceberg table, via the UC (Unity Catalog) Iceberg REST…

Jul 07, 2026 · 5 min read · 2 claps #databricks #snowflake #apache-iceberg

AI ART TCH DSN

BA Balu Rathinam · Towards Data Engineering

Rethinking SQL Server Agent Scheduling: An On-Demand Orchestration Pattern

Reducing Unnecessary SQL Server Agent Executions Through Application-Driven Scheduling

Jul 06, 2026 · 12 min read · 50 claps #sql-server #data-engineering #software-architecture

TCH

TH Think Data · Towards Data Engineering

The 5 Signals Every Data Engineering Interviewer Is Secretly Scoring

You think you’re being asked about SQL. Most of the time, you’re actually being measured on something else entirely.

Jul 06, 2026 · 5 min read · 5 claps paywalled #data-engineering #software-engineering #interview

ART TCH SOC

BA Balu Rathinam · Towards Data Engineering

Database Architecture Starts Before Database Selection

Are We Asking the Right Questions?

Jul 06, 2026 · 5 min read · 0 claps #data-architecture #database #data-engineering

AI TCH SCI

BV B V Sarath Chandra · Towards Data Engineering

In 2026, AI can write your PySpark scripts in three seconds.

If your only skill is moving data from Point A to Point B, your career is in danger.

Jul 06, 2026 · 4 min read · 56 claps paywalled #data-science #technology #programming

AI

CO CortexFlow · Towards Data Engineering

Mastering CUDA and High-Performance Computing, Part VI

A Deep Dive from Compiler Internals to High-Performance Parallel Computing

Jul 05, 2026 · 14 min read · 0 claps paywalled #software-development #software-engineering #hardware

AI TCH SCI

SH Shashwath Shenoy · Towards Data Engineering

Data Engineering for AI Agents: What Actually Changes in Your Pipeline Design!

Your DAGs were built for a world where the same input always produces the same output. Agents break that assumption, here’s what to…

Jul 04, 2026 · 5 min read · 101 claps paywalled #artificial-intelligence #data-engineering #technology

TCH

NE Nevenka Lukic · Towards Data Engineering

CDC Explained: Why Full Refresh ETL Is Dying

This is the third article in a trilogy on the modern single-engineer data stack. The first covered DuckDB replacing small-scale Spark. The…

Jul 04, 2026 · 11 min read · 0 claps #data-engineering #data-pipeline #python

AI ART TCH SCI

VI Vinayak Gole · Towards Data Engineering

Architecting SAP Core Transformation With Palantir

Why SAP Brought An Unlikely Partner For ERP Modernization

Jul 04, 2026 · 5 min read · 12 claps paywalled #sap #data-migration #ai

TCH ECO

PR Pravashish M · Towards Data Engineering

Fail Fast or Fail Silently — Why I Always Choose Fail Fast in Production Pipelines

A data engineer’s hard-learned lesson from processing 2TB of financial data daily

Jul 03, 2026 · 9 min read · 1 claps #data-engineering #aws #data-pipeline

AI TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

🤖 SDA 03: Bootstrapping with Declarative Automation Bundles

After a short one-month pause, the Synthetic Data Agent is back.

Jul 03, 2026 · 22 min read · 0 claps #databricks #data-engineering #synthetic-data

TCH

TH Think Data · Towards Data Engineering

Stop Optimizing BigQuery Until You Run These 5 INFORMATION_SCHEMA Queries

Most BigQuery optimization starts in the wrong place.

Jul 02, 2026 · 4 min read · 12 claps paywalled #bigquery #data-engineering #google-cloud-platform

SCI TCH HUM AI

DE Denis Hartl · Towards Data Engineering

Track Local YOLO Training in Azure ML with MLflow

Learn to track local YOLO training runs in Azure ML Workspace with MLflow.

Jul 02, 2026 · 17 min read · 5 claps paywalled #mlops #machine-learning #azure

TCH AI SCI

SU Subodh Shetty · Towards Data Engineering

What is Kafka Connect and why not just write a Consumer?

You Already Know How to Write a Kafka Consumer. Here’s Why You Shouldn’t.

Jul 02, 2026 · 6 min read · 0 claps paywalled #kafka #kafka-connect #data-science

ART TCH

SI Sini Sahadevan · Towards Data Engineering

Beyond Metadata: 15 Factors to Consider When Designing a Metadata-Driven Data Ingestion (MDDI)…

Introduction

Jul 02, 2026 · 3 min read · 0 claps #data-architecture #enterprise-architecture #data-engineering

AI TCH SCI

BV B V Sarath Chandra · Towards Data Engineering

Nobody Told Me Data Engineering Was Just One Giant System

I spent months memorizing ETL, Kafka, Spark, Delta Lake, CDC, Airflow, Data Modeling… until one production incident taught me what they all…

Jul 02, 2026 · 4 min read · 97 claps paywalled #data-science #technology #programming

TCH MDA MKT

NG Ángel Álvarez Pascua · Towards Data Engineering

🕵️ Data Forensics with Databricks, Zingg and GraphFrames🔍

Every dataset has a back alley. From the dashboard, everything looks clean — until production data walks in with fake names, duplicate…

Jul 01, 2026 · 25 min read · 0 claps #data-engineering #databricks #entity-resolution

HUM SOC AI

PH Ph.D. Javier Osuna · Towards Data Engineering

Measuring Flatfish Without Neural Networks

A deterministic OpenCV pipeline for real-time morphometric measurement of flatfish in aquaculture.

Jun 29, 2026 · 12 min read · 0 claps #computer-vision #opencv #aquaculture

AI DSN LIF

PE Peggie Mishra · Towards Data Engineering

How to Automate Daily Tasks with ChatGPT Scheduled Tasks

Dear Readers,

Jun 29, 2026 · 3 min read · 5 claps paywalled #chatgpt #artificial-intelligence #openai

TCH AI SPT SCI

SU Subodh Shetty · Towards Data Engineering

Kafka Connect: Setting up and what the Docs skip

The Three Kafka Topics Running Your Connect Cluster and Why You Should Create Them Yourself

Jun 27, 2026 · 6 min read · 10 claps paywalled #kafka #data-science #data-engineering

TCH MDA AI SCI

SU Subodh Shetty · Towards Data Engineering

Flink restart strategies: Stop letting your Job loop forever

The default retry config will loop your job into oblivion. Here’s what to set instead.

Jun 27, 2026 · 9 min read · 0 claps paywalled #apache-flink #data-science #data-streaming

FA Fahad Shamim · Towards Data Engineering

Why Is Dataflow Gen2 So Slow?

A Guide for Microsoft Fabric Engineers

Jun 27, 2026 · 6 min read · 0 claps #microsoft-fabric #data-factory #pyspark

TCH ECO

AB Abhishek Pan · Towards Data Engineering

Why Delta Lake MERGE Gets Slow on Wide Tables, and How I Actually Fix It

The cluster was never the problem. Write amplification on a wide table is, and throwing bigger nodes at it just burns money slower.

Jun 26, 2026 · 15 min read · 0 claps paywalled #data-engineering #big-data #databricks

TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

Lakebase 🐘 All That She Wants 🧠

The fraud PoC was lightning-fast until we tried to make it helpful.

Jun 24, 2026 · 4 min read · 0 claps #databricks #lakebase #data-engineering