Publication

Towards Data Engineering

@towards-data-engineering

articles: 175
total claps: 3,415
top claps: 384
avg read: 8.6 min
paywalled: 68
active: 2024-12-27 → 2026-08-01

Wiki topics

🔧 · Data Engineering · 133 ML · Machine Learning · 28 AI · AI · General · 26 🔬 · Science · General · 25 🏛️ · Architecture · 23 GRW · Growth & Analytics · 22 ☁️ · DevOps & Cloud · 20 💻 · Programming · 14

Top tags

#data-engineering ·105 #big-data ·36 #databricks ·35 #data-science ·25 #apache-spark ·19 #artificial-intelligence ·14 #pyspark ·14 #data-engineer ·12 #data-architecture ·11 #sql ·10

AI LIF TCH

VA Valentin Podkamennyi · Towards Data Engineering

AI role in software engineering growth

AI will expand the software engineering profession by shifting the focus from coding to complex problem solving and orchestration.

Jun 02, 2026 · 4 min read · 384 claps paywalled #software-engineering #artificial-intelligence #tech-trends

VA Valentin Podkamennyi · Towards Data Engineering

Remove Gemini Nano AI model from Google Chrome

Discover how to identify and delete the hidden 4GB Gemini Nano AI model that Google Chrome installs on your computer.

May 17, 2026 · 4 min read · 357 claps paywalled #google-chrome #ai #macos

Remove Gemini Nano AI model from Google Chrome

TCH AI SCI

AN Ansam Yousry · Towards Data Engineering

Databricks Just Leveled Up: New Features That Make Vibe Coding Fast, Safe & Super Cheap

At Data + AI Summit 2026, Databricks announced three new Databricks Apps features that, as a data engineer, actually make sense in real…

Jun 18, 2026 · 4 min read · 250 claps paywalled #data-science #data-engineering #data-analysis

Databricks Just Leveled Up: New Features That Make Vibe Coding Fast, Safe & Super Cheap

ECO

PH Phillip Moxley · Towards Data Engineering

Data Strategy for 2026: What Leaders Must Get Right

Organizations today generate more data than ever before. Yet many leadership teams still struggle to translate this data into meaningful…

Mar 12, 2026 · 5 min read · 110 claps #data-strategy-consulting #data-strategy #creating-a-data-strategy

Data Strategy for 2026: What Leaders Must Get Right

AI TCH SCI

SH Shashwath Shenoy · Towards Data Engineering

Data Engineering for AI Agents: What Actually Changes in Your Pipeline Design!

Your DAGs were built for a world where the same input always produces the same output. Agents break that assumption, here’s what to…

Jul 04, 2026 · 5 min read · 101 claps paywalled #artificial-intelligence #data-engineering #technology

Data Engineering for AI Agents: What Actually Changes in Your Pipeline Design!

AI MED TCH HUM

SW Swarup Dewanjee · Towards Data Engineering

Why Medical AI Cannot Recognize What It Does Not Know

A web based diagnostic tool presents a structured form. It asks for symptoms, duration, intensity, and associated signals. The inputs are…

Apr 15, 2026 · 4 min read · 101 claps #artificial-intelligence #healthcare-ai-challenges #ai-ethics

Why Medical AI Cannot Recognize What It Does Not Know

AI TCH SCI

BV B V Sarath Chandra · Towards Data Engineering

Nobody Told Me Data Engineering Was Just One Giant System

I spent months memorizing ETL, Kafka, Spark, Delta Lake, CDC, Airflow, Data Modeling… until one production incident taught me what they all…

Jul 02, 2026 · 4 min read · 97 claps paywalled #data-science #technology #programming

TCH MDA

SR Sriw World of Coding · Towards Data Engineering

How to Handle Slowly Changing Dimensions (SCD) in Databricks — A Complete Guide

Ever looked at a “customer” report and realized the customer’s city was wrong last month, but right now? If you’re building a data…

May 08, 2026 · 7 min read · 71 claps paywalled #databricks #scd #slowly-changing-dimension

How to Handle Slowly Changing Dimensions (SCD) in Databricks — A Complete Guide

AI HUM

PE Peggie Mishra · Towards Data Engineering

How Temperature Impacts AI Responses

Dear Readers,

Jul 10, 2026 · 2 min read · 71 claps paywalled #artificial-intelligence #machine-learning #llm

ECO TCH

SR Sriw World of Coding · Towards Data Engineering

🚀 Processing 10 TB Data in Apache Spark in 3 Hours: A Cost-Effective Tuning Guide

Apache Spark is a beast when it comes to massive datasets, but not every organization has a 6,000-core cluster lying around to finish a job…

Apr 02, 2026 · 3 min read · 66 claps paywalled #spark-submit #pyspark #cluster-configuration

🚀 Processing 10 TB Data in Apache Spark in 3 Hours: A Cost-Effective Tuning Guide

TCH MDA SPT

AL Alireza Khorami · Towards Data Engineering

Building a Production-Grade Apache Spark Platform on Kubernetes

Deploying the Spark Operator, Building Custom Images, Running Structured Streaming through Helm, Integrating Airflow, and Enforcing…

Jul 10, 2026 · 29 min read · 63 claps #spark #kubeflow #kubernetes

Building a Production-Grade Apache Spark Platform on Kubernetes

TCH MDA

SR Sriw World of Coding · Towards Data Engineering

Databricks Interview Question : How Databricks Handles Data Updates (MERGE INTO)

Imagine this: you have a Delta Lake table with millions of rows in Databricks, and every hour you receive new data. Some records are new…

Apr 16, 2026 · 4 min read · 63 claps paywalled #databricks #delta-lake #databricks-unity-catalog

Databricks Interview Question : How Databricks Handles Data Updates (MERGE INTO)

TCH DSN

MA Maksim Pachkouski · Towards Data Engineering

Databricks Apps Tutorial: Scalable Streamlit modular Template for Production

Databricks Apps is a convenient solution that lets you extend Databricks' capabilities. You can create a custom business app with virtually…

May 09, 2026 · 3 min read · 63 claps #databricks #databricks-apps #streamlit

Databricks Apps Tutorial: Scalable Streamlit modular Template for Production

TCH MDA

SR Sriw World of Coding · Towards Data Engineering

What Happens If Two Jobs Run a Delta MERGE on the Same Partition? (Concurrency Deep Dive)

Imagine this: You’re orchestrating two scheduled ETL jobs that both do MERGE operations into the same Delta Lake table. They happen to hit…

May 16, 2026 · 7 min read · 59 claps paywalled #merge #databricks #delta-lake

What Happens If Two Jobs Run a Delta MERGE on the Same Partition? (Concurrency Deep Dive)

TCH MKT

MO Mohit Daxini · Towards Data Engineering

Meta Data Engineering Interview Question — Data Engineer

Solving a 4-Step Conversion Funnel in SQL and PySpark

May 28, 2026 · 5 min read · 57 claps paywalled #meta #interview #data-engineering

Meta Data Engineering Interview Question — Data Engineer

TCH AI SCI

MU Muaaz · Towards Data Engineering

25 PySpark DataFrame Operations Every Data Engineer Should Know

A practical hands on guide covering joins, aggregations, window functions, schema handling, Spark SQL, and real world transformations.

Jun 15, 2026 · 4 min read · 57 claps #data-engineering #big-data #data-science

25 PySpark DataFrame Operations Every Data Engineer Should Know

AI LIF

PE Peggie Mishra · Towards Data Engineering

3 Hidden macOS Features for Faster Voice Notes

Dear Readers,

May 07, 2026 · 2 min read · 57 claps paywalled #productivity #mac #artificial-intelligence

3 Hidden macOS Features for Faster Voice Notes

AI TCH SCI

BV B V Sarath Chandra · Towards Data Engineering

In 2026, AI can write your PySpark scripts in three seconds.

If your only skill is moving data from Point A to Point B, your career is in danger.

Jul 06, 2026 · 4 min read · 56 claps paywalled #data-science #technology #programming

TCH AI

AS Ashkan Goleh Pour · Towards Data Engineering

Inside StarRocks: A Deep Dive from Data Ingestion to Distributed Query Execution

Following a single row through transactions, tablets, rowsets, segments, compaction, query planning, vectorized execution, and real-time…

Jun 07, 2026 · 15 min read · 55 claps #starrocks #olap #distributed-systems

Inside StarRocks: A Deep Dive from Data Ingestion to Distributed Query Execution

ART TCH AI SCI

RU Rutvij Acharya · Towards Data Engineering

The Data Engineering Landscape Is Shifting Fast. Here’s What Actually Matters.

By Rutvij Acharya — Lead Data & Integration Engineer | SnapLogic Architect

Jul 15, 2026 · 8 min read · 54 claps #data-science #data-architecture #data-engineering

The Data Engineering Landscape Is Shifting Fast. Here’s What Actually Matters.

TCH

TH Think Data · Towards Data Engineering

Kubernetes From First Principles

How Kubernetes actually keeps distributed systems alive

May 22, 2026 · 7 min read · 53 claps paywalled #kubernetes #kubernetes-cluster #kubectl

TCH MDA AI SCI

SA Saurav Singh · Towards Data Engineering

How Data Actually Flows Through a Modern Data Platform

Four systems failed quietly for three weeks. Here’s what actually broke, from the log to the dashboard.

Jul 13, 2026 · 7 min read · 53 claps paywalled #data-science #data-engineering #dbt

How Data Actually Flows Through a Modern Data Platform

TCH

TH Think Data · Towards Data Engineering

A Two-Week Data Migration Investigation Ended with One Line of SQL

How a tiny timezone difference nearly derailed our Hive-to-BigQuery migration

Jun 03, 2026 · 4 min read · 53 claps paywalled #data-engineering #data-migration #bigquery

A Two-Week Data Migration Investigation Ended with One Line of SQL

TCH MKT

SA Satish Kumar · Towards Data Engineering

Zero-Copy Connectors: Real-Time SAP Data in Snowflake Without a Single ETL Pipeline

There’s a class of integration problem that every enterprise data team eventually hits: SAP is the system of record, Snowflake is the…

May 28, 2026 · 12 min read · 52 claps #snowflake #data-engineering #sap

Zero-Copy Connectors: Real-Time SAP Data in Snowflake Without a Single ETL Pipeline

ART TCH

PH Phillip Moxley · Towards Data Engineering

3 Signs Your Business Has Outgrown Its Data Lake Architecture

And what to do before the cracks become craters

Apr 03, 2026 · 9 min read · 51 claps #data-lake #data-lake-consulting #data-lakehouse