#apache-spark

TCH

SR Sriw World of Coding

Data engineer Interview Question : The PySpark Timezone Bug That’s Silently Corrupting Your Global…

Two transactions happened at the exact same real-world moment. Your DataFrame thinks they’re 6.5 hours apart. Here’s how to fix timezone…

Aug 16, 2026 · 7 min read · 4 claps paywalled #apache-spark #timezone #delta-lake

TCH

DA DataWithRohit

Databricks Lakeflow Jobs — The Ultimate Associate & Professional Exam Guide

Repair Runs, Multi-Task Jobs, Dependencies, Task Values, Triggers, Parameters, Queueing, Monitoring & Exam Traps

Aug 09, 2026 · 15 min read · 0 claps #databricks #databricks-certification #apache-spark

TCH

DA DataWithRohit

Databricks Professional Exam Guide: Python File Dependencies, Relative Imports & Python Wheels

As Databricks projects grow, notebooks often evolve from simple, self-contained units into larger applications containing reusable…

Aug 08, 2026 · 11 min read · 0 claps #databricks #databricks-certification #apache-spark

TCH MDA

DA DataWithRohit

Databricks Stream-Static Joins: The Exam-Ready Guide for Associate and Professional Certifications

How stateless enrichment works, how Delta Lake snapshots behave, where streaming state really appears

Aug 04, 2026 · 25 min read · 0 claps #databricks #spark #apache-spark

MKT

JA Jaime Teixeira de Araújo Júnior

Construindo um lakehouse de dados públicos (parte 3): de arquivo solto a tabela de verdade

Na parte 2, o dado bruto das duas fontes chegou inteiro à camada Bronze, sem nenhuma transformação, guardado como JSON simples e…

Aug 02, 2026 · 10 min read · 0 claps #hive-metastore #spark #apache-spark

TCH SPT

SR Sriw World of Coding · CodeToDeploy

Client Mode vs Cluster Mode in Apache Spark: The Deployment Secret Most Developers Get Wrong

If you’ve ever stared at a Spark job failure and wondered “wait… where is my driver even running?” — this article is for you.

Aug 01, 2026 · 8 min read · 62 claps paywalled #apache-spark #deployment-model #cluster

TCH

TH The Data Forge

🏗️Writing Production-Grade PySpark — Patterns That Scale

Most PySpark Code Works in Dev and Breaks in Production. Here Are the Patterns That Survive Both.

Jul 31, 2026 · 11 min read · 2 claps #apache-spark #pyspark #data-engineering

AI TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

🤖 SDA 05: Profiling Source Data — Distributions, Nulls, Formats, and Outliers

SDA 04 gave the Synthetic Data Agent a governed map of the data estate. It can see tables, columns, types, tags, comments, and declared…

Jul 31, 2026 · 19 min read · 0 claps #databricks #synthetic-data #data-engineering

TCH MDA

MA Mamata

Understanding Databricks Auto Loader vs Delta Lake Schema Evolution: Lessons from a Real Debugging…

While building a streaming ingestion pipeline in Databricks, I came across a behaviour that completely confused me.

Jul 28, 2026 · 4 min read · 6 claps #big-data #delta-lake #autoloader

TCH

DA DataWithRohit

A Complete Associate and Professional Exam Guide on CDC and CDF

Change Data Capture questions may appear simple, often combine several related concepts:

Jul 28, 2026 · 29 min read · 2 claps #databricks #spark #apache-spark

ART MDA TCH AI

BH Bharani · DevOps.dev

Dead Letter Queue (DLQ) Explained Simply: Why One Bad Record Shouldn’t Stop a Million Good Ones

YouTube Video URL : https://youtu.be/pOJzL6ytnfo

Jul 27, 2026 · 4 min read · 1 claps paywalled #dead-letter-queue #apache-spark #databricks

TCH

DA DataWithRohit

Databricks Lakeflow Connect: The Complete Certification Guide to Managed and Standard Connectors

Lakeflow Connect is the ingestion layer of the Databricks data engineering platform. It provides a range of options — from fully managed…

Jul 23, 2026 · 15 min read · 0 claps #databricks #azure-databricks #databricks-certification

TCH

DA DataWithRohit

Databricks Expectations Explained: A Complete Associate and Professional Certification Guide

Warn, drop, or fail? Master the data-quality scenarios that repeatedly appear in Databricks exams

Jul 22, 2026 · 15 min read · 1 claps #databricks #databricks-certification #apache-spark

TCH

DA DataWithRohit

Databricks Data Engineer Associate Exam Prep: Auto Loader and COPY INTO

A practical, scenario-based guide to file ingestion, schema evolution, notifications, overwrites, idempotency, and important certification…

Jul 21, 2026 · 17 min read · 3 claps #databricks #autoloader #data-engineering

ECO TCH

TH The Data Forge

💥Spark Memory Management — Why Your Jobs Die and How to Stop It

Your Spark Job Failed with OutOfMemoryError. Again. Here Is Exactly What Is Happening and the Specific Configurations That Fix It.

Jul 20, 2026 · 12 min read · 0 claps #apache-spark #data-engineering #performance-engineering

TCH

VI Vinícius Pereira

Your Glue Job Is Slow. Where Do You Look?

On finding the bottleneck in the half of your job that CloudWatch can’t see.

Jul 16, 2026 · 4 min read · 2 claps #aws #aws-glue #apache-spark

ART TCH

SI Siddharth Shenoy · T3CH

Spark 02: The Spark Architecture

Part 1 covered why Spark exists in the first place. But who decides what runs where? What happens when you hit “run” on your code? Who’s…

Jul 15, 2026 · 3 min read · 4 claps #apache-spark #data-engineering #big-data

TCH LIF

MI Michael Preston · CodeX

Apache Spark vs DuckDB vs Polars: Which Tool Fits Your Data Workflow?

A practical comparison based on workload, performance, and developer productivity — not hype.

Jul 13, 2026 · 5 min read · 0 claps paywalled #apache-spark #duckdb #polar

TCH SPT

HI Himadri Talukder · IBM Data Science in Practice

Running Apache Spark Jobs on IBM watsonx.data

Learn how to build, submit, monitor, and troubleshoot Spark applications using the watsonx.data Spark Engine with real world examples and…

Jul 10, 2026 · 11 min read · 2 claps #watsonx #data #watsonxdata

TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

✨Spark UI Trek: The Klingon Attack

Spark UI: The Last Frontier. A chaotic telemetry matrix where cheerful progress bars pretend everything is fine, while red error logs…

Jul 07, 2026 · 54 min read · 0 claps #apache-spark #pyspark #data-engineering

TCH

A 강은호 · Monday9pm

왕초보를 위한 Apache Spark 소개

Apache Spark는 여러 대의 분산 컴퓨터를 묶어서 거대한 데이터를 빠르게 처리하는 엔진입니다. Spark 내부의 모든 디테일은 거대한 데이터를 분산된 환경에서 빠르게 처리하는 것에 집중 되어 있습니다.

Jul 06, 2026 · 22 min read · 0 claps #apache-spark #data #software-development

TCH AI

AN Ankit Raj

How the DAG Actually Works in Apache Spark (Without the Textbook Jargon)

The thing every Spark tutorial mentions in one line and then never explains. Let us fix that.

Jul 06, 2026 · 8 min read · 55 claps #apache-spark #gda #lazy-evaluation-in-spark

ART TCH

SI Siddharth Shenoy · T3CH

Spark 01: Introduction to Spark

Apache Spark is a unified computing engine and a set of libraries for parallel data processing on computer clusters.

Jul 02, 2026 · 2 min read · 51 claps #apache-spark #big-data #data-engineering

TCH MDA MKT

NG Ángel Álvarez Pascua · Towards Data Engineering

🕵️ Data Forensics with Databricks, Zingg and GraphFrames🔍

Every dataset has a back alley. From the dashboard, everything looks clean — until production data walks in with fake names, duplicate…

Jul 01, 2026 · 25 min read · 0 claps #data-engineering #databricks #entity-resolution

TCH HUM MDA MKT AI

SA Sanila Wijesekara

How I Turned a Customer Churn Model into a Production-Grade Real-Time Churn Prediction System

Customer churn is one of the biggest challenges faced by subscription-based businesses such as telecom providers, banks, streaming…

Jun 30, 2026 · 12 min read · 55 claps #machine-learning #apache-spark #apache-airflow

TCH ECO

AB Abhishek Pan · Towards Data Engineering

Why Delta Lake MERGE Gets Slow on Wide Tables, and How I Actually Fix It

The cluster was never the problem. Write amplification on a wide table is, and throwing bigger nodes at it just burns money slower.

Jun 26, 2026 · 15 min read · 0 claps paywalled #data-engineering #big-data #databricks

SCI TCH

JU Jubin Soni · CodeToDeploy

Apache Spark Query Optimization on Databricks: Catalyst, AQE, and Photon Engine

A deep dive into how Spark transforms your SQL into a physical execution plan — and how Databricks layers Adaptive Query Execution and the…

Jun 26, 2026 · 9 min read · 50 claps #apache-spark #query-optimization #photon

SPT TCH

RO Rolly Singh

Apache Spark Mastery (Part 4 of 6): Joins & Aggregations

Where tidy data finally turns into insight — combine it, then summarize it

Jun 24, 2026 · 4 min read · 0 claps #big-data #apache-spark #apache

TCH

TH Think Data · Towards Data Engineering

Adaptive Query Execution (AQE): How Spark Re-Optimizes Queries at Runtime

Two Spark queries can return exactly the same result.

Jun 24, 2026 · 4 min read · 2 claps paywalled #data-engineering #apache-spark #spark-optimization

TCH

OM Omsurana

Slowly Changing Dimensions (SCD) and Its Types in Data Warehousing

How Modern Data Platforms Preserve the Story Behind Data

Jun 21, 2026 · 6 min read · 0 claps #data-engineering #data-warehousing #apache-spark

TCH LIF

PR Pritam Sharma

Timestamp vs TIMESTAMP_NTZ: The Big Data Time Bomb

If you work in big data, engineering timelines has probably given you a headache more than once. Time zones have a frustrating habit of…

Jun 21, 2026 · 3 min read · 0 claps #spark #snowflake #timezone

TCH

SA Saurav Singh · Towards Data Engineering

14,000 Files, 40GB: Fixing a Slow Delta Table in Azure Synapse

What OPTIMIZE, Z-ORDER, partitioning, and VACUUM actually changed

Jun 20, 2026 · 7 min read · 38 claps paywalled #delta-lake #azure-synapse #data-engineering

TCH

SA Sairaj Jadhav

Hadoop in 2026: The Elephant’s Evolution, Modern Alternatives, and Cloud Survival

For more than a decade, Hadoop was the undisputed king of Big Data. It was the open-source framework that taught the world how to store and…

Jun 19, 2026 · 6 min read · 0 claps #data-engineering #big-data #data-engineer

TCH

HA Ha-nyung Chung · Uncommon Stories

Google Cloud Next ’26, Data Infrastructure Perspectives

“AI” was the headline at Google Cloud Next ’26, but underneath the AI announcements was a sustained data infrastructure story: Google is…

Jun 19, 2026 · 6 min read · 11 claps #google-cloud #google-cloud-next #apache-iceberg

TCH SCI SOC

TH Think Data · Towards Data Engineering

Cost-Based Optimizer (CBO) in Spark: Statistics, Cardinality Estimation, and Join Selection

Two Spark queries can return exactly the same result.

Jun 18, 2026 · 6 min read · 9 claps paywalled #pyspark #spark-optimization #data-engineering

TCH

AL Allan McAleavy

Can We Predict How Many Spark Tasks an Iceberg Ingestion Will Create?

I have recently started looking at how Apache Spark(Dataproc) works with Iceberg as a data lake, with a view to better understanding it…

Jun 17, 2026 · 3 min read · 0 claps #apache-spark #dataproc #system-performance

ART TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

Apache Spark WTF??? ✈️ Come Fly With Me🧑‍✈️

Welcome to the airport. Keep your boarding pass close and your schema assumptions closer. Spark 4 just opened a new terminal for JSON, XML…

Jun 17, 2026 · 39 min read · 0 claps #apache-spark #data-engineering #data-architecture

TCH

HA Harsh Verma

How a Friend Cut a Production Spark Job’s Runtime by 35–40%

A few months ago, a data engineer I know was handed a problem that many in the field will recognise: a Spark pipeline that had quietly…

Jun 16, 2026 · 4 min read · 0 claps #apache-spark #data-engineering #python

TCH DSN MDA

RA Rahuldonta

Scalable Real-Time Data Processing Using Azure Databricks Structured Streaming

Abstract As enterprises generate data at unprecedented velocity, the demand for sub-second, scalable, and operationally lean real-time…

Jun 16, 2026 · 15 min read · 0 claps #data-engineering #azure-databricks #apache-spark

ART TCH

RA Rathanak SENG

រចនាបទ​មួយ​នៃ​ Lakehouse ដែល​ខ្ញុំ​បាន​បង្កើត​។

មូលដ្ឋាន​មួយ​ដែលអាចរក្សាទុកទិន្នន័យទាំងអស់របស់អ្នក ក្នុងតម្លៃទាប​ និងផ្តល់នូវទំនុក​ចិត្ត​​ខ្ពស់​ក្នុងការទាញយកទៅប្រើប្រាស់ (Query)។

Jun 16, 2026 · 4 min read · 0 claps #data-engineering #data-architecture #data-lakehouse

MDA TCH

AM Amogh Ubale · CodeToDeploy

Spark Lineage, DAG, Stages and Tasks — What They Mean and How to Use Them to Fix Real Problems

A production engineer’s guide to reading execution plans, navigating the Spark UI, and diagnosing the four failure patterns that slow down…

Jun 16, 2026 · 12 min read · 50 claps paywalled #data-engineering #big-data #apache-spark

TCH MDA MKT AI

NG Ángel Álvarez Pascua · Towards Data Engineering

Apache Spark WTF??? 📖 The Neverending Real-Time Story 🐉

Four data engineers sat around the campfire, wrapped in old cluster logs, whispering about the thing that had eaten their micro-batches.

Jun 16, 2026 · 45 min read · 0 claps #apache-spark #streaming #data-engineering

AI TCH

CH Chengzhi Zhao · Data Engineering Space

Stop Breaking Production Data Pipeline: Implementing Write-Audit-Publish (WAP) with Spark and…

Build Write-Audit-Publish in Data Pipeline for Downstream Sanity

Jun 15, 2026 · 4 min read · 5 claps paywalled #data #ai #data-engineering

ART TCH

HA Hanish Savla

Gold Is Not Just Aggregated Silver — Designing for Business Questions

Series: Building an Enterprise Data Platform on Databricks — Part 3 of 5

Jun 15, 2026 · 8 min read · 0 claps #data-engineering #databricks #apache-spark

TCH

AN Anurag Sharma

I Realized Spark UI Wasn’t Enough for Production Monitoring

Apache Spark is one of the most powerful technologies in modern data engineering. It can process terabytes of data, power massive ETL…

Jun 15, 2026 · 6 min read · 0 claps #data-engineering #big-data #apache-spark

TCH

SH Shashwath Shenoy

Spark on Kubernetes in Production: The Complete Guide!

Spark on YARN is dead. Spark on EMR is expensive. Spark on Kubernetes is the future!

Jun 15, 2026 · 12 min read · 60 claps paywalled #apache-spark #kubernetes #data-engineering

TCH

PR Pravash

Delta Lake vs. Iceberg vs. Hudi: I Finally Know When to Use Each One

Three open table formats. Three different jobs. One cheat sheet.

Jun 15, 2026 · 6 min read · 11 claps paywalled #data-engineering #apache-spark #big-data

TCH MDA

NG Ángel Álvarez Pascua · Towards Data Engineering

Apache Spark WTF??? 🧟 We Will RocksDB You 🧠

A stateless stream is a highway: rows pass and vanish. Add groupBy() or Spark 4’s transformWithState(), and the road loops into Stateful…

Jun 14, 2026 · 29 min read · 10 claps #apache-spark #data-engineering #rocksdb

TCH MDA

AV Avinash Thakur

NDJSON Explained: The JSON Format Built for Big Data and Streaming

If you’ve worked with APIs, log processing, Apache Spark, Kafka, Elasticsearch, or large-scale ETL pipelines, you’ve probably encountered…

Jun 14, 2026 · 4 min read · 1 claps #data-engineering #big-data #json

TCH DSN

NG Ángel Álvarez Pascua · Towards Data Engineering

Apache Spark WTF??? 📈 Nothing Breaks Like A Chart 📉

Data engineering at scale is a surrealist act: we manipulate variables that don’t exist yet across a cloud of invisible computers. But for…

Jun 13, 2026 · 25 min read · 0 claps #pyspark #apache-spark #data-engineering