#spark-sql

TCH SCI SOC

TH Think Data · Towards Data Engineering

Cost-Based Optimizer (CBO) in Spark: Statistics, Cardinality Estimation, and Join Selection

Two Spark queries can return exactly the same result.

Jun 18, 2026 · 6 min read · 9 claps paywalled #pyspark #spark-optimization #data-engineering

TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

Apache Spark WTF??? 🙏 Losing My Precision 🧮

Oh life, it’s bigger. It’s bigger than DoubleType.

Jun 05, 2026 · 19 min read · 2 claps #apache-spark #pyspark #data-engineering

TCH

SH Shamen

Databricks SQL 2026: Unlocking Limitless Hierarchies, Dynamic Execution, and Spatial Scale

If you spend your days building metadata-driven pipelines or wrangling massive hierarchical datasets, you know the pain of hitting hard…

May 30, 2026 · 3 min read · 2 claps #databricks #data-engineering #spark-sql

TCH HUM

PA Padmakypu

What Surprised Me Most About Learning Databricks

I started learning Databricks expecting a complex setup with multiple configurations, but the biggest surprise for me was how…

May 28, 2026 · 1 min read · 0 claps #data-engineering #databricks #big-data

PR Pritam Sharma

Why import spark.implicits._?

This import magically injects highly useful helpers and extensions into your Scala environment. In simple terms, it acts as a translator…

May 21, 2026 · 2 min read · 1 claps #spark #scala #apache-spark

ART TCH

MJ M Jahanzaib Anwar

Building a Watermark Control Table in Microsoft Fabric Lakehouse

In the first story, we introduced the foundation for building a Medallion Lakehouse architecture in Microsoft Fabric. We covered Microsoft…

May 20, 2026 · 6 min read · 0 claps #microsoft-fabric #incremental-load #data-engineering

TCH

AD Adarshyadav

Mastering SQL & PySpark Interview Questions: Identify Top 80% Sales-Contributing Products with…

Introduction

May 17, 2026 · 2 min read · 0 claps paywalled #data-engineer #sql #spark

TCH

NG Ángel Álvarez Pascua · Towards Data Engineering

Apache Spark WTF??? 😎 Straight Outta Comparison ⚖️

They say comparison is the thief of joy. And so it is. Especially when the comparison involves two Spark DataFrames, no clearly identified…

May 12, 2026 · 15 min read · 22 claps #spark #apache-spark #spark-sql

TCH

AN António Rosa

The Three Joins AQE Can’t Save You From

When and how to override Spark’s automatic join strategy — a technical deep-dive for senior engineers

May 11, 2026 · 7 min read · 2 claps #spark #apache-spark #databricks

TCH

SA Sandesh Kutwade

DataFrames and there functions in Pyspark

basically the RDDs are your foundations but the actual work happens on dataframes.

Apr 10, 2026 · 2 min read · 1 claps #spark-sql #apache-spark #databricks-basics

RA Rahma Dwi Kartika Sari

Spark SQL

Praktikum Big Data Pertemuan 9

Oct 25, 2025 · 19 min read · 0 claps #spark-sql #apache-sparksql #anaconda-navigator

PI Pinjari Akbar

how to dynamically load tables from a source schema (where you don’t know the schema ahead of time)…

PySpark and want to dynamically load tables from a source schema (where you don’t know the schema ahead of time) into a target location…

Aug 22, 2025 · 1 min read · 0 claps #pyspark #spark #apache-spark

TCH

PI Pinjari Akbar

How to Load Unknown Schema Tables from Source to Target Using SQL

When integrating data from external systems, it’s common to encounter situations where the source schema isn’t known in advance. Instead of…

Aug 22, 2025 · 2 min read · 0 claps #sql #mysql #sql-server

ART TCH

DA DataWithRohit

Unpacking Apache Spark: Architecture, RDDs, Spark SQL & Databricks

Apache Spark is a powerful, multi-language distributed computing engine designed to simplify complex data operations. In this deep dive…

Aug 15, 2025 · 4 min read · 4 claps #apache-spark #databricks #spark-sql

TCH

SR Sriw World of Coding · Data And Beyond

Optimizing Spark Resource Allocation with spark-submit

While there are several ways to optimize Spark jobs, one of the most fundamental steps is properly configuring resource allocation. Let’s…

May 04, 2025 · 3 min read · 153 claps paywalled #spark #spark-sql #pyspark

AI

CH Chunting Wu

Benchmarking Lakehouse with TPC-DS

A step-by-step guide to importing TPC-DS data into Apache Iceberg

Feb 10, 2025 · 4 min read · 0 claps paywalled #apache-iceberg #data-lakehouse #tpc-ds

ECO

NE Nethaji Kamalapuram

Spark Memory allocation & Management

Lets talk about how memory allocation works for spark driver and executors. Understanding driver and executor memory allocation is crucial…

May 13, 2024 · 6 min read · 71 claps #spark #spark-memory #spark-driver-memory

TCH

SA Sajjad Aghapour

Custom listener on SQL parser in Spark

Sometimes, it’s required to have some customization on input SQL from end users in Apache Spark. In one of these such use cases, we were…

Oct 27, 2023 · 4 min read · 0 claps #spark #spark-sql #antlr4

TCH

TS tshihyi

SQL數據整理 by pySpark(1)

在大數據的學習裡，除了使用python的Pandas處理資料，Spark更是數據處理的好用工具，不但可以做到與Pandas相同的數據處理(包括data extraction及 data…

Aug 28, 2022 · 3 min read · 0 claps #pyspark #hive-sql #spark-sql