How a Friend Cut a Production Spark Job’s Runtime by 35–40%

A few months ago, a data engineer I know was handed a problem that many in the field will recognise: a Spark pipeline that had quietly grown from acceptable to embarrassing.

The job processed nearly a full year of historical reporting data, ran several multi-table joins, generated downstream reporting datasets, and finally wrote the output to the enterprise data warehouse. When it was first built, the runtime was fine. By the time it landed on his plate, it was nudging two hours — and burning through cluster resources the team couldn’t spare.

The instinct in situations like this is usually to throw hardware at it. Bigger executors, more cores, more money. He took a different approach: figure out why it was slow before deciding what to do about it.

Understanding the problem first

Before touching a single line of code, he opened Spark UI and spent time with it.

Three patterns appeared almost immediately.

Skewed tasks. A handful of tasks were still running long after the rest had finished. That asymmetry is the signature of data skew — certain partitions were carrying far more data than others.

Shuffle-heavy stages. The majority of execution time was concentrated in stages that involved large data movement across the cluster. These are the expensive ones: joins, aggregations, repartition calls.

Disk spill. Executors were spilling intermediate data to disk. Disk I/O is orders of magnitude slower than in-memory computation, and it tends to cascade — once one executor starts spilling, the downstream pressure compounds.

The Spark UI, at this point, had stopped being a monitoring tool and started being a roadmap.

Pipeline — before optimization

Optimization 1 — Replacing LEFT JOINs with INNER JOINs

The first place he looked was the join logic.

Several joins had been written as LEFT JOINs, even though every downstream transformation only ever used records where a match actually existed. The LEFT JOIN was producing null rows that immediately got filtered out further down the DAG — costing memory and shuffle to materialise data that would never be used.

After validating the business logic, he converted the eligible joins to INNER JOINs.

# Before
fact_df.join(dim_df, "id", "left")
# After
fact_df.join(dim_df, "id", "inner")

Less data moving through the DAG meant smaller shuffles, lower memory pressure, and faster intermediate stages.

Optimization 2 — Broadcasting small dimension tables

One of the joins was between a large reporting dataset and a small lookup table — the classic setup where Spark’s default shuffle join is the wrong tool entirely.

Without an explicit hint, Spark was redistributing data across every executor to co-locate matching keys. With a broadcast hint, the small table gets sent to every executor once, and the join happens locally.

from pyspark.sql.functions import broadcast

fact_df.join(broadcast(dim_df), "key")

This single change had a noticeable effect on the overall runtime. Eliminating a large shuffle is not incremental — it removes an entire expensive stage from the DAG.

Optimization 3 — Pushing filters earlier

The pipeline was scanning an entire year of data before any filtering happened. Not every downstream operation needed all of that.

He moved filtering logic as close to the source read as possible.

filtered_df = source_df.filter(col("reporting_date") >= start_date)

In Delta Lake and Parquet-backed sources, this also enables file pruning and column projection at the storage layer. The downstream stages simply had less to work with — smaller shuffles, lower memory consumption, faster execution throughout the DAG.

Optimization 4 — Reducing wide transformations

Wide transformations — those that require Spark to redistribute data across the cluster — are unavoidable in some cases, but unnecessary in others.

The pipeline had accumulated several of these: repeated repartition calls, sequential joins that could be restructured, transformation chains that triggered more shuffles than the logic actually required.

By reviewing the DAG in Spark UI and simplifying portions of the transformation logic, he reduced the number of shuffle boundaries substantially. Fewer stages requiring data movement meant faster end-to-end execution.

Optimization 5 — Unpersisting cached DataFrames

A subtler issue, but a real one.

Several DataFrames had been persisted earlier in the pipeline and never explicitly unpersisted. They were sitting in executor memory — taking up space that executors needed for active computation and contributing to the disk spill problem identified at the start.

cached_df.unpersist()

The result

Pipeline — after optimization

After these five changes — none of them exotic, all of them targeted at bottlenecks identified through the Spark UI — the pipeline dropped from roughly 120 minutes to 85–90 minutes.

Optimizations applied

No new hardware. No configuration tuning. Just understanding where the time was actually going.

What this taught him

A few lessons worth keeping.

Measure first. The Spark UI told him exactly where the time was being spent. He didn’t guess — he looked. That distinction matters more than it sounds.

Not all joins are equal. The type of join and whether to broadcast the smaller side can matter more than any amount of executor tuning.

Data volume is the real variable. Every optimisation here was, at its root, about reducing the amount of data that stages had to process. Filter early. Join on less. Move less across the network.

Cache deliberately. Caching is powerful, but caching everything — or caching and forgetting — is a liability. Memory held by stale caches is memory unavailable to active computation.

The pipeline is still running. It’s not the slowest job in the workflow anymore. And the cluster cost stayed flat.

Sometimes the best performance work looks a lot less like engineering and a lot more like paying attention.