Apache Spark Query Execution Plan Explained: Code to Cluster How Spark Executes Queries

Before jumping into the execution flow, let’s understand how a Spark application starts.

Spark applications can be run interactively through notebooks or as standalone applications using spark-submit. Regardless of how the application is started, Spark supports two deployment modes: Client mode and Cluster mode.

In Client mode, the driver process runs on the machine where the application is submitted (for example, your local machine or the machine hosting the notebook).

In Cluster mode, the driver does not run on the submission machine. Instead, when the application is submitted using spark-submit, the cluster manager (such as Spark Standalone, YARN, or Kubernetes) launches the driver on one of the worker nodes in the cluster.

Once the driver starts, it creates the Spark context and coordinates the entire execution. It communicates with the cluster manager to request resources, and the cluster manager launches executor processes on worker nodes. These executors are responsible for executing the actual tasks in parallel.

So the overall flow in cluster mode looks like this:

Client Laptop (spark-submit)
        ↓
Spark Master (Cluster Manager)
        ↓
Driver starts inside Worker Node
        ↓
Driver requests resources
        ↓
Executors launched on Worker Nodes
        ↓
Task execution begins

When you write Spark code in your notebook like this,

-- DataFrame
df.filter(df.age > 25).groupBy("state").count().show()

-- SQL
SELECT state, COUNT(*) AS count
FROM table_df
WHERE age > 25
GROUP BY state;

the flow starts on our local machine where the Python application is submitted. The local Python process packages the application and submits it to the Spark Master (Cluster Manager). The Master then allocates this to Driver node inside the cluster.

Since this is a PySpark application, a Python process and a Driver JVM are started on the Driver node. A Py4J bridge is then established between the Python process and the Driver JVM, allowing Python DataFrame APIs to invoke Spark’s JVM APIs.

Once this communication layer is established, the Driver begins building the execution flow which contain multiple steps. Inside the JVM, Spark doesn’t immediately execute anything. Instead it builds plan.

Now let’s go step by step through the Spark execution process.

Step 1: Unresolved Logical Plan

Spark first converts the DataFrame or SQL query into an Unresolved Logical Plan. Think of this as Spark creating a rough blueprint of the query.

At this stage, Spark only understands what operations you want to perform, such as Filter, Project, Aggregate, or Join.

In other words, Spark understands what we want to do, but it has not validated anything yet whether tables, columns, or functions actually exist.

Example:

SELECT state, COUNT(*)
FROM table_df
WHERE age > 25
GROUP BY state;

At this point, table_df, state, and age are simply names referenced by the query.

Step 2: Analyzer (Catalog Validation)

Next, Spark passes this blueprint to the Analyzer. The Analyzer consults Spark’s Catalog, which stores metadata about tables, columns, databases, and data sources.

The Catalog contains metadata such as: Table names , Column names , Database information , Data source information..

The Analyzer verifies that every referenced object exists and resolves data types.

Step 3: Resolved Logical Plan

Once validation succeeds, Spark creates a Resolved Logical Plan.

At this point, Spark has a complete understanding of the query because all tables, columns, and data types have been verified. Spark now knows exactly what needs to be executed. It fully understands the query semantics.

Step 4: Optimized Logical Plan

Now the Catalyst Optimizer takes over. The Catalyst Optimizer now applies optimization rules to improve performance.

Spark starts looking for ways to execute the query more efficiently without changing the final result. It may move filters closer to the data source, remove unnecessary columns, or combine operations together to reduce the amount of data being processed.

Common optimizations include:

• Filter Pushdown
• Column Pruning
• Constant Folding
• Predicate Simplification
• Combining adjacent operations

The goal here is simple: perform less work and make the query faster.

Step 5: Physical Plan

After optimization, Spark decides how the query will actually run.

There are often multiple ways to execute the same query. For example, if a query contains a JOIN, Spark can choose different join strategies depending on the amount of data involved. Spark generates all possible execution plans.

Step 6: Cost Modeling

During physical planning, Spark may compare alternative execution strategies. When Cost-Based Optimization (CBO) is enabled and table statistics are available, Spark estimates the cost of these alternatives based on factors such as data size, shuffle operations, and resource usage, then selects an efficient physical plan.

Step 7: Whole Stage Code Generation

Once Spark has selected the best physical plan, it starts preparing executable code. Instead of executing every operation separately, Spark combines multiple operators into a single optimized block of Java code, which is compiled at runtime into JVM bytecode for execution.

For example, reading data, filtering rows, and selecting columns may all be merged into one execution pipeline. This is one of the reasons why Spark is so fast.

Step 8: Building the RDD Execution Graph

Internally, Spark converts this plan into an RDD-based execution graph. Every piece of work is divided into partitions, and Spark tracks the dependencies between those partitions.

This work is done by the Driver. Every operator from the Physical Plan is translated into one or more RDDs connected through dependencies.

Each RDD knows:

• How many partitions it contains
• What function should run on each partition
• Which parent RDD it depends on

At this stage, Spark still has not executed anything. It is simply building the execution graph that will later be distributed across the cluster.

Step 9: Creating Stages and Tasks

Spark then divides the work into stages. The DAG Scheduler inside the Driver analyzes the RDD graph and looks for shuffle boundaries.

A stage continues as long as data can be processed without moving across machines. Whenever Spark encounters an operation that requires data exchange, such as groupBy, join, or repartition, a new stage is created.

Each stage is then split into smaller tasks, and usually one task is created for each partition. For example, if a stage contains 200 partitions, Spark will create 200 tasks.

Each task contains:

• The partition it needs to process
• The function that must be executed
• References to the generated execution code

The Driver then sends these tasks to Executors.

Step 10: Executors Start Processing

Finally, the Driver sends these tasks to Executors running on worker nodes. Executors are JVM processes running on worker nodes.

Each executor loads its assigned partition, executes the generated Java bytecode, processes the data, and either writes intermediate shuffle data or returns the final result.

The Driver remains the brain of the application and continuously coordinates all Executors until the job finishes. At the end, the Driver collects the results and returns them to the user.

Driver:

DataFrame -> Logical Plan -> Physical Plan -> RDD DAG -> Stage -> Generate Bytecode -> Split tasks

Executor:

Receive task -> Load partition -> Run bytecode -> Emit rows

What is Whole Stage CodeGen?

Whole stage code gen, the driver generates Java code for each stage.

Whole Stage CodeGen takes that physical plan and generates Java bytecode directly not Java source code that a human can read, but binary bytecode that the JVM can run immediately. This is what makes Spark fast because it fuses multiple operations into a single compiled function eliminating per-row overhead.

Whole-stage code generation happens on the DRIVER, not executer side.

First: what does “stage” mean in Spark? (this is the key)

In Spark a stage is NOT a single operator. A stage is A sequence of operators that can run without a shuffle (i.e. in one pipeline).

Example: Scan → Filter → Project → Aggregate

As long as there is no shuffle, Spark treats this as one stage. The moment you hit: groupBy, join, repartition, sort (global). Spark ends the stage and starts a new one. what Spark used to do (old model)

What Spark used to do (old model)

Inside one stage, Spark used to execute operators like this:

Stage: can → Filter → Project → Aggregate

Each operator:

• Had its own class
• Had its own exec() / next() method
• Passed rows one-by-one

So even within the same stage, execution was fragmented.

What Whole-Stage Codegen actually means

Spark generates ONE piece of Java code for the ENTIRE STAGE, not per operator.

So instead of:

Scan.exec()
↓
Filter.exec()
↓
Project.exec()
↓
Aggregate.exec()

You get: ONE generated Java function, ONE loop, ONE control flow

That’s why it’s called: Whole Stage (entire stage) and Code Generation (runtime Java code)

Why it is NOT called “operator codegen”

Because Spark is not generating code for:

• Filter alone
• Project alone

It’s generating code for: Scan + Filter + Project + Aggregate as one unit. That’s the “whole” part.

Mapping the name to the behavior (line by line)

# For example:
while (input.hasNext()) {
    InternalRow row = input.next();
    int a = row.getInt(0);
    if (a > 10) {
        int b = row.getInt(1);
        outputRow.setInt(0, a);
        outputRow.setInt(1, b);
        append(outputRow);
    }
}

This code: Scans , Filters , Projects , Emits output

All in: One loop , One function , One stage

So Spark:

• Didn’t generate code for Filter
• Didn’t generate code for Project

Generated code for the whole stage

What happens to the flow when a Python UDF is introduced

Now when you introduce a Python UDF, the whole story changes. The JVM has absolutely no understanding of your Python function. So when you define a UDF, PySpark immediately pickles your Python function meaning it serializes it into raw bytes using Python’s pickle library. These bytes get stored inside a special PythonUDF node in the logical plan tree. The JVM just carries these bytes around like a black box it has no idea what the function does, what logic is inside it, nothing. It just knows the input column, the return type, and the serialized bytes.

When execution reaches the executor, this is where things get expensive. The executor’s JVM side reads the PythonUDF node, extracts those pickled bytes, and launches a separate Python worker process. The rows of data then travel from the JVM to the Python process through a pipe, the Python process unpickles your function and runs it on each row, then sends the results back through the same pipe to the JVM, which then continues the rest of the execution plan. This back and forth serialization across the pipe for every batch of rows is exactly why Python UDFs are significantly slower than native Spark operations — because data has to leave the JVM, cross into Python, and come back every single time.

So in the entire flow your Python code is just a remote control, the JVM builds and optimizes the plan, CodeGen compiles it to bytecode, and Python UDFs are just pickled bytes being carried along until an executor finally wakes up a Python process to run them.

Resources:

blog1

Closing Thoughts

When we write a simple DataFrame or SQL query in Spark, a lot more happens behind the scenes than we realize. Spark does not execute our code line by line; instead, it transforms it through multiple stages such as logical planning, optimization, physical planning, code generation, and finally distributed execution on executors. Understanding this internal flow not only helps us debug and optimize Spark applications, but also builds a much stronger intuition for writing efficient Spark code.