Photon vs Tungsten: Understanding Databricks’ Next-Generation Spark Execution Engine

Introduction

If you’ve been working with Apache Spark or Databricks, you’ve likely heard statements like:

Photon is much faster than Tungsten. Photon replaces Tungsten. Photon makes Spark 2–5x faster.

While these statements are common, they often leave an important question unanswered:

What actually changes inside Spark when Photon is enabled?

Many engineers know that Photon improves performance, but few understand why. Is it a new query optimizer? Does it replace Catalyst? Does it eliminate the JVM? Or is it simply another optimization layer on top of Spark?

To answer these questions, we first need to understand how Spark executes a SQL query.

Every Spark SQL query follows a well-defined execution pipeline. The SQL statement is parsed, optimized by the Catalyst Optimizer, converted into an execution plan, and finally executed by an execution engine.

For years, that execution engine was Tungsten, a major innovation introduced to overcome JVM limitations by optimizing memory management, CPU utilization, and code generation. Tungsten dramatically improved Spark’s performance and became the foundation of modern Spark execution.

As data volumes continued to grow and enterprises demanded even faster analytics, Databricks introduced Photon — a native, vectorized execution engine written in C++ that works alongside Spark to execute SQL and DataFrame workloads more efficiently. Rather than changing how queries are planned, Photon changes how they are executed, taking advantage of modern CPU architectures, SIMD instructions, and native memory management to reduce execution time.

In this article, we’ll explore the internal architecture of both Tungsten and Photon, understand how each executes a Spark query, compare their execution models, and explain why Photon can significantly outperform the traditional Spark execution engine in many real-world workloads.

By the end of this article, you’ll understand:

• How Spark SQL queries are executed internally.
• The role of the Catalyst Optimizer.
• Why Tungsten was introduced and the problems it solved.
• How Photon differs from Tungsten.
• How Photon leverages native C++ execution and vectorized processing.
• When Photon provides significant performance improvements — and when it does not.

Let’s start by understanding the complete Spark SQL execution pipeline before diving into the internals of Tungsten and Photon.

Why Understanding the Execution Pipeline Matters

Before comparing Photon and Tungsten, it’s important to understand how Apache Spark executes a SQL query internally.

When we write a SQL query in Databricks, Spark doesn’t execute it immediately. Instead, the query passes through multiple stages, where it is parsed, validated, optimized, and transformed into an executable plan.

Only after these steps does Spark hand the query to an execution engine such as Tungsten or Photon.

Understanding this pipeline is essential because Photon does not replace the entire Spark engine. Instead, it replaces only the execution layer, while the earlier planning and optimization stages remain unchanged.

The Complete Spark SQL Execution Pipeline

Every Spark SQL query follows the same high-level execution flow:

Each stage has a specific responsibility.

Step 1: SQL Parser Consider the following SQL query:

SELECT customer_id,
       SUM(loan_amount)
FROM loan
WHERE branch='Pune'
GROUP BY customer_id;

The SQL Parser performs the first level of processing.

Its responsibilities include:

• Checking SQL syntax.
• Identifying SQL keywords.
• Recognizing tables, columns, operators, and functions.
• Converting the SQL statement into an Unresolved Logical Plan.

At this stage, Spark does not know whether the table or columns actually exist. The parser simply understands the structure of the query.

Step 2: Analyzer Next, Spark validates the query.

The Analyzer:

• Confirms that the loan table exists.
• Verifies that **customer_id, `loan_amount**, andbranch` exist.
• Resolves aliases.
• Determines data types.
• Validates aggregation rules.

If a column does not exist, this is the stage where Spark throws an error. After successful validation, Spark produces a Resolved Logical Plan.

Step 3: Catalyst Optimizer This is the brain of Spark SQL.

Catalyst applies dozens of optimization rules before execution.

Some common optimizations include:

• Predicate Pushdown
• Constant Folding
• Projection Pruning
• Filter Reordering
• Join Reordering
• Expression Simplification
• Null Propagation

For example, instead of reading the entire table and then filtering:

SELECT *
FROM loan
WHERE branch='Pune';

Catalyst pushes the filter as close to the data source as possible, reducing unnecessary I/O. This produces the Optimized Logical Plan.

Step 4: Physical Planner

Spark now generates one or more physical execution strategies.

For example, if two tables are joined, Spark may choose:

• Broadcast Hash Join
• Sort Merge Join
• Shuffle Hash Join

The decision depends on:

• Table size
• Statistics
• Configuration
• Adaptive Query Execution (AQE)

The result is the Physical Plan.

Step 5: Execution Engine

Finally, Spark executes the Physical Plan.

This is where Tungsten or Photon comes into the picture.

The optimizer has already decided what should be executed.

The execution engine decides how to execute it efficiently.

• Open-source Apache Spark uses Tungsten.
• Databricks can use Photon for supported SQL and DataFrame workloads.

This distinction is the core of the Photon vs Tungsten discussion.

The Spark SQL execution pipeline remains the same regardless of whether Photon is enabled.

Key Points

Both execution engines rely on the same parser, analyzer, Catalyst optimizer, and physical planner.

The only difference lies in the execution layer:

• Tungsten executes the physical plan using optimized JVM-based techniques.
• Photon executes the physical plan using a native C++ vectorized engine designed for modern CPUs.

This is why Photon is often described as an execution engine, not a replacement for Spark itself.

What is Tungsten?

The Birth of Tungsten

When Apache Spark was first introduced, it provided an easy-to-use distributed computing framework built on the Java Virtual Machine (JVM). Although Spark was significantly faster than Hadoop MapReduce, engineers soon discovered that many workloads were not limited by distributed computing — they were limited by how efficiently Spark used CPU and memory.

As datasets grew from gigabytes to terabytes, Spark spent a considerable amount of time:

• Creating millions of Java objects.
• Running Java Garbage Collection (GC).
• Moving data between JVM objects.
• Executing row-by-row operations.
• Performing inefficient memory access.

These JVM overheads prevented Spark from fully utilizing modern multi-core CPUs.

To solve these problems, the Spark community introduced Project Tungsten in Spark 1.4.

The goal was simple:

Bring Spark execution closer to the hardware by making better use of CPU caches, memory, and processor instructions.

Rather than redesigning Spark’s APIs, Tungsten redesigned how Spark executes queries internally.

Why Was Tungsten Needed?

Imagine processing a table containing 1 billion loan records.

Without Tungsten, Spark would:

• Create a Java object for each row.
• Store each object separately in JVM memory.
• Continuously allocate and deallocate objects.
• Trigger frequent Garbage Collection.

Instead of spending CPU cycles processing data, the JVM spent a significant amount of time managing memory.

This resulted in:

• Higher memory usage
• Frequent GC pauses
• Poor CPU utilization
• Slower query execution

Tungsten was introduced to eliminate these bottlenecks.

What Problems Does Tungsten Solve?

Tungsten improves Spark execution through four major innovations.

1. Off-Heap Memory Management

Instead of storing data as millions of Java objects inside the JVM heap, Tungsten stores data in off-heap memory.

Benefits:

• Less Garbage Collection
• Lower memory overhead
• Faster memory access
• Better CPU cache utilization

Instead of this:

Spark now manages memory directly.

2. Binary Memory Format

Traditional JVM objects contain:

• Object headers
• References
• Metadata

All of these consume additional memory. Tungsten stores rows in a compact binary format.

Example: Instead of

Customer Object

ID
Name
Loan
Branch

Spark stores

010101100101011...

This compact representation:

• Uses less memory
• Improves cache locality
• Reduces serialization overhead

3. Whole-Stage Code Generation

Before Tungsten, Spark executed many small iterator-based functions. Tungsten combines multiple operators into one optimized Java function. Spark generates Java bytecode at runtime.

This significantly reduces:

• Virtual function calls
• Object creation
• Intermediate data structures

The CPU executes one optimized pipeline instead of many smaller operations.

4. Cache-Aware Algorithms

Modern CPUs are extremely fast. Memory is comparatively slow.

Tungsten optimizes execution so that frequently accessed data remains in CPU cache whenever possible. This reduces memory latency and allows Spark to process data more efficiently.

How Tungsten Executes a Query

Suppose we execute:

SELECT customer_id,
       SUM(loan_amount)
FROM loan
GROUP BY customer_id;

Internally, Tungsten performs the following steps:

Notice that Spark is still running on the JVM. Tungsten simply makes JVM execution far more efficient.

Benefits of Tungsten

Tungsten dramatically improved Spark by:

• Reducing Garbage Collection overhead
• Improving CPU utilization
• Using compact binary memory representation
• Minimizing object creation
• Generating optimized Java code
• Increasing query execution speed
• Reducing memory consumption

These innovations became the foundation of modern Spark SQL execution.

Limitations of Tungsten

Despite its improvements, Tungsten still has some limitations.

It continues to rely on the JVM, which introduces unavoidable overhead.

For example:

• Generated Java code must still execute inside the JVM.
• Java object model constraints still exist.
• SIMD instructions are used only in limited scenarios.
• Some CPU optimizations available to native applications cannot be fully exploited.

As hardware evolved, Databricks recognized that there was still room for significant performance improvements.

Instead of optimizing JVM execution further, they took a different approach:

What if Spark could execute the physical plan using native C++ instead of Java?

That idea led to the development of Photon.

What is Photon?

Why Did Databricks Build Photon?

Project Tungsten transformed Apache Spark by optimizing execution within the Java Virtual Machine (JVM). It introduced off-heap memory management, binary data representation, and whole-stage code generation, enabling Spark to process data much more efficiently than before.

However, even with these optimizations, Spark still depended on the JVM for executing SQL queries. While the JVM is highly optimized, it introduces unavoidable overhead such as runtime interpretation, object management, and limitations in exploiting modern CPU features.

As cloud data warehouses became increasingly popular, organizations expected analytical queries to complete in seconds rather than minutes. Databricks needed an execution engine capable of delivering database-like performance while remaining fully compatible with Apache Spark APIs.

This led to the development of Photon, a next-generation execution engine introduced by Databricks.

Unlike Tungsten, Photon is not a new query optimizer. It is a native execution engine written in C++ that executes Spark SQL and DataFrame workloads more efficiently by leveraging modern processor architectures.

What Exactly is Photon?

Photon is Databricks’ native vectorized query execution engine.

It executes the same physical plan produced by Spark’s Catalyst Optimizer but replaces the traditional JVM execution layer with highly optimized native C++ code. This means developers continue writing the same Spark SQL or DataFrame code:

SELECT customer_id,
       SUM(loan_amount)
FROM loan
GROUP BY customer_id;

The application code remains unchanged. The difference lies entirely in how the physical plan is executed.

Spark With Photon

Notice something important:

The parser… The analyzer… The Catalyst Optimizer… The physical planner… are exactly the same.

Only the execution engine changes. This is one of the most common Databricks interview questions.

How Photon Executes Queries

Photon is designed around three fundamental ideas.

1. Native C++ Execution

Instead of generating Java bytecode like Tungsten, Photon executes operators using highly optimized native C++.

Benefits include:

• Lower execution overhead
• Better CPU instruction utilization
• Reduced JVM dependency
• Faster operator execution

Because C++ is compiled directly into machine code, the processor executes instructions more efficiently than JVM-generated code in many analytical workloads.

2. Vectorized Processing

Traditional execution often processes data one row at a time. Example:

Row 1

↓

Process

↓

Row 2

↓

Process

↓

Row 3

Photon processes data in columnar batches. Instead of one value: Photon Processes many values simultaneously reduces CPU overhead and improves throughput.

3. SIMD Instructions

Modern processors support Single Instruction Multiple Data (SIMD). Instead of executing:

1000 + Tax

↓

2000 + Tax

↓

3000 + Tax

one row at a time,

Photon performs the operation on multiple values with a single CPU instruction.

This dramatically accelerates operations such as:

• Aggregations
• Filters
• Arithmetic
• Comparisons
• Hash calculations

4. Native Memory Management

Photon manages memory directly rather than relying on JVM-managed objects.

Benefits include:

• Better cache locality
• Lower allocation overhead
• Reduced memory fragmentation
• Higher throughput

Combined with vectorized execution, this allows Photon to fully exploit modern CPU architectures.

Supported Workloads

Photon provides significant performance improvements for workloads such as:

• SQL Queries
• DataFrame APIs
• Aggregations
• Hash Joins
• Sort Merge Joins
• Window Functions
• Delta Lake Operations
• ETL Pipelines
• BI Dashboards

Since Photon operates below the Spark API layer, users generally do not need to modify their application code.

Benefits of Photon

Compared with traditional Spark execution, Photon offers:

• Native C++ execution
• Advanced vectorized processing
• SIMD acceleration
• Lower CPU overhead
• Faster joins
• Faster aggregations
• Faster scans
• Improved cache utilization
• Reduced execution time for SQL and DataFrame workloads

Does Photon Replace Spark?

No. This is one of the biggest misconceptions.

Photon does not replace Spark. Spark still performs:

• SQL Parsing
• Analysis
• Catalyst Optimization
• Physical Planning
• Scheduling
• Task Distribution

Photon simply replaces the execution layer for supported operators.

Think of it like this: Catalyst decides what to execute. Photon decides how to execute it efficiently.

Photon vs Tungsten — A Deep Dive into Spark’s Execution Engines

At first glance, Photon and Tungsten appear to solve the same problem: executing Spark SQL queries efficiently. Both work with the physical execution plan generated by Spark, and both aim to reduce query execution time.

However, the way they achieve this is fundamentally different.

Tungsten focuses on optimizing execution within the JVM, while Photon goes a step further by moving execution outside the JVM into a native C++ engine.

The following sections compare each aspect of the two execution engines.

1. Execution Model

Tungsten

After Catalyst produces the physical plan, Tungsten generates optimized Java bytecode using Whole-Stage Code Generation.

That generated code runs inside the JVM.

Photon

Photon skips Java code generation.

Instead, the physical plan is executed directly by highly optimized native C++ operators.

Key Difference

Tungsten optimizes JVM execution. Photon minimizes JVM involvement by executing operators natively.

2. Memory Management

Memory management is one of the biggest differences.

Tungsten

Uses:

• Off-heap memory
• Binary row format

But execution still depends on JVM infrastructure.

Photon

Photon manages memory directly. There is no dependency on Java object allocation during execution.

Advantages:

• Better cache locality
• Lower allocation overhead
• Better NUMA awareness
• Reduced memory fragmentation

3. Code Generation

Tungsten

Whole-Stage Code Generation creates Java source code.

Java Compiler

↓

Bytecode

↓

JVM

↓

CPU

Photon

No Java code generation. Instead:

Native C++

↓

Compiled Machine Code

↓

CPU

This removes an entire execution layer.

4. Data Processing Model

Tungsten

Processes rows efficiently. Although Spark performs some vectorized operations, Tungsten’s execution model is largely row-oriented.

Example:

Customer 1

↓

Process

↓

Customer 2

↓

Process

↓

Customer 3

Photon

Processes large batches of columnar data.

Customer IDs

101
102
103
104
105
106
...

↓

Single Vectorized Operation

The CPU performs work on hundreds or thousands of values simultaneously.

5. SIMD Utilization

Modern CPUs contain SIMD instructions.

Tungsten Limited SIMD usage through JVM optimizations.

Photon Extensive SIMD usage.

Instead of

100

↓

Process

↓

200

↓

Process

↓

300

Photon performs

100
200
300
400
500

↓

Single CPU Instruction

This greatly improves:

• Filters
• Aggregations
• Arithmetic
• Comparisons

6. CPU Cache Utilization

CPU cache is much faster than RAM. Photon is specifically designed to maximize cache efficiency.

Instead of repeatedly reading scattered rows, Photon stores values contiguously.

Result: Fewer cache misses. Higher throughput.

7. Join Performance

Consider:

SELECT *
FROM customer c
JOIN loan l
ON c.customer_id=l.customer_id;

Tungsten

• Optimized hash joins
• Efficient shuffle
• Whole-stage code generation

Photon

• Native hash tables
• Vectorized joins
• Better cache locality
• Faster probing

Large joins often benefit significantly from Photon.

8. Aggregation Performance

Example:

SELECT
branch,
SUM(amount)
FROM loan
GROUP BY branch;

Photon executes vectorized aggregation algorithms. Instead of updating one row at a time, multiple rows are aggregated together.

9. SQL Compatibility

The good news: No code changes. This query:

SELECT *
FROM loan
WHERE amount>500000;

works identically. Photon is completely transparent.

10. Performance

Performance depends on workload. Typical gains are often observed in:

• SQL Analytics
• BI dashboards
• ETL pipelines
• Delta Lake workloads
• Large joins
• Aggregations

Workloads with unsupported operators or heavy Python UDFs may see smaller gains because execution can fall back to the standard Spark engine.

Real-World Example

Imagine an NBFC loan table with 5 billion rows.

A business analyst runs:

SELECT branch,
       SUM(loan_amount)
FROM loan
WHERE disbursal_date >= '2026-01-01'
GROUP BY branch;

With Tungsten:

• Catalyst creates the physical plan.
• Whole-stage code generation produces optimized Java bytecode.
• The JVM executes operators using off-heap memory and binary row format.

With Photon:

• Catalyst creates the same physical plan.
• Photon executes the operators as native C++ code.
• Data is processed in columnar batches.
• SIMD instructions accelerate filtering and aggregation.
• Fewer CPU cycles are spent per row, leading to faster completion for supported workloads.

Conclusion

Photon and Tungsten are often compared as competing technologies, but they actually represent two generations of Spark’s execution engine evolution.

Tungsten revolutionized Apache Spark by introducing off-heap memory management, binary data representation, and whole-stage code generation, enabling Spark to utilize modern hardware far more efficiently than its early versions.

Photon builds on this foundation by taking execution a step further. Instead of relying on JVM-generated code, it executes Spark SQL and DataFrame workloads using a native C++ vectorized engine optimized for modern CPUs. By leveraging vectorized processing, SIMD instructions, and native memory management, Photon significantly reduces CPU overhead and accelerates analytical workloads without requiring any changes to existing Spark applications.

The most important point to remember is that Photon does not replace Spark, Catalyst, or Tungsten’s planning process. Spark still parses, analyzes, and optimizes every query exactly as before. Photon simply replaces the execution layer for supported operators, allowing the same optimized physical plan to run more efficiently.

As a data engineer, understanding this architecture helps you make better design decisions, interpret query performance more effectively, and explain why the same Spark SQL query can execute much faster in Databricks than in open-source Spark.

Ultimately, Photon isn’t about writing different SQL — it’s about enabling the same SQL to execute smarter, faster, and more efficiently on modern hardware.

If you found this article helpful, follow me on Medium for more deep dives into Databricks, Apache Spark, Delta Lake, Azure Data Engineering, and real-world data engineering architectures.