Why Databrick’s Photon Makes Spark SQL Faster

A Memory & Execution Perspective

What Photon Really Is

Photon is not a Spark configuration or a library. It is a native, C++-based vectorized execution engine that replaces Spark’s JVM-based execution for SQL and DataFrame workloads.

In simple terms: Spark plans the query, Photon executes it.

Spark still handles:

• Query planning (Catalyst optimizer)
• Scheduling
• Task orchestration
• Metadata and lineage

Photon takes over:

• Joins
• Aggregations
• Filters
• Projections
• Sorts
• Delta Lake read/write execution

The Core Problem with JVM-Based Spark Execution Even with Tungsten:

• Execution happens in the JVM
• Data structures live in or near the heap
• GC must scan memory repeatedly
• Processing is still largely row-oriented

For large analytical queries, this leads to:

• GC pauses
• Higher memory pressure
• CPU underutilization
• Spills during joins and aggregations

What Photon Changes Under the Hood

1. Fully Off-Heap Native Execution

Photon executes operators in native memory, completely outside the JVM:

• No JVM heap allocation
• No garbage collection
• Deterministic memory release

This alone removes one of the biggest bottlenecks in Spark SQL.

2. Vectorized (Columnar) Processing

Instead of processing one row at a time, Photon processes batches of column values:

• Thousands of values per operation
• Fewer function calls
• Better CPU cache locality

This drastically reduces per-row overhead.

3. CPU-Level Optimizations (SIMD)

Photon uses modern CPU instructions:

• SIMD (AVX/AVX2)
• Cache-aligned memory access
• Tight native loops

As a result, a single CPU instruction can process multiple values at once, something JVM-based Spark cannot reliably achieve.

4. Native Hash Tables for Joins & Aggregations

In classic Spark:

• Join hash tables often live in JVM memory
• GC scans them repeatedly
• Pointer-heavy structures reduce performance

In Photon:

• Hash tables live entirely in native memory
• Compact, cache-friendly layouts
• No GC scanning

This makes joins and aggregations significantly faster and more memory-efficient.

5. Reduced Serialization & Shuffle Overhead

Photon keeps data in binary columnar format throughout execution:

• Less serialization/deserialization
• Fewer intermediate buffers
• More efficient shuffle pipelines

Even though shuffles still happen, they are cheaper and faster.

6. Operator Fusion

Photon fuses multiple operators (scan → filter → project → aggregate) into a single execution pipeline, reducing:

• Memory reads/writes
• Temporary buffers
• Peak memory usage

Why Queries Finish Faster

Photon improves performance in three fundamental ways:

1. Less memory overhead → lower peak usage, fewer spills
2. No GC pauses → stable and predictable execution
3. Better CPU efficiency → higher throughput per node

Even with the same cluster size, jobs complete faster because memory is used more efficiently and released sooner.

What Photon Does NOT Accelerate

Photon does not speed up:

• Python UDFs
• RDD-based code
• Non-SQL JVM logic
• Driver-side operations

Photon accelerates only SQL, DataFrame API, and Delta Lake operations.

* Typical Performance Gains

Actual gains depend on data size, skew, and IO characteristics.

Spark with Photon = JVM control plane + native execution engine

• JVM handles what to run
• Photon handles how fast it runs

This separation is what allows Photon to eliminate JVM bottlenecks while keeping Spark’s flexibility.