Spark, Trino, Polars and DuckDB on 157 GB of TPC-H data: What 22 queries actually reveals

A hands-on benchmark of four query engines on the TPC-H SF400 dataset -one where a single node doesn’t just tie an 80-vCPU cluster, it beats it outright.

Spark, Trino, Polars and DuckDB on 157 GB of TPC-H data

TL;DR

I ran the 22 TPC-H queries against the SF400 dataset (~157 GB of Parquet, with a 102 GB lineitem table, 2.4 B rows in 20927 row groups) across four engines and two very different kinds of machine. Free article here

Five-nodes EMR cluster:

• Spark on EMR finished all 22 queries in 12.2 minutes on an 80-vCPU cluster.
• Trino on EMR, on the same cluster and same data, took 27.8 minutes, Spark was ~2.3× faster.

On a single Linux ec2 node:

• DuckDB finished all 22 queries in 3.3 minutes on a 32-vCPU box and 6.8 minutes on a 16-vCPU box;
• Polars-streaming did it in 5.0 minutes. A single node -even one with the same specs as one of the cluster’s five machines -beat the entire 80-vCPU cluster.

The headline isn’t “engine X wins.” It’s that hardware, memory, and execution mode are inseparable from the engine at this scale, and that for 157-odd GB of data, one well-fed Linux node is faster and cheaper than a cluster. Original SF1000 benchmark by Polars is here https://pola.rs/posts/polars-pyspark-benchmarks/

The setup

Dataset

TPC-H at scale factor 400, about 157 GB of Parquet, dominated by lineitem table at 102 GB. This is the interesting regime: too big to shrug off, small enough that a beefy single node can still hold its own.

The machines:

|          | Cluster                               | Linux single node                                                         |
|----------|---------------------------------------|---------------------------------------------------------------------------|
| Engines  | Spark, Trino                          | Polars, DuckDB                                                            |
| Hardware | 5× `r6id.4xlarge` (80 vCPU / 640 GiB) | `r6id.8xlarge` (32 vCPU / 256 GiB) and `r6id.4xlarge` (16 vCPU / 128 GiB) |
| Storage  | Parquet in HDFS on local NVMe         | Parquet on local NVMe                                                     |
| Platform | EMR 7.13 (Linux)                      | Ubuntu 24.04

Note the second column: the 16-vCPU r6id.4xlarge is exactly one node of the five-node cluster , the cleanest possible “1 node vs 5 nodes, same silicon” test.

Method

Every query is a single, cold, timed pass with no warm-up. That choice matters: warming a query primes the JIT and the page cache and flatters the second run. For a fair cross-engine comparison, cold single-pass is the honest number.

For the cluster engines, the data was copied from S3 into HDFS with s3-dist-cp and replicated to every data node (setrep) so scans are balanced across the cluster instead of hammering one node. On the single nodes, the SF400 Parquet was generated straight onto local NVMe with tpchgen-cli

Round 1: Spark vs Trino, same cluster, same data

This is the cleanest comparison in the whole study : identical 80-vCPU cluster, identical HDFS data, identical cold single-pass method. The only variable is the engine.

|                   |              Spark |               Trino |
|-------------------|-------------------:|--------------------:|
| Total, 22 queries | 732.8 s (12.2 min) | 1667.3 s (27.8 min) |
| Relative          |               1.0× |         2.3× slower |

Spark won nearly every query, and it won the expensive ones decisively:

|                                                                 Query | Spark (s) | Trino (s) | Spark advantage |
|----------------------------------------------------------------------:|----------:|----------:|:----------------|
|                                           q18 (large-volume customer) |      40.5 |     170.3 | 4.2×            |
|                                            q17 (small-quantity order) |      47.7 |     136.4 | 2.9×            |
|                                                q6  (forecast revenue) |      20.8 |      61.2 | 2.9×            |
| q21 (suppliers who kept orders waiting - triple `lineitem` self-join) |      74.5 |     209.8 | 2.8×            |
|                                                q14 (promotion effect) |      23.9 |      61.6 | 2.6×            |
|                                              q19 (discounted revenue) |      25.1 |      62.2 | 2.5×            |

The only near-ties were the tiny queries: q2, q13, q16, q22, where fixed startup overhead dominates and the engines land within ~10% of each other. Why does Spark pull ahead? On these wide r6id nodes with plenty of RAM per core, Spark’s whole-stage code generation and codegen’d hash joins turn the big join/aggregate pipelines into tight, vectorized loops. Trino is a superb *interactive *engine, but for this batch-style, scan-and-join-heavy workload on a small dedicated cluster, Spark’s execution model simply did more work per second.

Round 2: a single Linux node beats the cluster

Now the surprise. I ran Polars and DuckDB on single Linux boxes with fast local NVMe: a 32-vCPU / 256 GiB r6id.8xlarge, and a 16-vCPU / 128 GiB r6id.4xlarge (one cluster node’s worth of hardware). Same cold single-pass, all 22 queries. The single node wins -decisively.

| Engine / mode      | Hardware            | Queries             |      Total |
|--------------------|---------------------|---------------------|-----------:|
| DuckDB             | 32 vCPU / 256 GiB   | 22/22               |    3.3 min |
| DuckDB             | 16 vCPU / 128 GiB   | 22/22               |    6.8 min |
| Polars (streaming) | 32 vCPU / 256 GiB   | 22/22               |    5.0 min |
| Polars (streaming) | 16 vCPU / 128 GiB   | 20/22 (q5, q18 OOM) |    8.8 min |
| *Spark (cluster)*  | *80 vCPU / 5 nodes* | *22/22*             |  12.2 min  |
| *Trino (cluster)*  | *80 vCPU / 5 nodes* | *22/22*             |  27.8 min  |

Read that again: DuckDB on a single 16-vCPU box (6.8 min) beat the 80-vCPU, five-node Spark cluster (12.2 min) by ~1.8× on the cluster’s own per-node hardware, one machine against five. On the 32-vCPU box, DuckDB was ~3.7× faster than Spark and Polars-streaming ~2.4× faster. At SF400 the whole dataset fits one big box’s NVMe and RAM, and a columnar engine skips every ounce of distributed-execution overhead. Two things the single-node runs pinned down about the ceiling:

• DuckDB is the robust engine. It finishes all 22 at both box sizes, never OOMs (it spills to disk cleanly), and halving the RAM+cores merely doubled the time (199 → 411 s) -textbook linear scaling.
• Polars streaming has a memory floor between 128 and 256 GiB at SF400. It sweeps all 22 with 256 GiB but OOMs two queries (q5, q18) at 128 GiB. Its in-memory mode is a dead end at this scale, it OOMs on q1 and q9 even with 256 GiB (q9 wanted ~257 GB of RAM), and where it survives it’s 2–7× slower than streaming. Use DuckDB anywhere, or Polars streaming on ≥256 GiB

The real lesson: engine, hardware, and memory don’t separate

What survived contact with 157 GB of Parquet:

1. Spark is the batch workhorse among cluster engines -its codegen’d execution beat Trino by 2.3× on identical hardware and data.

2. You probably don’t need a cluster. A single Linux node, even one the size of a single cluster node, running DuckDB or Polars-streaming beat the whole 80-vCPU cluster. For a lot of “big data” that isn’t actually that big, one fat node is the boring, cheap, correct answer.

3. Memory and execution mode are destiny at this scale. Polars OOMs without streaming and OOMs two queries at 128 GiB; DuckDB spills and sails through. The engine you pick matters less than whether it fits the machine and how it behaves when it almost doesn’t.

Caveats (read these before quoting a number)

• Cross-machine comparisons mix hardware and engine. The Spark-vs-Trino comparison is apples-to-apples; the single-node-vs-cluster comparison is systems, not engines in isolation , but that’s the real-world decision you actually face.
• Cold, single-pass timings throughout. Warm numbers would be lower and less comparable.
• One total is partial: Polars-streaming misses q5+q18 at 128 GiB (OOM). Full-22 totals are labeled as such.
• Storage differs: the cluster used HDFS-on-NVMe; the single nodes used local NVMe.Both are “local fast disk,” but not identical.
• TPC-H is a synthetic star-schema workload. Your joins, selectivity, and data layout will move these numbers. Benchmark your own.

Closing

The tidy conclusion “engine X is fastest” didn’t survive contact with 120 GB of Parquet. What survived is more useful: Spark dominates cluster batch analytics, a single Linux node beats a small cluster outright, and at these scales your memory budget and execution mode decide more than the logo on the engine.

If you’re spinning up a cluster to crunch 100-odd gigabytes, it’s worth asking first whether one big Linux box and a columnar engine would just… do it faster. On this workload, it did.

Links

Original Polars benchmark

This benchmark github repository

Thanks for reading!