Your Athena query already finished. Your client is still working.

The same SELECT took 26.84 seconds one way and 2.37 seconds the other. Athena was not the difference.

By: Mayur, Navdeep

not yet arrived

Here is a number that stopped us mid-build.

At 100,000 returned rows, Athena reported a median execution time of 1.34 seconds for our paginated result path and 1.43 seconds for our Parquet path. Nearly identical. The engine did the same analytical work in the same amount of time.

But end to end — SQL submitted, data sitting in a local columnar structure, ready to use — the first path took 26.84 seconds and the second took 2.37 seconds.

That is an 11.33x difference on identical SQL, an identical Hudi snapshot, the same MacBook, the same region. And crucially: Athena was not 11 times faster. The query engine was never the bottleneck. Everything expensive happened after Athena had finished, in the unglamorous business of getting the result onto the machine.

That result is not an accident. It’s the whole OrcaSheets bet, stated as a number.

For decades, the client has been the laggard. Every serious optimization — query planners, columnar storage, distributed execution, vectorized scans — went into the server. The last mile to the machine in front of the user stayed dumb: row-at-a-time APIs, string serialization, pagination. We keep making the engine faster and then hand its output to a delivery path designed like it’s 2009. OrcaSheets exists to move that optimization back to where the user actually sits — and this crate is the first concrete proof of the thesis.

It’s a milestone for us in another way, too: [athena-arrow-core](https://crates.io/crates/athena-arrow-core/0.1.1) is Dataorc/OrcaSheets's first open-source Rust crate — the start of us publishing the low-level tooling OrcaSheets is built on, in the open.

This is the story of why we rebuilt that part, what survived contact with AWS, and what 195 benchmark queries actually taught us.

The problem nobody benchmarks

Most Athena benchmarks measure the query. Ours started by measuring the delivery — because that is where our users were actually waiting.

OrcaSheets is a local-first analytics application. The whole premise is that you pull a useful slice out of a cloud lake, materialize it locally, and keep exploring without turning every keystroke into another remote round trip. That premise depends entirely on one operation being fast:

Run this SQL, move the analytical result efficiently to the user’s machine, preserve its types, and hand it to the local columnar engine.

Athena is already a distributed, columnar query engine. It can write Parquet directly through UNLOAD. And yet the standard way to retrieve a result — the GetQueryResults API — returns string-shaped rows, at most 1,000 rows per response. A 100,000-row result therefore needs roughly 100 paginated API calls, each one followed by string-to-type conversion in the client. (AWS documents the limit and the response shape.)

Meanwhile the consumers we wanted to feed — Polars, DataFusion, DuckDB, modern Pandas, and our own runtime — all speak Apache Arrow natively.

So we had a columnar engine handing pages of rows to a columnar consumer, with a lossy string detour in between. That mismatch was the entire bottleneck. It was not a mystery to solve so much as an obvious piece of plumbing to replace.

We decided to build the bridge ourselves, publish the design before scoping it up, and release the reusable core from Dataorc as [athena-arrow-core](https://crates.io/crates/athena-arrow-core/0.1.1).

One insight: small results and bulk results are different animals

Athena already ships a bulk primitive. UNLOAD writes a SELECT result to S3 as Parquet and hands back a manifest of the output objects — no catalog table required. (AWS documents UNLOAD and its constraints.)

So the crate exposes two paths behind one API.

The bulk path:

SELECT → Athena UNLOAD → S3 Parquet → Arrow RecordBatch

The small-result path:

SELECT → GetQueryResults pages → typed Arrow arrays → Arrow RecordBatch

The tempting conclusion — “just always use Parquet” — is wrong, and the benchmark proves it. UNLOAD carries fixed query, manifest, and S3 overhead. On tiny results, pagination is genuinely cheaper. The interesting engineering is knowing where the crossover lives, and being honest that it moves with result width, network location, and file count.

The receipts

The setup, so you can argue with it:

• Apple M3 Max MacBook Pro, 36 GB RAM
• a client outside AWS, matching real OrcaSheets usage
• Athena, Hudi, and S3 in ap-south-1
• a fixed nine-column projection, frozen Hudi commit cutoff
• ~0.24 MiB scanned per query
• three discarded warmups, ten measured runs per cell
• cache and Athena result reuse disabled

The source partition held 8,514 rows at the cutoff. Larger results were built by cross-joining that frozen source with a small integer sequence and applying an exact limit — keeping the projection and scan comparable while only the delivered row count changed.

The two core paths:

At 100 rows, pagination won. As the result grew, pagination became the dominant cost and the two paths tore apart. This is the whole thesis in one table: the retrieval path, not the query, decides how fast bulk results feel.

We benchmarked against strong opponents, not a strawman

It would have been easy to compare our Parquet path only against our own slow path and declare victory. Instead we put it in a wider ring: vanilla Athena CSV pulled from S3, that same CSV parsed into a local PyArrow table, PyAthena’s Arrow CSV path, and PyAthena’s Arrow UNLOAD path.

Median end-to-end wall time

At 100,000 rows, core UNLOAD was 11.33x faster than core GQR, 1.29x faster than PyAthena’s Arrow CSV path, and 1.09x faster than PyAthena’s Arrow UNLOAD path.

One honest caveat, stated up front rather than buried: the observed comparison against vanilla CSV-to-Arrow was 5.95x, but we are not using that as a headline. The vanilla matrix was added later in the session instead of interleaved per iteration, and its 100,000-row network times were noisy — 14.08s median but 21.43s at p90. The next run interleaves every client before that ratio earns a claim.

“Arrow is faster” is too vague to be true

The most useful thing the benchmark did was kill a lazy narrative we could have leaned on.

We measured the vanilla path in three separate pieces: download Athena’s CSV, parse those exact bytes into a PyArrow table, then serialize that table to an Arrow IPC stream in memory — so the Arrow size is a measured value, not a guess.

At 100,000 rows, converting the downloaded CSV to Arrow added 11.6 milliseconds. The 14.10-second vanilla median was dominated by query completion and hauling a 14.34 MB CSV object across the network — not by parsing.

So Arrow did not magically make CSV parsing disappear, because CSV parsing was never the problem. Arrow gave us a consistent columnar boundary for the application. The real win came from choosing a bulk delivery path before that boundary — Parquet from S3 instead of paginated rows or a single fat CSV artifact. Precision matters more than a good-sounding slogan.

The router is a policy, not a theorem

Once both paths existed, execute_arrow had to choose between them. It parses the SQL first; opaque or unsupported syntax falls back conservatively to first-keyword and limit inspection.

And here the benchmark bit its own author. At 10,000 rows, version 0.1.1 routes to GQR — but forced UNLOAD had the lower median on this workload. Our 10,000-row cutoff looked reasonable on paper and is probably too conservative for local clients. One dataset is not enough to move the default globally, but it is enough to turn the threshold from an intuition into an evidence-backed open question. That is the point of shipping with a spec and a reproducible harness instead of a vibe.

The decisions we made against a launch headline

Two choices cost us marketing opportunities, on purpose.

Cache is off by default. The design includes content-addressed result reuse, and canonicalizing SQL into a deterministic key is the easy part. Proving the source has not changed is the hard part — Iceberg snapshot IDs, Hudi timelines whose relationship to Athena’s read state needs real validation, Hive/Glue metadata that cannot prove objects inside a partition are untouched. Reusing a fast stale result is a correctness bug wearing a performance costume. So 0.1.1 exposes an opt-in reuse primitive, keeps it off, and excludes cache from every benchmark number above.

The ADBC and FFI layers are labeled honestly. The workspace is three crates: athena-arrow-core (AWS execution, routing, Parquet decode, Arrow conversion), athena-arrow-adbc (a Rust-native ADBC-shaped surface over the core), and athena-arrow-ffi (a reserved C ABI boundary that is still a scaffold). A universal C ADBC driver that Python, R, and the JVM can load needs the full metadata surface, parameter binding, cancellation, richer types, structured errors, and the conformance suite. We are not calling 0.1.1 that before it earns the label.

Keeping the core independent of both layers matters: OrcaSheets already has a Tokio runtime and can use the fastest native path today while the standards-based distribution matures on its own schedule.

What 0.1.1 ships — and what it doesn’t

One more comparison we owe you: Amazon’s current JDBC 3.x driver. Its default auto fetcher can pull results straight from S3, so presenting modern JDBC as a row-API-only baseline would be dishonest. (AWS documents those fetchers.) Until that runner is in the matrix, this is a comparison of Arrow delivery paths — not a claim that we have tested every Athena client.

Try it in five minutes

[dependencies]
athena-arrow-core = "0.1.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

use athena_arrow_core::{AthenaConfig, Client};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut config = AthenaConfig::new(
        "ap-south-1",
        "primary",
        "s3://my-bucket/athena-results/",
    );
    config.database = Some("default".to_owned());

    let client = Client::new(config).await?;
    let result = client
        .execute_arrow("SELECT order_id, total FROM orders LIMIT 1000")
        .await?;

    println!("query_id: {}", result.query_id);
    println!("path: {:?}", result.path);
    for batch in result.batches {
        println!("{} rows, {} columns", batch.num_rows(), batch.num_columns());
    }

    Ok(())
}

Notice that the result tells you which path it used — the routing decision is data, not a hidden internal.

There’s a runnable [orcasheets.rs example](https://github.com/dataorchestration/adbc_driver_athena/blob/master/crates/athena-arrow-core/examples/orcasheets.rs) and the full benchmark protocol in the repo.

Help us find the real boundary

Install athena-arrow-core 0.1.1, point it at a non-production workgroup and output prefix, and keep the Athena query ID when you report back. The tests we'd most like to see:

• narrow versus wide projections
• 100 / 1,000 / 10,000 / 100,000-row results
• Hive, Iceberg, and Hudi-backed tables
• nested, decimal, and timestamp types
• local clients versus same-region compute
• the modern Athena JDBC 3.x S3 fetcher

Every raw benchmark row carries wall time, Athena engine and queue time, bytes scanned, row and column counts, Arrow bytes, and the query ID, with SHA-256 checksums on the result files. Inspect the raw CSV, p90 summary, environment metadata, query IDs, and checksums.

The goal was never to make every Athena query go through Parquet. It was to build a driver that understands the difference between a small control result and a bulk analytical one, exposes that decision, and earns its defaults with evidence instead of adjectives.

And underneath that: the server has been optimized for years. It’s time the client stopped being the slow part. athena-arrow-core 0.1.1 is our first open-source step toward that, and we're glad to be doing it in the open.

The crate lives on crates.io; the source and reproducible protocol are in the [adbc_driver_athena repository](https://github.com/dataorchestration/adbc_driver_athena).

About DataOrc & OrcaSheets

This crate is the kind of work we’ve been doing since founding DataOrc in 2018 — make data accessible, at scale, at sane cost. Across 70+ enterprise clients, we’ve built petabyte-scale platforms and 50K+ TPS systems while obsessing over reliability, performance, and total cost of ownership. Cutting a 100,000-row result from 26.84 seconds to 2.37 — and open-sourcing the core that did it — is exactly the kind of outcome that philosophy is aiming at.

OrcaSheets is that philosophy turned product: cloud-grade analytics that run on your machine. Open formats, local-first compute, billions of rows — no data center rental required. athena-arrow-core is the first piece of that stack we've published in the open, and more will follow.

Want a no-strings consulting session with our tech team, or just want to sanity-check an architecture decision before you commit to it? Ping us. We’re happy to help — whether you end up using OrcaSheets or not.