Why StarRocks Is Better Than DuckDB for Data Lake and Apache Iceberg Analytics

If there is one use case that made DuckDB famous, it is this one: point SQL directly at Parquet files on S3 — no warehouse, no loading, no cluster — and get answers in seconds. A single SELECT * FROM read_parquet('s3://bucket/...') turned the data lake from something you needed a Spark cluster to touch into something you could query from a laptop. With the Iceberg extension maturing rapidly — read support, then write support in DuckDB 1.4, then schema evolution via ALTER TABLE and even Iceberg-in-the-browser via WebAssembly — DuckDB has become, for many engineers, the default way to look at a lakehouse.

And for looking at a lakehouse, it is superb. The trouble starts when “looking at” becomes “running on”: when the Parquet-on-S3 prototype becomes the team’s shared query layer, when the Iceberg tables grow from gigabytes to terabytes and beyond, when dozens of analysts, dashboards, and data products all need fast SQL over the same lake at the same time. Data lake and Iceberg analytics as a production workload — the shared, concurrent, governed, always-on query layer over an organization’s object storage — has a fundamentally different shape from single-user file querying. It is bounded not by how clever one process can be, but by how much scan bandwidth, metadata processing, caching, and concurrency an architecture can bring to bear on petabytes of remote data.

In this post, we examine why StarRocks, an open-source, distributed MPP analytical database with native lakehouse integration, is the better engine for production data lake and Apache Iceberg analytics — drawing on architectural analysis and real-world evidence from companies that run both ends of this spectrum.

What Production Lakehouse Analytics Demands

Querying files on a laptop and serving a company’s analytics from its data lake are different sports. The production version of this workload imposes requirements that the exploratory version never surfaces:

1. Scan throughput that scales with data, not with one machine — a query over a multi-terabyte Iceberg partition is, physically, a race to pull columnar data out of object storage and through CPUs. A single node’s network interface and core count put a hard ceiling on that race. Production lake analytics needs aggregate bandwidth: many nodes scanning many files in parallel.
2. Fast, repeated access to hot data — object storage is cheap and slow. The same dimension tables, recent partitions, and popular datasets get queried thousands of times a day. Without a distributed caching layer, every query pays the full S3 latency and request-cost tax again — the “scan the whole bucket machine” failure mode.
3. Metadata at Iceberg scale — large Iceberg tables carry tens of thousands of manifest files and millions of data files. Query planning means parsing that metadata quickly, pruning aggressively, and caching the results. At petabyte scale, metadata handling alone can dominate query latency if it runs on one thread of one machine.
4. Concurrency for the whole organization — the lake is shared infrastructure. BI dashboards, ad-hoc analysts, dbt jobs, and customer-facing data products all hit it simultaneously. The query layer must absorb hundreds to thousands of concurrent queries with workload isolation, so one analyst’s full-table scan doesn’t freeze everyone’s dashboards.
5. Full table-format semantics — production Iceberg tables use merge-on-read deletes, position and equality delete files, schema evolution, hidden partitioning, and time travel. An engine that silently mishandles or refuses delete files isn’t queryable infrastructure; it’s a demo.
6. Writes, compaction, and freshness — a lakehouse is not read-only. Streaming pipelines append continuously; CDC produces row-level updates; small files need compaction. The query layer either participates in this lifecycle or forces a second engine into the stack.
7. Governance and serving — shared lake analytics needs authentication, fine-grained access control, auditability, and a standard protocol every BI tool speaks — because the whole point of the lakehouse is one copy of data, many consumers.

Let’s see how DuckDB’s architecture holds up against these demands, and where StarRocks excels.

Where DuckDB Falls Short for Production Lake Analytics

None of what follows is a bug. DuckDB’s limitations here are deliberate consequences of its core design choice — to be an in-process, single-node analytical engine. That choice is what makes it magical on a laptop and structurally mismatched with a petabyte-scale shared lakehouse.

One Node’s Bandwidth Is the Whole Engine’s Bandwidth

Every byte DuckDB processes must flow through one machine: one network link to S3, one pool of cores, one memory hierarchy. On a 10 Gbps instance, that is roughly 1.25 GB/s of theoretical scan bandwidth from object storage — shared across all concurrent queries. A query that needs to scan 2 TB of Parquet after pruning is looking at tens of minutes of pure I/O, no matter how perfectly vectorized the execution is. The only remedy is a bigger machine, and machines stop getting bigger long before lakes stop growing.

This is precisely the wall that pushed Intuit’s data platform team away from DuckDB when they evaluated it (alongside Druid, ClickHouse, and Pinot) for their analytics platform: its “single-node architecture was an operational non-starter.” A distributed engine spreads the same scan across dozens or hundreds of nodes, turning tens of minutes into seconds — not by being smarter, but by being wider.

Every Query Pays the S3 Tax Again

DuckDB performs prefetching and parallel range reads within a query, but it has no persistent, shared cache of remote data. Each new process, each new analyst, each dashboard refresh re-reads the same hot Parquet footers, the same dimension tables, the same current-day partition from object storage — paying both the latency and the per-request API costs every time. Practitioners who build “DuckDB + S3 + Parquet” stacks warn about exactly this: without careful partition discipline and external caching layers, you accidentally build a slow, expensive bucket-scanning machine. The workaround — copying data to local disk or memory first — quietly abandons the lakehouse premise of querying data in place.

Iceberg Support Is Young, Partial, and Single-Threaded at Heart

DuckDB’s Iceberg extension has improved at an impressive pace, and the team deserves credit for it. But for production use against large, actively maintained Iceberg tables, the gaps documented by reviewers of the extension remain material: it has historically lacked or trailed on delete-file handling (tables with row-level deletes — i.e., any table receiving CDC upserts — could not be read correctly or at all), Parquet remains the only supported data-file format (Avro and ORC data files in mixed-format tables are ignored), write support only arrived with DuckDB 1.4 in late 2025 and is still maturing, and metadata processing for planning — parsing manifests, pruning files — happens within the single process. On an Iceberg table with millions of data files, planning alone becomes a bottleneck before a single row is scanned. An engine that can’t fully honor the table format’s semantics under real CDC and compaction activity can’t be the organization’s source-of-truth query layer.

Concurrency: Built for One User, Not One Company

DuckDB is in-process: it lives inside your Python script, your notebook, your job. There is no shared server, no admission control, no resource governance between users — because there are no “users,” only the host process. Teams that put DuckDB behind an API for shared lake querying report it serves up to a couple hundred concurrent dashboard-style queries on a well-provisioned node before latency degrades — a fine number for a team tool, and a hard cap for an organization’s lakehouse. Worse, in the shared-lake context every one of those concurrent queries is competing for the same single NIC to S3. Concurrency and scan bandwidth degrade together.

No Governance, No Serving Layer, No Single Source of Truth

A lakehouse exists so that one copy of data serves many consumers — which makes the query layer the natural place to enforce who may see what. DuckDB, as a library, has no authentication, no RBAC, no row- or column-level security, no audit log, no standard network protocol for BI tools. Every team that productionizes it builds that scaffolding themselves in an API tier — and every laptop running DuckDB against the bucket bypasses it entirely. For regulated data on the lake (the Fanatics team cited “inconsistent governance and unclear ownership” as a core failure of their pre-consolidation lake stack), this is disqualifying.

The Lake Lifecycle Needs More Than Reads

Production Iceberg tables are living things: streaming appends, CDC merges, compaction, snapshot expiration. DuckDB participates in almost none of this — its Iceberg writes are new and basic, it has no streaming ingestion, no continuous materialization, no background refresh. In practice a DuckDB-centric lake stack still needs Spark or Flink for everything that changes data, leaving DuckDB as a read-only window onto tables it cannot help maintain — one more engine in the stack rather than a consolidation of it.

Why StarRocks Excels at Data Lake and Iceberg Analytics

StarRocks treats the data lake not as an import source but as a first-class storage layer. Its external catalogs connect natively to Apache Iceberg, Delta Lake, Hudi, Hive, and raw Parquet/ORC on object storage, and the same vectorized, SIMD-optimized MPP engine that powers its internal tables executes directly against lake data. This is the architecture behind some of the largest Iceberg deployments in production today.

Distributed Scans: Aggregate Bandwidth Instead of One NIC

A StarRocks query over an Iceberg table is planned by a cost-based optimizer and fanned out across every compute node in the cluster. Each node scans its assigned data files in parallel with the others — predicate pushdown, late materialization, and runtime filters minimizing what’s read; dozens of network links and hundreds of cores doing the reading. Scan throughput scales linearly with cluster size: when the lake grows or latency targets tighten, you add nodes, not heroics.

The numbers from production bear this out. Fanatics, running a 6 PB lakehouse, found ad-hoc queries on Iceberg via StarRocks ran 10× faster than Athena — the distributed serverless engine they were replacing, let alone a single-node one. TRM Labs runs petabyte-scale blockchain intelligence analytics on StarRocks over Iceberg. WeChat queries trillion-row Iceberg tables interactively. These are not workloads any single machine can address at any price.

A Distributed Data Cache That Makes S3 Feel Local

StarRocks’ data cache transparently keeps hot lake data on local NVMe across the cluster, populated on first access and shared by all subsequent queries from all users. The current day’s partition, the dimension tables every dashboard joins against, the datasets an analyst team is actively working — after first touch, they’re served at local-disk speed without another S3 round trip or API charge. Combined with metadata caching, this routinely delivers near-warehouse latency on data that never leaves the lake — the economics DuckDB-on-S3 promises, but shared across the organization and sustained under load.

Iceberg Metadata Handling Built for Millions of Files

StarRocks parses Iceberg manifests in parallel across the cluster and caches metadata aggressively, so planning a query against a table with millions of data files stays fast. It honors the format’s full read semantics — position and equality deletes (so CDC-fed tables read correctly), schema evolution, hidden partition transforms, time travel — and on the write side supports INSERT/CTAS into Iceberg tables, so transformation results can land back in the open format. Materialized views can be built on lake tables with automatic, transparent query rewrite: the expensive aggregation over raw Iceberg data is computed once on refresh, and thousands of dashboard queries are silently served from the precomputed result — no application changes, no second copy of the pipeline.

Concurrency and Governance for the Whole Organization

StarRocks is a real serving layer over the lake: MySQL wire protocol (every BI tool connects out of the box), authentication and fine-grained RBAC down to rows and columns, resource groups that isolate workloads so the ad-hoc power user cannot starve the executive dashboards, and proven production concurrency in the thousands of QPS. One engine gives the data science team, the BI tools, and customer-facing data products governed access to one copy of the data — which is the lakehouse’s entire value proposition. HerdWatch and Fresha both serve customer-facing dashboards directly over Iceberg through this stack; Fanatics consolidated Redshift, Snowflake, Athena, and Druid workloads onto it, cutting Snowflake usage by up to 95% and overall analytics cost by roughly 90%.

One Engine from Lake to Real Time

Because StarRocks also offers internal table storage with streaming ingestion (Kafka Routine Load, Flink connectors) and primary-key upserts, the lakehouse query layer and the real-time layer stop being separate systems. Hot, fast-changing data can live in StarRocks tables with second-level freshness; historical volumes stay in Iceberg; a single SQL statement joins both. Fanatics used exactly this to retire Apache Druid alongside its warehouse consolidation. A DuckDB-based stack, by contrast, must bolt on yet another engine the moment freshness requirements appear.

When DuckDB Still Makes Sense

Credit where due — for its intended job, DuckDB is not just adequate but best-in-class:

• Exploring the lake from a laptop — for one engineer poking at Parquet or Iceberg tables, nothing beats pip install duckdb and a query thirty seconds later. Every data team should have this in its toolbox.
• Local development and testing — developing transformations against a sample of lake data, in CI, without a cluster, is a perfect DuckDB fit.
• In-pipeline transformation — as an embedded engine inside a Python job that reads some files, reshapes them, and writes results, DuckDB is fast, cheap, and dependency-free.
• Small, single-tenant lake stacks — a small company whose entire lake is a few hundred gigabytes, queried by a handful of people with no strict SLAs, can run happily on DuckDB for a long time — with eyes open about what growth will eventually require.
• In-browser analytics — DuckDB-WASM querying Iceberg from the browser is a genuinely novel capability with no server-side equivalent.

The healthiest pattern we see is complementary: engineers explore and prototype against the lake with DuckDB, and the organization serves the lake with StarRocks. The open formats make this painless — both engines read the same Iceberg tables, so graduating a workload from laptop to production means changing the connection string, not migrating the data.

Conclusion

DuckDB taught the industry that the data lake could feel as accessible as a local file — and for individual exploration, it delivers on that promise brilliantly. But production data lake and Iceberg analytics is an infrastructure workload: petabytes of remote data, millions of metadata files, full table-format semantics under continuous CDC, organization-wide concurrency, and governance over a shared source of truth. An in-process, single-node engine — bounded by one NIC, one cache, one process’s view of the world — cannot be configured into that role, as teams from Intuit (“an operational non-starter”) to the architects of failed bucket-scanning stacks have learned.

StarRocks provides what the workload actually requires: MPP scans whose bandwidth scales with the cluster, a shared distributed cache that makes object storage feel local, parallel Iceberg metadata processing with full delete-file and schema-evolution semantics, materialized views over lake tables, thousands-of-QPS concurrency with RBAC and workload isolation, and a path to real-time freshness in the same engine. That is why Fanatics runs a 6 PB Iceberg lakehouse on it at 10× Athena’s speed and ~90% lower cost, why TRM Labs and WeChat trust it at petabyte and trillion-row scale, and why, when your DuckDB lake queries need to serve more than one laptop, StarRocks is the engine to grow into.