Why StarRocks Is Better Than DuckDB for Real-Time Streaming Analytics

DuckDB deserves every bit of its popularity. It is the in-process analytical engine that made “just query the Parquet file” a viable data strategy — no servers, no clusters, a single pip install, and astonishing single-node performance on analytical SQL. For local exploration, batch ETL, and dbt-style transformation pipelines, it is hard to beat. And so, predictably, the community has started pushing it toward the next frontier: streaming. Blog posts now describe Kafka-to-DuckDB micro-batch patterns, rolling-window aggregations, and "good enough real-time" architectures where a buffer process drains events into DuckDB tables or Parquet partitions every few seconds.

These patterns are clever, and for a single-user dashboard over modest event volumes, they work. But there is a meaningful difference between making a batch engine feel real-time and running a database built for real-time. Real-time streaming analytics — continuously ingesting high-velocity event streams from Kafka, Kinesis, or CDC pipelines, and serving fresh, low-latency queries to many concurrent users while data keeps arriving — is a workload with structural requirements that an in-process, single-writer, single-node engine cannot satisfy, no matter how well-engineered.

This post examines why StarRocks, an open-source distributed MPP analytical database, is the better foundation for real-time streaming analytics, drawing on architectural analysis, practitioner experience with DuckDB streaming patterns, and production evidence from teams — including Intuit — that evaluated DuckDB for exactly this workload.

What Real-Time Streaming Analytics Actually Demands

“Real-time analytics” gets used loosely, so let’s be precise about the workload:

1. Continuous, high-velocity ingestion. Events arrive from Kafka topics, Kinesis streams, Flink jobs, or CDC feeds at thousands to hundreds of thousands of events per second — all day, every day. Ingestion is not a nightly job that finishes; it is a permanent, concurrent workload.
2. Seconds-level end-to-end freshness. The promise of real-time analytics is that an event generated now is queryable in seconds. Freshness SLAs of 2–5 seconds end-to-end (event → broker → database → dashboard) are common in fraud monitoring, operational dashboards, personalization, and in-product analytics.
3. Mutable data, not just appends. Real-world streams carry updates: orders change status, accounts change balances, CDC feeds replay UPDATE and DELETE operations from OLTP systems. The engine needs first-class, high-throughput upserts — not append-only tables with periodic deduplication jobs.
4. Queries and ingestion running concurrently, forever. Dashboards, alerting rules, and APIs query the data continuously while it is being written. Neither side can pause for the other, and ingestion load must not destroy query latency (or vice versa).
5. Exactly-once correctness. Streams fail and retry. Without transactional ingestion semantics, retries become duplicates and metrics silently drift. Financial and operational use cases cannot tolerate this.
6. Concurrent consumers of the results. Real-time analytics is rarely for one person. Ops teams, alerting systems, and customer-facing dashboards all hit the same fresh data, often at hundreds or thousands of queries per second.
7. No maintenance windows. A streaming pipeline has no natural “downtime” in which to compact files, rebuild tables, or restart the process. The system must do its housekeeping online.

Measured against these requirements, DuckDB’s design choices — brilliant for its intended purpose — become liabilities.

Where DuckDB Falls Short for Streaming Workloads

None of what follows is a criticism of DuckDB’s engineering. The DuckDB team is explicit that it is not a stream processor, and its own documentation on streaming patterns describes micro-batch workarounds rather than native streaming. The problems below are structural consequences of being an in-process, single-node, single-writer engine.

There Is No Native Streaming Ingestion — You Build It Yourself

DuckDB has no built-in connector that consumes from Kafka, Kinesis, or Pulsar, manages consumer offsets, handles schema evolution, or guarantees delivery semantics. Every “DuckDB streaming” architecture in the wild is the same DIY pattern: an external process (Node.js, Python, Go) subscribes to the stream, buffers events, and flushes micro-batches into DuckDB on an interval.

That external process is now load-bearing infrastructure that you wrote and you operate. It must track Kafka offsets durably (practitioners note that since Kafka retention will eventually evict messages, you must materialize offsets or timestamps into tables and manage your own checkpointing), handle backpressure when the database stalls, dedupe on retries to fake exactly-once semantics, and restart cleanly after crashes. The DuckDB community’s own guidance is candid: DuckDB “is not Kafka, not Flink, not a distributed stream processor” and has no built-in broker integration — it can deliver seconds-level latency for modest streams if you assemble the surrounding machinery yourself.

StarRocks ships this machinery as a database feature. Routine Load is a declarative, continuously running ingestion job: you write one CREATE ROUTINE LOAD statement pointing at a Kafka topic (JSON, CSV, or Avro with Schema Registry support), and the cluster consumes partitions in parallel, manages offsets, applies transformations, and commits data transactionally — with exactly-once semantics guaranteed by the database, not by your application code. Batch intervals of a few seconds are a configuration property, not an engineering project. For other paths there are Stream Load (HTTP-based push), the Flink and Spark connectors with two-phase commit, and Kafka Connect. The difference is categorical: with StarRocks, streaming ingestion is something you declare; with DuckDB, it is something you build and babysit.

One Writer, One Machine: Ingestion and Queries Fight for the Same Box

DuckDB follows a single-writer concurrency model — one process writes at a time, and within that process, writes serialize. Practitioner write-ups of DuckDB streaming patterns repeat the same warning: “only one writer can hold the pen at a time.” Micro-batching exists precisely to work around this — you cannot afford per-event writes, so you batch to amortize the single writer.

This collides with requirement #4 above. In a streaming system, ingestion is permanent. On DuckDB, that permanent ingestion workload and your entire query workload share one machine’s cores, memory, and I/O bandwidth. A heavy analytical query slows ingestion (your freshness SLA degrades); a burst of events ahead of compaction slows queries (your latency SLA degrades). There is no way to scale ingestion capacity and query capacity independently, because there is only one box and one process. As event volume grows, your only lever is a bigger machine — until there isn’t a bigger machine.

StarRocks decouples all of this. Ingestion runs as distributed jobs spread across the cluster, consuming Kafka partitions in parallel. Queries fan out across the same cluster under an MPP execution model with vectorized, SIMD-optimized operators. Resource groups let you cap the CPU and memory share of ingestion versus queries so neither starves the other. And when either side needs more capacity, you add nodes. Intuit’s production deployment ingests 100,000 events per second while serving real-time queries — this is not a scale at which micro-batching into a single process is a conversation.

Upserts and CDC Are a First-Class Workload, Not an Afterthought

Append-only streams are the easy case. The hard case — and the common one, once CDC enters the picture — is streams of updates. DuckDB can execute UPDATE statements, but a sustained, high-velocity stream of row-level upserts running concurrently with analytical queries is far outside its design center: every update contends for the single writer, and there is no primary-key table type optimized for merge-on-write.

This was one of the specific findings in Intuit’s engine evaluation. Building a real-time analytics platform for in-product personalization, they assessed DuckDB alongside Druid, ClickHouse, and Pinot, and needed native real-time upsert support and performant multi-table joins at scale. DuckDB’s “single-node architecture was an operational non-starter,” and ClickHouse required upsert workarounds. StarRocks won on exactly the streaming-shaped capabilities: native upserts via the Primary Key table, multi-table join performance, and horizontal scaling. The Primary Key table model handles high-frequency upserts and deletes with a delete-and-insert strategy that keeps queries fast (no merge-on-read penalty), making it a natural CDC sink — Flink CDC pipelines from MySQL/PostgreSQL into StarRocks, with transactional guarantees, are a standard, documented pattern.

The results at Intuit: 2-second end-to-end data freshness at 100K events/second, a 98% reduction in data aggregation time, and the elimination of denormalization workarounds.

Freshness Cliffs: Snapshots, Compaction, and the Parquet Treadmill

Many DuckDB “streaming” architectures sidestep the single-writer problem by not writing to DuckDB at all — instead, the buffer process writes Parquet partitions to object storage, and DuckDB queries the files. This buys decoupling at the cost of a new treadmill: small-file proliferation, background compaction jobs you must schedule, partition-pruning discipline you must enforce, and a freshness floor set by your flush interval plus file listing overhead. Each piece is manageable; together they are a part-time job, and the architecture’s “real-time” is really “as fresh as your last flush plus your last compaction.”

StarRocks ingests into its own storage engine, where freshly committed data is immediately queryable, compaction is automatic and online, and there are no snapshot-staleness or file-listing concerns. The freshness SLA is a property of the ingestion configuration, not an emergent property of a pipeline of cron jobs.

When the Results Need an Audience, the Serving Problem Returns

Real-time analytics is consumed by dashboards, alerts, APIs, and increasingly customer-facing features — which means concurrent access by many users against the freshest data. DuckDB is a library inside one process: no network serving layer, no authentication or RBAC, no admission control, no high availability. If the process dies mid-stream, ingestion and serving die together, and you are restoring from files while events pile up in Kafka.

StarRocks is a fault-tolerant distributed service: data is replicated, nodes fail without taking the system down, ingestion jobs resume from committed offsets, and production clusters routinely serve thousands to tens of thousands of QPS with sub-second latency. This is the operational substrate a permanent streaming workload requires, and it is precisely what teams whose workloads involve continuous data and concurrent consumers find missing when the single-node path runs out.

The right mental model is one of division of labor: DuckDB as the brilliant in-process, single-file analysis tool — “like SQLite” for analytics — and StarRocks as the always-on streaming backbone. Data can even flow between them: export a slice from StarRocks into DuckDB for someone to explore or manage locally, while the live streaming pipeline keeps running on the cluster.

Why StarRocks Excels at Real-Time Streaming Analytics

To consolidate the architectural argument, here is what StarRocks brings that is purpose-built for this workload:

Declarative, exactly-once streaming ingestion. Routine Load continuously consumes Kafka with database-managed offsets and transactional commits; Stream Load handles push-based micro-batches; Flink/Spark connectors provide two-phase-commit sinks for CDC and stream processing pipelines. Seconds-level batch intervals are configuration, not code.

Primary Key tables for mutable streams. Native upserts and deletes at high throughput with merge-on-write semantics, so CDC streams land directly and queries stay fast — no dedup jobs, no merge-on-read penalty, no single-writer bottleneck.

Independent, horizontal scaling of ingestion and query. Distributed ingestion across the cluster, MPP query execution across the cluster, resource groups to isolate the two workloads, and node-based scaling for either. In shared-data mode, compute scales elastically over object storage.

Real-time joins instead of denormalization pipelines. A cost-based optimizer with colocate joins and runtime filters executes multi-table joins on fresh data at interactive speed — so you don’t need a Flink job flattening streams into wide tables just to make the database happy. Intuit eliminated denormalization entirely; Demandbase migrated off ClickHouse to “ditch denormalization.”

Serving-grade concurrency and availability. Replication, automatic failover, online compaction, rolling upgrades, RBAC, and production-proven concurrency in the thousands of QPS — the operational substrate a permanent streaming workload requires.

Open and standard. Apache 2.0 licensed, MySQL-protocol compatible, standard SQL — like DuckDB, it avoids lock-in; unlike DuckDB, it does so as a shared, always-on service.

When DuckDB Is Still the Right Choice

Honesty makes the comparison credible. DuckDB remains an excellent choice when:

• The “stream” is really periodic batch. If refreshing every 10–15 minutes (or hourly) satisfies the business, a scheduled job loading into DuckDB is simpler and cheaper than any streaming system.
• One user, one process. A single analyst tailing fresh-ish data locally, an edge device aggregating its own telemetry, or a notebook over recent Parquet exports — in-process is a feature, not a bug.
• Post-hoc analysis of streaming data. Querying yesterday’s Kafka archive on S3 is a batch problem DuckDB handles beautifully — including as a downstream consumer of data exported from StarRocks.
• Prototyping the analytics before productionizing the pipeline. Many teams (reasonably) prove out metrics logic in DuckDB before standing up streaming infrastructure.

The boundary is crisp: the moment ingestion becomes continuous, data becomes mutable, freshness becomes an SLA, and the audience becomes concurrent, you have left DuckDB’s design envelope. Bolting buffers, file compactors, offset trackers, and read replicas onto an in-process library doesn’t change its nature — it just means you are building a distributed real-time database out of application code, one incident at a time.

Conclusion

DuckDB is a masterpiece of single-node analytical engineering, and the micro-batch patterns its community has developed are genuinely useful for “good enough” near-real-time on small streams. But real-time streaming analytics as a production discipline — continuous high-velocity ingestion with exactly-once guarantees, native upserts for CDC, seconds-level freshness SLAs, joins on live data, and concurrent serving with high availability — is a distributed-systems problem. StarRocks was designed for precisely this profile, and the production record (Intuit’s 100K events/second with 2-second freshness being the clearest example, achieved after explicitly ruling out DuckDB) bears it out.

Use DuckDB where it shines: local analysis, batch transformation, and the edge of your data estate. When the events never stop coming and the dashboards never stop refreshing, build on StarRocks.