How to Choose the Right Compute Tool for Your Data Pipeline

From Kafka to DuckDB, Spark to dbt — a practical decision schema for data engineers

Every data team reaches a point where a new pipeline is on the roadmap, and suddenly everyone has an opinion: “We should use Spark for this.” “Spark? We barely have 10 GB of data.” “What about Flink?” “Isn’t dbt the right call here?”

The modern data stack is rich — but that richness comes with decision fatigue. Kafka, Spark, Flink, dbt, DuckDB, Polars, Trino, Prefect, Airflow, Apache Beam. Each tool was built to solve a real problem brilliantly. The challenge is knowing which problem you actually have.

This article walks through a set of questions you can ask about any pipeline to arrive at the right tool, not just the most familiar one.

The mistake isn’t picking the wrong tool. It’s picking a tool before understanding the problem.

Start with orchestration

Before thinking about compute, ask a more fundamental question: does this workload involve multiple steps with dependencies, retries, and scheduling? If so, you need an orchestrator — and the compute question comes after that.

Prefect and Airflow are the standard answers here. These tools don’t do compute themselves — they coordinate compute tools. If your workload is a single transformation or ad-hoc query, skip orchestration entirely. But if you’re building anything resembling a production pipeline, orchestration is the first decision — not the last.

Does data arrive in real time?

The second question splits the ecosystem cleanly in two: batch or streaming. Real-time data — sensor events, clickstreams, logs, financial ticks — demands a fundamentally different architecture than data that lands in a warehouse once a day.

Batch vs Streaming Pipelines

The streaming path

If your data arrives as a continuous stream, the next question is whether you need a durable message broker. Durability means you can replay events, fan-out to multiple consumers, and decouple producers from consumers. If yes, Apache Kafka is the standard answer.

Once events are in Kafka, latency requirements determine your stream processor:

• Sub-second latency (fraud detection, IoT anomaly detection, real-time recommendations) → Flink
• Micro-batch intervals (a few seconds to minutes, unified batch/streaming logic) → Spark Structured Streaming

The batch path: it’s all about scale

If your data arrives in batch — daily dumps, nightly exports, periodic API pulls — the decisive variable is volume. Below the TB threshold, distributed compute could be overkill. Above it, you need distributed processing.

When your data is already in a warehouse like Snowflake, BigQuery, or Redshift, and your logic is expressible in SQL, you don’t need a compute engine at all — you need a transformation layer. That’s where dbt and SQLMesh shine.

For medium-sized data with Python processing — especially if you’ve already written transformations in a previous project — DuckDB and Polars are worth getting considered. Both need no cluster.

Summary

The common failure in data engineering isn’t ignorance of tools — it’s defaulting to familiar ones regardless of fit. Spark on 5 GB of data. A full Airflow setup for a single job.

Sources & Further Reading

• Apache Kafka documentation
• Apache Flink documentation
• Apache Spark documentation
• Apache Beam documentation
• dbt documentation · SQLMesh documentation
• DuckDB documentation · Polars documentation
• Trino documentation
• Prefect documentation · Airflow documentation
• Kleppmann, M. Designing Data-Intensive Applications. O’Reilly, 2017.
• Reis, J. & Housley M. Fundamentals of Data Engineering. O’Reilly, 2022.