Spark + Kafka Under the Hood: Consumer Pools, Offset Management and Producer Flow Explained

Note for Readers

This article is focused purely on building a strong conceptual understanding of how Spark and Kafka work together under the hood. It is not a step-by-step guide, setup tutorial, or hands-on workshop.

If you’re looking for a quick setup guide or a ready-to-run implementation, this might not be the right place and you may want to skip ahead to more hands-on resources.

For everyone else who’s here to really understand what’s happening behind the scenes, stick along this will go deep, but it will be worth it.

Introduction:

When you run a streaming query, there is a complex “dance” happening between the Spark Driver, the Executors, and the Kafka Brokers. Before Spark can even start interacting with Kafka, it needs a special connector package:

org.apache.spark:spark-sql-kafka

This package is not just an add-on, it’s the bridge that allows Spark and Kafka to talk to each other properly.

Spark, by itself, is a processing engine. It understands DataFrames, transformations, tasks, and execution plans but it has no built-in knowledge of how to communicate with Kafka brokers, how to fetch messages, or how to push data back. On the other side, Kafka exposes its own client APIs KafkaConsumer for reading and KafkaProducer for writing which handle all the low-level communication, batching, partitioning, retries, and network interactions.

Instead of tightly coupling Spark with Kafka’s internal logic, the architecture is designed to decouple responsibilities:

• Spark focuses on what to process and how to execute it in parallel
• Kafka client libraries handle how data is fetched from and written to brokers

This connector package essentially plugs Kafka’s consumer and producer capabilities into Spark’s execution model. It translates Spark’s world (partitions, tasks, micro-batches) into Kafka’s world (topics, partitions, offsets, records), and vice versa.

This design makes the system easier to maintain, easier to upgrade (just update the connector when Kafka evolves) and much more robust in production

With this bridge in place, Spark and Kafka can work together seamlessly each doing what it does best. Now let’s walk through what actually happens behind the scenes when a message flows through this pipeline.

End to End architecture

1. Driver and Executors

When a streaming query kicks off in spark, the Driver takes the lead. It acts as the coordinator using an internal engine StreamExecution and a specialised KafkaOffsetReader.

The Driver’s Job: Metadata & Planning

Crucially, the Driver never reads the actual data. Its role is purely administrative:

• Fetch Metadata: It talks to Kafka through KafkaConsumer to find out which partitions exist.
• Track Offsets: It identifies the “from” and “to” offsets for the current micro batch.
• Create the Plan: It constructs a KafkaSourceRDD where each RDD partition typically corresponds to one Kafka partition.

The Executor’s Job: Data Processing

Once the driver computes this, it creates offset ranges for the micro-batch like:

Partition 0 → offset 100 to 200

Partition 1 → offset 50 to 120

it sends that task to an Executor. This is where the data reading happens:

• The Executor uses the Kafka client library to create a Consumer. (We will deep dive into Consumers down in the article)
• The Spark tasks through the KafkaConsumer, performs a seek(startOffset) to find the right spot.
• It calls poll() to pull the actual records into memory.

Partition Mapping: The Rule of 1:1

At the heart of this integration is a simple rule:

1 Kafka Partition = 1 Spark Task.

If your Kafka topic has 10 partitions, Spark will launch 10 tasks. This mapping exists to ensure data ordering and prevent duplicate reads. If two tasks tried to read the same partition simultaneously, they might overwrite each other’s progress or process messages out of order.

The “minPartitions” Tweaks:

Sometimes, your Kafka partitions are too few, but your data volume is massive. This is where minPartitions comes in.

Scenario: You have 4 Kafka partitions but want more parallelism for faster processing. By setting minPartitions = 8, Spark will split each Kafka partition into two “sub-ranges” (e.g., offsets 0–500 and 501–1000).

The Catch: This can only increase parallelism. If you have 10 Kafka partitions and set minPartitions = 5, Spark will ignore you and stick with 10.

2. The Kafka Consumer Pool

Kafka Consumer Architecture

Before we look at the pool, we have to understand the swimmer: the Kafka Consumer.

What is a Kafka Consumer?

Think of a Kafka Consumer as a specialized courier service. Its only job is to connect to a Kafka broker, ask for data from a specific topic and partition, and bring those records back to the application (in this case, a Spark Executor).

To do this effectively, a consumer has to do a lot of work:

• A TCP Connection: A persistent open pipe to the Kafka broker.
• Metadata: A map of the entire Kafka cluster so it knows which broker is the “leader” for the partition it needs.
• Fetch Buffers: Internal memory space used to pre-fetch and hold data before Spark asks for it.
• Heartbeat Thread: A background process that constantly tells Kafka, “I’m still alive!” so Kafka doesn’t rebalance the group.

Why “Pooling” is Necessary

Creating this “Consumer” is incredibly expensive in terms of time and resources. It involves a multi-step network handshake and fetching cluster information.

If Spark created a brand-new consumer for every single task in every 100ms micro-batch, the system would spend 90% of its time setting up connections and only 10% actually processing data. This is where the Kafka Consumer Pool (or Consumer Cache) saves the day.

How the Pool Operates

Instead of destroying the consumer after a task finishes, Spark’s Executor drops it into a local “library” (the pool).

1. The Identifier (The Library Card): Each consumer in the pool is indexed by a unique key: (topic, partition, groupId). If there are multiple streaming queries in the same Spark application reading from the same topic but using different checkpointLocations, they will not share the same consumer because their "Internal Group ID" (generated by Spark) will be different.
2. The Reuse Flow:

• Batch 1: A Spark task starts on Executor A. It looks in the pool for a consumer for Topic-X, Partition-0. It finds nothing, creates one, uses it, and then returns it to the pool.
• Batch 2: The next micro-batch starts. A new task for processing the same partition arrives at Executor A. It checks the pool, finds the existing consumer, and borrows it.
• The Result: No new TCP handshakes, no metadata fetching, and near-zero latency for the connection.

Critical Safety: The “No-Sharing” Rule

While consumers are reused, they are never shared simultaneously.

Kafka Consumers are not thread-safe(i.e two threads cannot run simultaneously). If two Spark tasks tried to use the same consumer, the internal fetch buffers would become corrupted, and offsets would get scrambled. Spark’s pool logic ensures that if a consumer is currently “checked out” by Task A, Task B must either wait or (if the pool is configured to do so) create a temporary new one.

Can refer to this link below for better managing the consumer configuration for the best use of resources.

[embed]Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) Structured Streaming integration for Kafka 0.10 to read data from and write data to Kafka. For Scala/Java applications…spark.apache.org

3. Offset Management: Spark Takes the Wheel

In standard Kafka apps, Kafka tracks what you’ve read. In Spark Structured Streaming, Spark is the source of truth.

Spark ignores Kafka’s internal offset storage and instead stores offsets in its own Checkpoint Directory. This provides two massive benefits:

1. Deterministic Retries: If a batch fails, Spark knows exactly which offsets to re-read.
2. Fault Tolerance: Even if the entire cluster goes down, the checkpoint log allows Spark to resume exactly where it left off, regardless of what Kafka thinks.

Note: Because Spark owns the offsets, enable.auto.commit is set to false by default. Overriding this manually usually leads to chaos.

4. The Write Path: Producers, Buffers, and Flushes

Kafka Producer Architecture

Similar to how Spark interacts with Kafka through the Kafka Consumers(InternalKafkaConsumerPool) for reading, it interacts with the Kafka through the Kafka Producers(InternalKafkaProducerPool) for writing. All tasks do not necessarily write to a single producer, but they also don’t create a new one for every task. Instead, Spark uses a Producer Pool, very similar to the Consumer Pool we discussed earlier.

The Producer Pool (Caching the Writers)

On each Executor, Spark maintains a cache of Kafka Producers.

• The Key: Producers are cached based on their configuration (e.g., bootstrap.servers, retries, acks).
• The Shared Instance: If all your tasks are writing to the same Kafka cluster with the same settings, they will typically share a single KafkaProducer instance per Executor.
• Why share? Unlike Consumers, Kafka Producers are thread-safe. A single producer can handle multiple threads (tasks) calling .send() simultaneously. It internally manages its own buffers and network I/O.

How the Flow Works Under the Hood

1. Task Starts: Multiple tasks run in parallel on an Executor.
2. Request Producer: Each task asks the local InternalKafkaProducerPool for a producer that matches its configuration.
3. Concurrent Writing: Since the producer is thread-safe, Task 1, Task 2, and Task 3 all call producer.send(record) at the same time.
4. Internal Batching: The single Producer instance collects records from all these tasks into its internal memory buffers (grouped by Kafka partition).
5. Background Send: The Producer’s own I/O thread sends these batches to Kafka brokers independently of the Spark tasks.

When are there multiple Producers?

You would see more than one producer on a single Executor only if:

• Different Configs: You are writing to two different Kafka clusters in the same job.
• Different Security: One task needs SSL while another uses Plaintext.
• Exceeding Cache: If you have a very complex setup and the spark.kafka.producer.cache.timeout triggers, old producers might be closed while new ones are opened.

The Delivery Guarantee

Spark’s Kafka sink provides At-least-once delivery.

• Why? If a task sends data to Kafka but the Spark job crashes before the checkpoint is updated, Spark will re-run that task upon restart. This sends the same data to Kafka a second time.
• The Fix: To achieve true exactly-once, you generally need an idempotent downstream sink.

Why are Producers thread safe while Consumers are not?

The design of Kafka clients centers on a fundamental trade-off: Producers are “fire-and-forget” aggregators, while Consumers are “state-sensitive” managers.

The Producer: A Shared “Outbox”

The KafkaProducer is thread-safe because its primary role is simply adding data to a buffer.

• Concurrent Buffering: When multiple threads call send(), they drop messages into a local memory accumulator. Using concurrent data structures (like CopyOnWriteMap and synchronized Deques), the producer allows hundreds of threads to append data simultaneously without conflict.
• Background I/O: A dedicated internal thread independently picks up these batches and handles the network transmission.

The Consumer: A Sequential “State Machine”

The KafkaConsumer is NOT thread-safe because it must maintain a precise, moving "cursor" of its progress. Multi-threading would break three critical areas:

• Offset Integrity: If Thread A reads message 10 and Thread B reads message 11, a “race condition” occurs. If Thread B commits offset 11 first but Thread A fails its task, message 10 is lost to the system forever.
• TCP & Heartbeats: The consumer uses a single socket to talk to brokers and send “I’m alive” heartbeats. Multiple threads trying to write to the same socket would mangle the data packets.
• Partition Ownership: Kafka uses “Group Rebalancing” to assign partitions. If Thread B triggers a poll() that results in a rebalance, the partition Thread A is currently processing might be revoked and given to another machine, leading to "zombie" processing.

Similar to Consumer caching we do producer caching, the configuration details and tweaking values can be used based on this link below.

[embed]Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) Structured Streaming integration for Kafka 0.10 to read data from and write data to Kafka. For Scala/Java applications…spark.apache.org

Conclusion: Bringing Spark and Kafka Together

At a glance, a Spark–Kafka pipeline looks simple: read data from Kafka, process it in Spark, and write it back. But under the hood, it’s a well-coordinated interaction between two powerful distributed systems.

Spark takes care of planning, parallel execution, and fault tolerance, while Kafka handles durable storage and high-throughput data movement. The integration between them is designed in a way that keeps both systems loosely coupled but deeply aligned.