The Cheapest Kafka Consumer Is One That Doesn’t Read From Kafka

For non-real-time workloads spread across multiple AZs, the biggest Kafka cost savings usually come from removing those consumers from Kafka entirely.

In a multi-AZ Kafka deployment, every additional consumer group can add another broker-mediated cross-AZ read path. If those consumers don’t need low-latency streaming, moving them off Kafka entirely is often the cleanest way to cut the data transfer bill by reducing cross-AZ data transfer cost.

The Pattern — Move Reads Off Kafka

One Kafka consumer writes to object storage, and downstream readers read from there instead of from Kafka. That removes repeated cross-AZ reads from the Kafka path and shifts the fan-out boundary to S3.

One practical implementation is writing to Iceberg tables on S3 via Flink or Spark Structured Streaming, then letting downstream consumers query those tables directly.

Before: each consumer group adds another cross-AZ billing stream. After: one Kafka consumer writes to S3, and the readers move off Kafka, so the fan-out multiplier disappears.

The Constraint

This pattern only works when consumers can tolerate batch latency. Files appear in object storage with a delay, so it is not a fit for near-real-time data, compacted topics, or workloads that cannot tolerate writer lag.

Why KIP-405 (Tiered Storage) Doesn’t Solve Cross-AZ Costs

KIP-405 moves cold data to S3 to reduce broker disk usage, but the broker stays in the read path. That means cross-AZ egress still applies when consumers need the data. Move Reads Off Kafka removes the broker from the read path entirely, so consumers read object storage directly instead of from Kafka.

KIP-405 keeps the broker in the read path — cross-AZ egress still applies. Move Reads Off Kafka (using S3 indirection) removes it entirely. Same S3, opposite effect on the network bill.

The Economic Breakeven Point

Introducing a Move Reads Off Kafka architecture trades network savings for operational complexity and batch latency.

The savings scale with fan-out: each additional consumer group adds more cross-AZ traffic. The tradeoff is that you are now paying for a writer, table management, and batch latency. The pattern becomes interesting once fan-out is high enough that repeated cross-AZ reads materially affect the bill.

The primary signal is fan-out above 3× with consumers that can tolerate batch latency.

Key Takeaways

Separate real-time consumer groups from analytical ones. For analytical groups on high fan-out topics:

1. Verify that S3 access goes through a VPC Gateway Endpoint. Without it, the economics usually fall apart.
2. Estimate current Kafka network cost from throughput and fan-out before changing the architecture.
3. Prefer a table format over raw S3 files. Iceberg via Flink or Spark Structured Streaming is a common implementation because it handles metadata, partitioning, and read-time usability better than ad hoc file dumps.
4. Monitor writer lag and align Kafka retention accordingly. If the writer falls behind long enough, the source data can expire before it reaches S3.

The architectural insight is simple: the cheapest consumer is the one you never put on Kafka in the first place.

Part 3 of the Kafka Network Cost series. Part 1: cross-AZ topology and fan-out attribution. Part 2: Kafka’s Real Compression Problem Is Batch Depth.