The Kafka reference with config diffs, protocol changes, and tuning advice

The Kafka Options Explorer contains the context the official docs skip: what really changed, at the API and protocol level.

What does Produce exactly? https://kafka-options-explorer.conduktor.io/wire-format/Produce/response/v13/

Most Kafka documentation answers one question at a time and stops there. “What does fetch.max.bytes do?” — here’s the description, default is 52428800, have a nice day.

What it doesn’t tell you: whether that default changed in Kafka 4.0. Whether the equivalent setting in librdkafka is named the same thing. Whether setting it to 10MB for a low-latency consumer is safe or likely to hurt you.

Kafka Explorer is a free reference built to answer those follow-up questions. Six tools on one site, no account needed. Here’s what’s actually useful about it.

Config diffs all the way down

If you’ve maintained Kafka infrastructure across multiple releases, you’ve hit this: a config you’ve been using for years changes default in a new version, silently, and you only notice too late when some behavior have changed.

Configuratoin Explorer

The Configuration Explorer tracks every broker, producer, consumer, Streams, and Connect property from Kafka 0.7 through 4.2. Pick any two versions and you see exactly what changed: new configs that appeared, old ones deprecated, defaults that changed.

e.g.: linger.ms had a default of 0ms from the beginning. Kafka 4.0 changed it to 5ms. If you’re running a latency-sensitive producer and upgraded to 4.0 without noticing this, you just added up to 5ms to every produce call.

You can search by property name, filter by component (producer, broker, connect, etc.), and jump directly to the version where a config was introduced. For properties with a lot of history, the timeline view shows the full lifecycle in one screen.

The wire format, byte by byte

This is what your client need to send to Kafka when you are consuming data (Fetch):

https://kafka-options-explorer.conduktor.io/wire-format/Fetch/response/v18/

This one takes a minute to appreciate if you’ve never had to implement a Kafka client or debug a binary protocol issue.

Every Kafka API has a precise wire format: a specific byte layout for the request and the response, version by version.

The official protocol documentation has field-level tables and all the details, check here: https://kafka.apache.org/42/design/protocol, but no enough visual for me, and no way to see what changed between versions at a glance.

The Wire Format Visualizer fills that gap. Select an API and a version, and you get the complete byte layout: field names, types, byte offsets, encoding (varint vs fixed-width, flexible vs legacy format). Schema on one side, a concrete example payload on the other.

I’m particularly fan of the Changelog tab. Switch to it and you get a git-diff-style view of what changed between protocol versions, fields added in green, removed in red, nullability changes in yellow. For instance, for Fetch Request/Response have quite an history! https://kafka-options-explorer.conduktor.io/wire-format/Fetch/changelog/

If you’re implementing a client, debugging a protocol mismatch, or just trying to understand why a broker running version X rejects a request from a client targeting version Y, this is the view you want.

Config Advisor: what should I actually set?

A common question when using Kafka: which configuration should I use for my Kafka client? We try to squeeze the knowledge into The Config Advisor. It may not be the reference, but a recommendation engine, putting the spotlight on certain properties to raise the right questions. (in Java and with librdkafka the forgotten!).

You pick your component (producer, consumer, broker) and your workload profile : low latency, high throughput, durability, cost-optimized. It gives you a prioritized list of configs to tune, with:

• The exact technical tradeoff. For instance: “setting linger.ms=0 eliminates accumulation wait, dispatching a ProduceRequest the moment a record arrives, at the cost of multiplying the number of TCP round-trips by your record rate.”
• A danger level. Each recommendation is marked safe, caution, or dangerous. Setting acks=1 for low latency? Dangerous and the advisor explains why: messages acknowledged but not yet replicated are permanently lost if the leader crashes before followers catch up.
• The librdkafka equivalent. The Java client and librdkafka don’t share property names. e.g.: buffer.memory in Java is queue.buffering.max.kbytes in C (and the unit is KB, not bytes, don’t ask). The advisor maps them automatically so you can apply the same tuning to both clients.

The rest

Beyond those three:

• a KIP Explorer with all the 1258 proposals, each summarize in layman terms (what it does, why it exists) and protocol impact.
• An API Versions Matrix showing which protocol API versions each Kafka release supports , practical when you’re checking client-broker compatibility.
• A Kafka Upgrade Changelog that aggregates config changes, protocol bumps, and related KIPs per version pair. Before upgrading from 3.6 to 4.0, you can pull up exactly what changed: not just release notes prose, but a structured diff.

All of it is at kafka-options-explorer.conduktor.io. Free, no account.