Kafka Terminology Breakdown: A Step-by-Step Guide from Basic to Advanced

Apache Kafka is a distributed event streaming platform, widely used for building real-time data pipelines and streaming applications. This blog will guide you through Kafka terminologies from basic to advanced levels.

My articles are open to everyone; non-member readers can read the full article by clicking this **link**.

Kafka

1. Basic Kafka Terminologies

1.1 Kafka Broker

A Kafka broker is a server that receives and stores messages from producers and serves these messages to consumers. A Kafka cluster is made up of multiple brokers, which work together to provide scalability and fault tolerance.

1.2 Topic

A topic is a category or feed name to which records are sent. Topics are further divided into partitions. Each partition is an ordered, immutable sequence of records.

1.3 Partition

Partitions are segments within a topic that divide the data into smaller, manageable chunks. Kafka distributes partitions across brokers to allow parallelism in processing. The order of records is guaranteed within a partition.

1.4 Producer

A producer is an application that writes data to Kafka topics. Producers push messages to specific topics using a partition key, which determines which partition the message will go to.

1.5 Consumer

A consumer reads data from Kafka topics. Kafka consumers belong to a consumer group; each consumer within a group reads from a different partition, allowing parallel consumption and processing.

1.6 Consumer Group

A consumer group is a set of consumers that share the load of reading data from partitions. Kafka ensures that each partition of a topic is consumed by exactly one consumer within the group, providing scalability and fault tolerance.

1.7 Offset

Offsets are unique identifiers assigned to each record within a partition. They help consumers keep track of the messages they have read, ensuring that they can resume processing from where they left off in case of a failure.

1.8 Replication

Kafka replicates partitions across multiple brokers to achieve fault tolerance. The replication factor determines how many copies of each partition exist in the cluster. A replication factor of 3 is typical, ensuring that the data survives the failure of up to two brokers.

2. Intermediate Kafka Terminologies

2.1 Zookeeper

Kafka uses Zookeeper for managing and coordinating the Kafka brokers in the cluster. It maintains metadata about topics, partitions, and consumers. However, recent Kafka versions are moving towards KRaft (Kafka’s own consensus mechanism) to remove the dependency on Zookeeper.

2.2 Log Compaction

Log compaction is a mechanism for retaining only the most recent update for each key in a partition. This is useful for use cases where maintaining the latest state of data is important (e.g., user profiles).

2.3 Retention Policy

Kafka allows you to configure how long data is retained in topics. Retention can be based on time (e.g., 7 days) or size (e.g., 100 GB). Once the retention period is exceeded, Kafka deletes old records.

2.4 Acknowledgments (acks)

This producer setting determines the durability guarantee when writing records to Kafka:

• **acks=0**: The producer does not wait for any acknowledgment.
• **acks=1**: The producer waits for the leader broker to acknowledge.
• **acks=all**: The producer waits for all replicas to acknowledge (highest durability).

2.5 In-Sync Replicas (ISR)

These are replicas that are currently in sync with the leader partition. The min.insync.replicas setting defines the minimum number of replicas that must acknowledge a write for it to be considered successful.

2.6 Schema Registry

A schema registry manages the schema (structure) of data sent to Kafka. This ensures compatibility between producers and consumers, supporting schema evolution and maintaining data consistency.

https://www.redpanda.com/blog/schema-registry-kafka-streaming

2.7 Kafka Connect

Kafka Connect is a tool for streaming data in and out of Kafka. It enables integration with various systems (e.g., databases, file systems) without the need to write custom code.

https://www.linkedin.com/pulse/kafka-connect-why-exists-how-works-chandan-prakash/

2.8 Kafka Streams

Kafka Streams is a library for building real-time processing applications. It allows developers to transform, filter, and aggregate data as it flows through Kafka topics.

3. Advanced Kafka Terminologies

3.1 Exactly-Once Semantics (EOS)

Exactly-once semantics guarantee that a message is processed only once, even in case of failures. Kafka achieves this through idempotent producers and transactional consumers, ensuring data consistency.

https://blog.bytebytego.com/p/at-most-once-at-least-once-exactly

3.2 KRaft (Kafka Raft Consensus)

KRaft is Kafka’s replacement for Zookeeper, built to handle metadata management internally. This makes Kafka more scalable and reduces the operational complexity of managing Zookeeper clusters.

3.3 Rack Awareness

Rack awareness is a feature that allows Kafka to distribute replicas of partitions across different racks or availability zones, ensuring fault tolerance in case of a rack or AZ failure.

3.4 Quotas

Kafka supports quotas to manage resource usage by producers and consumers. You can set limits on bandwidth usage, message rates, and other metrics to prevent any application from overwhelming the cluster.

3.5 Monitoring and Metrics

Kafka provides metrics through JMX (Java Management Extensions). Tools like Prometheus, Grafana, and Confluent Control Center can be integrated for monitoring broker performance, consumer lag, and other key metrics.

3.6 Log Segment

A log segment is a portion of a partition’s log file. Kafka splits logs into segments to manage memory and disk efficiently, allowing for faster reads and writes.

3.7 Message Compression

Kafka supports message compression using codecs like Snappy, GZIP, and LZ4 to reduce the size of messages, saving bandwidth and storage space while increasing throughput.

If you are someone reading preparing for kafaka and getting started with ,I would suggest to practice scenario based questions on kafka to further deep dive into its concepts. You can checkout https://topmate.io/archana_goyal/1479665