Kafka Offsets Explained Like You’re 5

Goal of This Article

In the previous articles, we learned:

• What Kafka is?
• **Producers**
• **Consumers**
• **Topics**
• **Brokers**
• **Partitions**
• **Consumer Groups**

One important question still remains. Imagine a consumer has already processed thousands of messages. How does Kafka remember where that consumer stopped?

Does it start reading from the beginning every time? Thankfully, no. The answer lies in one simple concept: Offsets.

By the end of this article, you’ll understand:

• What an Offset is
• Why Kafka needs Offsets
• How Offsets work
• Offset Commit
• Auto Commit vs Manual Commit
• Where Kafka stores Offsets
• Why different Consumer Groups have different Offsets

Let’s understand it the easy way.

Imagine You’re Reading a Storybook

You start reading your favourite storybook. Today, you stop at Page 57. Before going to sleep, you place a bookmark inside the book. The next day, you open the book. Do you start from Page 1 again? Of course not. You simply open the bookmark and continue from Page 58.

That bookmark is exactly what an Offset is. Kafka uses Offsets as bookmarks so consumers always know where to continue reading.

What Is an Offset?

Simple Definition

An Offset is the position of a message inside a Kafka partition. Think of it as the serial number of every message.

For example:

Partition 0

Offset 0  --> Order Created
Offset 1  --> Payment Completed
Offset 2  --> Invoice Generated
Offset 3  --> Email Sent
Offset 4  --> Order Delivered

Every new message gets the next available Offset. Just like every page in a book has a page number.

Why Does Kafka Need Offsets?

Let’s imagine another example. A teacher asks a student to read a 500-page novel. The student reads until Page 250 and goes home. The next morning,

the teacher asks: “Where will you start?” The student replies: “From Page 251.”

Imagine if there were no page numbers. The student would have to search for the last page every single day. Kafka faces the same problem. Consumers continuously read messages. Without Offsets, Kafka would never know where they stopped.

What Happens Without Offsets?

Suppose a consumer processes three messages.

Offset 0
Offset 1
Offset 2

Suddenly… The application crashes.

If Kafka had no Offsets, after restarting it would read:

Offset 0
Offset 1
Offset 2
Offset 3

again. The same messages would be processed twice. Imagine receiving the same food order confirmation email five times. Not a great user experience. Offsets solve this problem.

What Happens With Offsets?

Now let’s repeat the same example. The consumer processes:

Offset 0
Offset 1
Offset 2

Before shutting down, Kafka remembers:

Last Processed Offset = 2

After restart: The consumer begins from:

Offset 3

Exactly where it left off. Just like reopening a bookmarked page.

What Is an Offset Commit?

Knowing the last processed Offset is useful. But how does Kafka remember it? After processing messages successfully, the consumer tells Kafka:

“I’ve finished processing messages up to Offset 25.”

Kafka stores this information. This process is called an Offset Commit. Think of it as updating your bookmark after finishing another chapter.

Auto Commit vs Manual Commit

There are two ways to save Offsets.

Auto Commit

Kafka automatically saves Offsets after a configured interval.

Read Message
      |
Process Message
      |
Auto Commit Offset

Advantages

• Easy to configure
• Less code
• Good for simple applications

Disadvantages

If the application crashes after committing but before completing business logic, messages may be skipped.

Manual Commit

Here, your application decides exactly when the Offset should be committed.

Read Message
      |
Business Logic
      |
Database Updated
      |
Commit Offset

The Offset is saved only after successful processing.

Advantages

• More control
• Better reliability
• Preferred in production systems

Disadvantages

Requires additional code and careful handling.

The Cool Thing Most Beginners Don’t Know

Many developers think Offsets belong to a Topic. They don’t. Others think Offsets belong to a Partition. Not exactly.

Offsets are tracked per Consumer Group. This means two Consumer Groups can read the same Topic independently. Imagine two students reading the same novel.

Student A reaches:

Page 150

Student B reaches:

Page 75

Both are reading the same book. Both have different bookmarks. Kafka works exactly the same way.

Where Are Offsets Stored?

A common interview question is:

Where does Kafka store Offsets?

Modern Kafka versions store committed Offsets in a special internal topic called:

__consumer_offsets

Consumers don’t manage this topic directly. Kafka updates it automatically whenever Offsets are committed. This allows consumers to restart and continue from the correct position.

Visual Flow

Summary

Let’s recap.

Offset

A bookmark that remembers a consumer’s position inside a partition.

Why Do We Need It?

To allow consumers to continue reading exactly where they left off.

Offset Commit

The process of saving the last successfully processed Offset.

Auto Commit

Kafka saves Offsets automatically. Simple, but offers less control.

Manual Commit

The application saves Offsets after successful processing. Preferred for production systems.

Important Things to Remember

• Offsets are unique within a partition.
• Offsets belong to Consumer Groups.
• Kafka stores committed Offsets in __consumer_offsets.
• Reading a message does not delete it.

If Partitions help Kafka process data in parallel, and Consumer Groups help distribute the workload, then Offsets are what make Kafka reliable by ensuring consumers always know where to continue.

In the next article, we’ll put everything together by building a complete Spring Boot + Kafka application, where you’ll see Producers, Consumers, Partitions, Consumer Groups, and Offsets working together in a real project.