Kafka Offsets

Before We Talk About Kafka Offsets…

The word offset didn’t originate with Kafka. In fact, it’s a fundamental concept in computer science that appears in many areas of software engineering.

At its core, an offset answers a very simple question:

“How far away is this from the starting point?”

The starting point could be the beginning of an array, a memory address, the start of a file, or the first message in a Kafka partition. The idea is always the same.

Let’s look at a few examples.

1. Arrays

Imagine you have the following array:

Index:    0    1    2    3    4
Value:   15   22   31   48   67

Suppose you want to retrieve the value 48.

You could imagine searching through the array one element at a time…

15 no
22 no
31 no
48 yes!

But arrays don’t work that way.

Instead, the computer already knows that 48 is three positions away from the beginning of the array.

The offset is simply the distance from the beginning of the array. Since 48 is the fourth element, its offset is 3 (remember, arrays are zero-indexed).

2. Memory Addressing

Now imagine the computer’s memory.

Memory is simply a very large collection of numbered locations.

Address
1000
1001
1002
1003
1004
1005
...

Suppose an array begins at address 1000.

Each integer occupies 4 bytes.

Address
1000   First Integer
1004   Second Integer
1008   Third Integer
1012   Fourth Integer

If you want the fourth integer, the processor doesn’t memorize address 1012.

Instead, it thinks like this:

Base Address = 1000

Offset = 12 bytes

1000 + 12 = 1012

Notice something important.

The processor doesn’t really care that the data starts at 1000.

Tomorrow it might start at 5000.

The same calculation still works.

5000 + 12 = 5012

The base address changes while the offset stays exactly the same.

That’s one of the reasons offsets are so useful, they describe a position relative to a known starting point rather than depending on an absolute location.

Figure 1: Memory addresses are calculated, not memorized. The CPU computes the offset (index × element size), adds it to the base address, and reads the value stored at the resulting memory location.

3. Files

A file is simply a long sequence of bytes.

Imagine a text file containing one million bytes.

Byte Position
0
1
2
3
4
...
999999

Suppose a program wants to read information beginning at byte 250,000.

One option would be:

Read byte 0…

Read byte 1…

Read byte 2…

…

Read byte 249,999…

That would be incredibly inefficient.

Instead, operating systems allow programs to say:

Move directly to byte offset 250,000.

The operating system immediately jumps to that location.

This is why video players can instantly skip to the middle of a movie and why databases can quickly access parts of very large files without reading everything first.

Again…

The offset simply represents:

How many bytes from the beginning of the file?

4. Pagination

You’ll also encounter the term offset when working with SQL databases.

Suppose you have the following table:

ID   Name
1    Ali
2    Sara
3    Omar
4    Lama
5    Noor
6    Fahad
7    Ahmed
8    Reem

Now imagine your application displays three records per page.

To retrieve the first page, you could write:

SELECT *
FROM Users
LIMIT 3 OFFSET 0;

Result:

Ali
Sara
Omar

The offset is 0, meaning:

Skip 0 rows from the beginning, then return the next 3 rows.

For the second page, you simply increase the offset:

SELECT *
FROM Users
LIMIT 3 OFFSET 3;

Result:

Lama
Noor
Fahad

This time, the database skips the first 3 rows before returning the next 3.

5. Streaming Systems

In distributed messaging systems like Apache Kafka, an offset represents the position of a message within a partition.

A Kafka topic is not one long sequence of messages. Instead, each topic is divided into one or more partitions, and each partition maintains its own independent sequence of offsets.

For example, suppose we have a topic named orders with two partitions.

topic: orders

Partition 0

Offset   Event

0        Order Created
1        Payment Received
2        Driver Assigned

----------------------------

Partition 1

Offset   Event

0        Order Created
1        Payment Received
2        Order Cancelled

Notice that both partitions contain messages with offsets 0, 1, and 2.

This is perfectly normal because offsets are only unique within a single partition.

To uniquely identify a message, Kafka uses:

Topic + Partition + Offset

For example:

orders
Partition 0
Offset 2

is a completely different message from

orders
Partition 1
Offset 2

Why does Kafka use partitions?

Partitions allow Kafka to process data in parallel.

Imagine an e-commerce platform receiving thousands of orders every second.

Instead of storing every order in one long log, Kafka distributes them across multiple partitions.

             orders

        ┌──────────────┐
        │ Partition 0  │
        │ Offset 0     │
        │ Offset 1     │
        │ Offset 2     │
        └──────────────┘

        ┌──────────────┐
        │ Partition 1  │
        │ Offset 0     │
        │ Offset 1     │
        │ Offset 2     │
        └──────────────┘

        ┌──────────────┐
        │ Partition 2  │
        │ Offset 0     │
        │ Offset 1     │
        │ Offset 2     │
        └──────────────┘

Each partition grows independently, and each keeps its own offsets.

How consumers use offsets

Suppose a consumer is assigned to Partition 0.

It reads:

Offset 0
Offset 1
Offset 2

Then it remembers:

Last processed offset = 2

If the application crashes and later restarts, it doesn’t reread every message.

Instead, it simply tells Kafka:

“Resume reading Partition 0 starting from offset 3.”

In other words, the offset acts like a bookmark indicating how far that consumer has progressed within that partition.

What if there are multiple consumers?

Within a consumer group, a partition is assigned to only one consumer at a time. This prevents multiple consumers from processing the same messages. However, different consumer groups can consume the same partition independently because each group maintains its own offsets.

Imagine two different applications consuming the same topic.

Topic: orders

Analytics Service

Partition 0 → Offset 105
Partition 1 → Offset 98

Fraud Detection Service

Partition 0 → Offset 81
Partition 1 → Offset 77

Both applications are reading the same messages, but they progress independently because each consumer group stores its own offsets.

One application may already be processing today’s orders, while another is still replaying messages from yesterday.

Neither affects the other.

What if there are multiple topics?

Offsets are also isolated by topic.

For example:

orders

Partition 0
Offset 250

---------------------

payments

Partition 0
Offset 620

---------------------

shipments

Partition 0
Offset 91

Even though all three topics have a Partition 0, their offsets are completely independent.

This means Kafka tracks progress separately for every combination of:

• Topic
• Partition
• Consumer Group

The key idea

An offset is not a global message ID.

It only tells you:

“This is message number N within this specific partition.”

That’s why in Kafka, a message is uniquely identified by the combination:

Topic + Partition + Offset

This design allows Kafka to scale horizontally, process messages in parallel, and let multiple applications consume the same data independently without interfering with one another.

Seeing Offsets in Action

Theory is great, but offsets become much easier to understand when you actually see them.

We’ll use a simple Kafka topic called orders and produce a few messages.

Order Created
Payment Received
Driver Assigned
Order Delivered

Example 1 — A Single Consumer

Create a topic with one partition.

Start one consumer.

Consumer A

Received:
Order Created
Payment Received
Driver Assigned
Order Delivered

Now look at the offsets.

Partition 0

Offset 0  Order Created
Offset 1  Payment Received
Offset 2  Driver Assigned
Offset 3  Order Delivered

If Consumer A stops after processing Offset 2, Kafka remembers something like:

Consumer Group: orders-group

Partition 0 → Offset 2

When the consumer starts again, it resumes from:

Offset 3

instead of reading every message again.

Example 2 — Multiple Consumers in the Same Consumer Group

Now create a topic with three partitions.

orders

Partition 0
Partition 1
Partition 2

Start three consumers in the same consumer group.

Consumer A

Partition 0

--------------------

Consumer B

Partition 1

--------------------

Consumer C

Partition 2

Notice that each consumer receives a different partition.

No message is processed twice.

If Consumer B crashes, Kafka automatically reassigns Partition 1 to another consumer.

Example 3 — Different Consumer Groups

Now start another application.

Consumer Group 1

analytics-group

Consumer Group 2

fraud-group

Both subscribe to the same topic.

orders

Now both receive every message.

analytics-group

Offset 0
Offset 1
Offset 2
Offset 3

fraud-group

Offset 0
Offset 1
Offset 2
Offset 3

But notice something important.

Kafka stores offsets separately.

analytics-group

Partition 0 → Offset 103

fraud-group

Partition 0 → Offset 27

The fraud service may still be replaying old events while the analytics service is already processing today’s orders.

Neither affects the other.

Example 4 — Multiple Topics

Suppose your application subscribes to three topics.

orders

payments

shipments

Kafka tracks offsets independently for every topic.

orders
Partition 0 → Offset 152

payments
Partition 0 → Offset 83

shipments
Partition 0 → Offset 17

Even though every topic has a Partition 0, the offsets are completely unrelated.

Looking at Offsets with Code

Everything we’ve discussed so far can be observed by running a few simple Python programs. I’ve put together a small GitHub project that demonstrates each offset scenario step by step. Clone the repository, start Kafka with Docker, and follow the examples to see offsets in action.

GitHub Repository: https://github.com/ShouqSaadRu/Kafka-Offsets

Although the term offset appears in many areas of computer science, the idea is always the same: it represents a position relative to a known starting point.

• In arrays, it’s the number of elements from the beginning.
• In memory, it’s the number of bytes from a base address.
• In files, it’s the number of bytes from the start of the file.
• In SQL pagination, it’s the number of rows to skip before returning the next set of results.
• In Kafka, it’s the position of a message within a partition.

Next in this series: If offsets tell Kafka where to continue reading, how does a streaming engine like Spark or Flink recover its entire processing state after a crash? That’s where checkpoints come in.