Understanding Cassandra Partitioning & Sharding: The Concept That Finally Made It Click

When I first started reading about Slack’s system design, one line kept appearing:

“Messages are partitioned by channel ID.”

Sounds simple.

But then I started wondering:

• Does Cassandra create millions of partitions?
• Does every partition become a separate file?
• If Slack has 100 million channels, does Cassandra create 100 million storage objects?
• How is this different from PostgreSQL partitioning?
• And where does sharding fit into all this?

It took me a while to realize that the biggest source of confusion wasn’t Cassandra itself — it was the word partition.

Let’s build the intuition from scratch.

The Biggest Misconception

When most people hear the word partition, they imagine something like this:

📁 channel-1
📁 channel-2
📁 channel-3

As if Cassandra creates one folder or one file for every channel.

It doesn’t.

Not even close.

Think About a Python Dictionary

Suppose you have this dictionary:

messages = {
    "channel-1": [...],
    "channel-2": [...],
    "channel-3": [...]
}

How many files exist?

Just one.

The dictionary internally stores millions of keys.

It doesn’t create one file per key.

Cassandra works much more like this than like a filesystem.

Python dictionary analogy

Understanding Cassandra’s Primary Key

Consider this schema:

CREATE TABLE messages (
    channel_id text,
    sequence bigint,
    message text,
    PRIMARY KEY ((channel_id), sequence)
);

This one line explains almost everything.

Let’s break it down.

PRIMARY KEY (
    (channel_id),
    sequence
)

Notice something unusual.

There are two sets of parentheses.

Those parentheses have completely different meanings.

The Partition Key

The first parentheses:

(channel_id)

tell Cassandra:

Keep all rows having the same channel_id together.

Imagine these rows:

Cassandra groups them like this:

Partition: general

1 → Hello
2 → Hi
3 → Bye

--------------------

Partition: random

1 → Welcome
2 → Thanks

Every channel becomes one logical partition.

Drawer/partition visualization

The Clustering Column

Now look at the second part:

sequence

This is called the clustering column.

It tells Cassandra:

Inside each partition, store rows ordered by sequence.

Instead of:

3
1
2

the partition becomes

1
2
3

That’s perfect for chat applications because conversations are naturally ordered.

Ordered messages inside one partition

Why Not Make Both Columns the Partition Key?

Imagine we wrote:

PRIMARY KEY ((channel_id, sequence))

Now every message gets its own partition.

(general,1)

(general,2)

(general,3)

Loading a channel would require reading thousands of different partitions.

That would completely defeat the purpose.

Instead,

PRIMARY KEY ((channel_id), sequence)

creates exactly one partition per channel.

So What Is Actually a Partition?

This is the key realization.

A Cassandra partition is not:

• a file
• a folder
• a table
• a database

A partition is simply:

All rows having the same partition key.

That’s it.

Nothing more.

Does Cassandra Create Millions of Partitions?

Yes.

And that’s completely normal.

Suppose Slack has:

• 50 million channels

Then Cassandra has roughly:

• 50 million logical partitions.

That sounds scary until you realize these aren’t separate storage objects.

Think of a huge Excel sheet:

ChannelSequenceMessagegeneral1Hellogeneral2Hirandom1Welcomerandom2Thanks

Every unique channel represents one logical partition.

The data still lives inside the same table.

Excel sheet grouped by channel

So Where Is The Data Actually Stored?

Internally Cassandra stores data inside SSTables.

Imagine one huge sorted file.

SSTable

channel-1
    msg1
    msg2
    msg3

channel-2
    msg1
    msg2

channel-3
    msg1

There isn’t one file per channel.

Instead, the file simply contains groups of rows sharing the same partition key.

SSTable layout

How Does Cassandra Find The Right Partition?

Suppose Slack asks:

SELECT *
FROM messages
WHERE channel_id='general';

Does Cassandra scan the whole table?

No.

Each SSTable maintains metadata like:

• Partition Index
• Partition Summary
• Bloom Filter

These help Cassandra jump directly to the required partition instead of scanning millions of rows.

Enter Sharding

Everything we’ve discussed so far works even if Cassandra has only one machine.

Now let’s add more nodes.

Node A

Node B

Node C

What happens now?

Cassandra hashes the partition key.

hash(channel_id)

The hash decides which node stores the partition.

For example:

channel-1 → Node A

channel-2 → Node C

channel-3 → Node B

This process is called sharding.

Cluster showing partitions distributed across nodes

Partitioning vs Sharding

This was the biggest realization for me.

Partitioning and sharding are not the same thing.

Instead, they happen one after another.

Grouped into partitions

↓

Partitions distributed across nodes

In other words:

• Partitioning groups related rows together.
• Sharding distributes those groups across machines.

Why Doesn’t Cassandra Use PostgreSQL-style Partitions?

Because PostgreSQL partitioning means something completely different.

In PostgreSQL:

messages

├── messages_2025
├── messages_2026
├── messages_2027

Each partition is an actual physical table.

Every partition has:

• separate storage
• separate indexes
• separate statistics

In Cassandra,

there is still only one table.

The partition is simply a logical grouping of rows.

PostgreSQL vs Cassandra comparison

The Filing Cabinet Analogy

This analogy finally made everything click for me.

Imagine a filing cabinet.

Each drawer represents one Slack channel.

Drawer: #general

1
2
3
4

Another drawer:

Drawer: #random

1
2

The drawer is the partition.

The numbered documents inside are ordered by the clustering column.

Now imagine having millions of drawers.

Those drawers are spread across multiple filing cabinets.

Each filing cabinet represents a Cassandra node.

That’s essentially how Cassandra works.

Filing cabinet analogy for Cassandra

The Mental Model I’ll Remember Forever

Whenever I see this:

PRIMARY KEY ((channel_id), sequence)

I now read it like this:

• channel_id
• Which bucket does this row belong to?
• Which node stores that bucket?
• sequence
• In what order should rows be stored inside the bucket?

That’s it.

Everything else in Cassandra — efficient reads, scalability, SSTables, consistent hashing, and even Slack’s message storage architecture — builds on this single idea.

Sometimes the hardest part of learning distributed systems isn’t the algorithms.

It’s simply understanding what a single overloaded word like partition actually means.