🚀 Understanding Cassandra: The Engineering Trade-Offs Behind Massive Scale

A few weeks ago, I found myself asking a question that initially sounded simple: why do companies choose Cassandra when mature databases like PostgreSQL already exist?

As someone who has spent most of his career building backend systems around relational databases, my first instinct was to think in terms of transactions, joins, normalisation, indexes, and consistency guarantees. PostgreSQL has powered some of the world’s most critical applications for decades. It offers ACID transactions, MVCC, sophisticated query planning, rich indexing strategies, and an ecosystem that has evolved through years of production experience. Whenever I think about users, orders, products, payments, subscriptions, inventory, or CRM systems, PostgreSQL feels like the natural choice.

The deeper I dug into Cassandra, however, the more I realised I was asking the wrong question.

Cassandra is not trying to be a better PostgreSQL.

It is solving a completely different distributed systems problem.

The turning point for me came when I stopped thinking about databases as storage systems and started thinking about them as collections of engineering trade-offs. PostgreSQL optimises around transactional correctness, relational modelling, and query flexibility. Cassandra optimises around horizontal scalability, fault tolerance, write throughput, and availability across large distributed environments. Once I viewed Cassandra through that lens, almost every architectural decision started making sense.

The first concept that fundamentally changed my understanding was Cassandra’s use of consistent hashing. In many traditional database architectures, data placement is often something we worry about later through partitioning or sharding. Cassandra treats data distribution as a first-class concern from day one. Every partition key is hashed into a token using a partitioner such as Murmur3Partitioner. These tokens are organised within a logical token ring, which is Cassandra’s implementation of consistent hashing. Every node owns a range of tokens, and data ownership is determined by where the token falls on the ring.

What fascinated me was that scaling becomes a natural extension of the architecture. When a new node joins the cluster, it takes ownership of a portion of the token space and the corresponding data gradually rebalances itself. Unlike traditional sharding strategies where engineers often need to manually redistribute data, Cassandra uses consistent hashing to minimise movement while preserving balance across the cluster. The more I studied this design, the more I appreciated how much operational complexity it removes.

Understanding the token ring naturally led me to the role of the coordinator node. One of the most interesting aspects of Cassandra is that any node can accept client requests. There is no permanent leader or master node. When a request arrives, the receiving node temporarily becomes the coordinator. Its responsibility is to determine which replicas own the requested partition, forward the request, gather responses, apply consistency rules, and return the result to the client. The coordinator may not even store the data itself. This design eliminates a common bottleneck found in many distributed systems where a single leader becomes responsible for coordinating all activity.

The write path is where Cassandra’s architecture becomes particularly elegant. When a write reaches a replica node, Cassandra immediately writes the operation to a Commit Log for durability and stores the data in a MemTable in memory. The Commit Log ensures that data can be recovered even if the node crashes moments later. The MemTable provides a highly efficient in-memory structure for accepting writes without constantly touching disk. Once the MemTable reaches a threshold, it is flushed to disk as an immutable SSTable.

Initially, SSTables seemed like just another storage format. The more I explored them, the more I realised they are the foundation of Cassandra’s performance characteristics. Unlike PostgreSQL, Cassandra rarely updates data in place. SSTables are immutable. Once written, they never change. This decision is rooted in the principles of Log Structured Merge Trees (LSM Trees). Instead of repeatedly modifying existing disk pages, Cassandra accumulates writes in memory and periodically converts them into sequential disk operations. Since modern storage systems are significantly more efficient at sequential writes than random updates, Cassandra can achieve remarkable write throughput.

Of course, every architectural decision introduces a trade-off. The moment I understood Cassandra’s read path, I realised where the complexity had moved.

Imagine a record being updated ten times. Since SSTables are immutable, different versions of that record may now exist across multiple SSTables created at different points in time. When a read request arrives, Cassandra cannot simply open a single file and retrieve the answer. Instead, it must determine which SSTables might contain the requested data and then reconcile the latest version.

This is where Bloom Filters become one of the most fascinating parts of the architecture. Every SSTable maintains a Bloom Filter that allows Cassandra to determine whether a partition could exist within that file. If the Bloom Filter indicates that the partition definitely does not exist, Cassandra skips the SSTable entirely. If it indicates a possible match, Cassandra continues searching. Bloom Filters can produce false positives but never false negatives, making them incredibly effective at reducing unnecessary disk I/O while maintaining correctness.

As I continued exploring, another important realisation emerged. If Cassandra keeps creating SSTables indefinitely, read performance will eventually degrade because more files need to be examined. This is where compaction enters the picture. Compaction continuously merges SSTables, removes obsolete row versions, processes tombstones, rebuilds indexes, and creates larger consolidated files. What impressed me was that compaction is not simply a maintenance task. It is an essential component of the storage engine itself. Cassandra’s write efficiency is only possible because the system is willing to spend resources reorganising data later.

The more I learned about Cassandra, the more I realised that schema design requires an entirely different mindset compared to relational databases. In PostgreSQL, I typically begin by identifying entities and relationships. I create normalised tables and rely on joins to reconstruct information when needed. Cassandra forces a different conversation. Instead of asking how entities relate to one another, the first question becomes: what queries must be fast?

Consider a ride-sharing platform. In PostgreSQL, I might create Drivers, Riders, Trips, Payments, and Locations tables connected through foreign keys. If I need to retrieve trip information, joins can reconstruct the required view of the data. In Cassandra, I would start with access patterns instead. Suppose the application needs to display all trips for a driver during the last thirty days, retrieve the latest trip for a rider, show active trips within a city, and generate regional reporting data. Each of these access patterns may result in a different table structure optimised specifically for that query. Data duplication becomes intentional because predictable query performance is often more valuable than perfect normalisation at scale.

For example, a Cassandra table designed for retrieving trips by driver might use a partition key such as DriverId and a clustering key based on TripTimestamp. That allows Cassandra to efficiently retrieve all trips for a specific driver without scanning the entire dataset. If another query requires retrieving trips by city, a separate table may be created using CityId as the partition key. The same trip data may exist in multiple tables because Cassandra optimises for read efficiency rather than storage normalisation.

Another area that fundamentally changed my understanding was replication and consistency. Cassandra stores multiple copies of data according to a configured replication factor. If the replication factor is three, each partition exists on three different nodes. What makes Cassandra particularly interesting is that consistency becomes configurable. With a consistency level of ONE, the coordinator only needs a single replica to acknowledge the operation. With QUORUM, a majority of replicas must respond. With ALL, every replica must acknowledge the request. This flexibility allows engineers to make explicit trade-offs between latency, availability, and consistency depending on business requirements.

Maintaining consistency across replicas introduces another fascinating set of mechanisms. Nodes continuously exchange cluster state information through the Gossip Protocol, allowing them to remain aware of topology changes and node health. Hinted Handoff temporarily stores writes intended for unavailable replicas and replays them later when those replicas recover. Read Repair helps synchronise replicas when inconsistencies are detected during reads. Anti-Entropy Repair uses Merkle Trees to compare data across replicas and repair divergence over time. What impressed me most was that Cassandra does not rely on a single mechanism to maintain consistency. Instead, multiple distributed systems techniques work together to preserve availability while ensuring eventual convergence.

One detail I found particularly interesting is that Cassandra generally avoids consensus algorithms such as Raft for normal operations because running distributed consensus for every write would significantly reduce throughput. However, when stronger guarantees are required, Cassandra provides Lightweight Transactions (LWT), which internally use Paxos. This means strong consistency is available when necessary, but the entire system is not forced to pay the performance cost for every operation.

The deeper I went into Cassandra’s architecture, the more I realised that it is not simply a database. It is a carefully assembled collection of distributed systems concepts working together toward a common goal. Consistent hashing distributes ownership. Coordinator nodes distribute responsibility. LSM Trees optimise writes. Bloom Filters reduce read amplification. Compaction manages immutable storage. Gossip maintains cluster awareness. Repair mechanisms preserve consistency. Every subsystem exists because of a deliberate trade-off.

My biggest takeaway from learning Cassandra is that PostgreSQL and Cassandra are not competing technologies. They represent different philosophies of system design. PostgreSQL optimises around transactions, consistency, relationships, and query flexibility. Cassandra optimises around distribution, fault tolerance, horizontal scalability, and sustained write throughput.

The longer I work in software engineering, the more convinced I become that architecture is rarely about choosing the best technology. It is about understanding the constraints of the problem, understanding the trade-offs of the available solutions, and selecting the technology whose assumptions most closely align with the workload you are trying to build.

For me, that was the real lesson behind learning Cassandra.