ScyllaDB and Discord. A Migration Success Story with Trillions of Messages.

Seven years ago Jason Citron and Stanislav Vishnevskiy joined hands together to create a platform, that the world would remember as the future of the social communication community, Discord. Discord runs on Windows, macOS, Android, iOS, iPadOS, Linux, and in web browsers. As of 2021, the service has over 350 million registered users and over 150 million monthly active users.

This isn’t for the faint of the heart though because storing and managing a trillion messages per day is not a joke. However, this is where their remarkable journey into data architecture started. They started from MongoDB, and migrated to Cassandra hoping that it’ll put their miseries to an end but that was just the beginning. At the beginning of 2022, it had 177 nodes with trillions of messages and things were getting out of hands.

In Cassandra, reads are more expensive than writes. Writes are appended to a commit log and written to an in-memory structure called a memtable that is eventually flushed to disk. Reads, however, need to query the memtable and potentially multiple SSTables (on-disk files), a more expensive operation. Lots of concurrent reads as users interact with servers can hotspot a partition, which they refer to imaginatively as a “hot partition”. The size of their dataset when combined with these access patterns led to struggles for their cluster. This also meant that this architecture had a catastrophic flaw that servers with 2 users would get as many resources as huge servers like Midjourney.

Enters ScyllaDB, a Cassandra-compatible database written in C++. Its promise of better performance, faster repairs, stronger workload isolation via its shard-per-core architecture, and a garbage collection-free life sounded quite appealing. To write a migration tool they wrote it from scratch in Rust because using existing tools the migration time for that huge data was 3 months.

Rust touts fearless concurrency as one of its main benefits — the language should make it easy to write safe, concurrent code. Its libraries also were a great match for what they were intending to accomplish. The Tokio ecosystem is a tremendous foundation for building a system on asynchronous I/O, and the language has driver support for both Cassandra and ScyllaDB. And by the way migration was done in 9 days! 3.2 million messages per second. Let that sink in. Not like Elon Musks’ sink!

Whats the difference?

They came down from 177 cassandra nodes to 72 ScyllaDB nodes Latency to fetch went down from 40–125ms to 15 ms Message insert went from 5–70ms to consistent 5 ms.

What’s the lesson? That as much as data engineers love to play with novel architectures and simply add nodes to avoid work, thoughtful research and brave initiative to implement changes is what matters in the day. I bet they are saving millions of dollars just by those changes. One of the many success stories of engineering success!