The hotspot key problem

We know about Consistent Hashing and the problem it solves. For those who are unaware of Consistent Hashing in a nutshell — it’s a very powerful way to hash keys to servers, and helps to rebalance keys to servers when the number of servers increase or decrease. For more information check out the Wikipedia page about it.

Consistent Hashing

Suppose we have a distributed cache of Twitter and say Elon Musk tweets. We know that hundred of thousands of people would want to access his account — hence would be hitting the server where his key is present — which is because of Consistent Hashing. This is an example of a “Hot-key”.

Even though consistent hashing does help in distributing keys evenly through virtual nodes, it doesn’t mitigate the whole problem.

These hot keys emerge when popular items drive massive requests to a single hash ring position. Even though virtual nodes spreading the physical server points, extreme skew can cascade. Virtual nodes are not that effective in handling keys generating more than 10 times the normal load, traffic can generate due to hash collisions.

One server getting all the hits due a hot key

This will definitely cause the server to overload, cause performance degradation, latency or system crashes.

Let’s talk about the solutions now. We can replicate hot keys which distributes read traffic across multiple nodes holding copies of the same key, reducing load on any single shard. This is very cool, as it boosts read throughput and availability. The drawback is that every write to a replicated key must update all N copies, multiplying write operation to N times. So not so ideal for high-write hotspots compared to read-heavy hotspots.

Another solution is called ‘Salting’. Here we append random suffixes to a single hot key , which spreads it across multiple physical shards. This balances both read and writes for hotspots without making any system wide changes. The drawback of this approach is that it amplifies the read latency, (queries all K sub-keys, then aggregate/choose one) , also increases the write throughput since all K shards need to be updated. Below is a sample code snippet on how we can do that —

# Write
salted_keys = [f"user:123#{i}" for i in range(8)]
for sk in salted_keys: kv_store.write(sk, value)

# Read  
results = [kv_store.read(f"user:123#{i}") for i in range(8)]
return max(results, key=lambda x: x.timestamp)

We can also use multi-layer caching. The L1 and L2 caches sit in front of the hash ring, acting as protective layers to absorb hot key traffic before it hits the overloaded shard. The request first goes to the L1 cache which is on the client side (local in-memory — fastest, ~10μs hit). If it’s a miss then goes to L2 cache ( shared distributed cache like Redis) — checks all shards via consistent hashing. Finally if it’s an L2 miss, then it’s goes to the hash ring backend, but note only 1 to 5 percent of requests reach here for hot keys making it a viable solution.

Logic flow chart for multi-layer caching

While building systems one should always expect skew, and hot keys are a result of that. The real goal is to damage control, not elimination. As per your requirements, you can use the above discussed solutions to check which one works best for you. Ultimately, designing for hot keys is less about choosing a single solution and more about building systems that remain resilient under highly uneven access patterns.