The Brave Engineer’s Guide to Memory: Why Your Cache Needs TTL and Approximated LRU

In high-scale system design, memory is one of the most valuable resources and also one of the most limited. As engineers, the challenge is not to avoid complexity, but to choose the right complexity to build systems that are predictable, fast, and resilient.

While working on a distributed cache system, one of the most important architectural decisions was how to handle memory pressure. At its core, the question was simple:

Should memory be treated as something elastic and expandable, or as a fixed budget that must be carefully managed?

The answer to that question shapes everything from latency to cost to operational stability.

The Crossroads: Eviction or Scaling

When a cache node reaches its memory limit, there are two common paths forward.

1. Dynamic Node Scaling

This approach treats memory as elastic. Once usage crosses a threshold, such as 70 percent, new nodes are added and data is rebalanced across the cluster.

At first glance, this feels safe. No data is lost, and capacity keeps growing.

In practice, it introduces serious trade-offs:

• Infrastructure provisioning is not instantaneous
• Rebalancing data causes traffic spikes
• Latency becomes unpredictable during scale events

For a low-latency cache where P95 response time matters, these pauses are costly.

2. Space-Based Eviction Using TTL and LRU

This approach treats memory as a fixed budget. When the cache is full, less useful data is removed to make room for new entries.

This is the approach we chose.

It keeps performance stable, simplifies operations, and preserves the core purpose of a cache: serving hot, useful data as fast as possible.

The Strategy: Fresh Data and Controlled Memory

To make space-based eviction effective, we combined two techniques: TTL for correctness and an approximated LRU for capacity control.

Lazy TTL Expiration for Data Freshness

Time-To-Live ensures that stale data is never served.

Instead of running background cleanup jobs that scan the cache and consume CPU, the system uses lazy expiration. A key’s TTL is checked only when it is accessed.

• If the entry is still valid, it is returned immediately
• If it has expired, it is deleted on the spot and treated as a miss

This keeps correctness guarantees strong without adding constant background overhead.

Approximated LRU for Capacity Protection

A strict Least Recently Used policy requires tracking every access in a global data structure, often a linked list. In a concurrent environment, especially with structures like ConcurrentHashMap, this quickly becomes a bottleneck due to locking.

To avoid this, the cache uses an approximation.

When the cache is full:

1. A small random sample of keys is selected
2. Their last access timestamps are compared
3. The least recently used key within that sample is evicted

This probabilistic approach delivers nearly the same hit rate as true LRU while keeping read paths lock-free and fast.

Why This Is a Product Decision, Not Just a Technical One

Choosing eviction over scaling reflects how the system is meant to be used.

By treating memory as a constrained resource, the system stays honest. It stores what is actively used and discards what no longer provides value.

This keeps infrastructure costs under control and performance predictable.

Final Thoughts

Chaos is unavoidable in distributed systems. Memory fills up, access patterns change, and workloads shift without warning.

Resilience does not come from pretending resources are infinite. It comes from acknowledging limits and designing for them.

TTL combined with approximated LRU creates a cache that is fast, fair, and realistic about what it can hold. It prioritizes hot data, protects latency, and degrades gracefully under pressure.

Being a brave engineer is not about overengineering. It is about making clear trade-offs and standing by them.

And sometimes, that means letting go of old data so the system can keep moving forward.