Ditching the Defaults: Building a Blazing Fast Java Off-Heap Cache from Scratch

Introduction

We are in the age of AI and AI Agents. Every day, a new tool drops promising to write our code, orchestrate our systems, and practically do the thinking for us. We are surrounded by massive, distributed infrastructure solutions that can scale to the moon.

So, with all this readily available power, why do we still blindly reach for heavy, off-the-shelf in-memory cache systems the second we need to store some state? Why do we immediately accept the network overhead of an external system, or the bloated memory footprint of a standard library, just to save a few key-value pairs?

Why can’t we just build one for ourselves?

When you are architecting lean, high-performance microservices — perhaps building out the backend for a new SaaS product where keeping infrastructure costs low is just as important as raw throughput — building a bespoke, off-heap cache is not just a fun experiment. It is a highly viable architectural decision.

Thanks to modern Java’s Foreign Function & Memory (FFM) API (Project Panama), escaping the JVM’s heap and managing memory directly is safer and more accessible than ever. Let’s break down how to build a zero-GC, memory-mapped, off-heap cache in Java.

The Problem: The GC Tax and Network Tolls

If you use standard Java collections like ConcurrentHashMap for massive caches, you are going to pay the Garbage Collector (GC) tax. Millions of small object references live on the heap, forcing the GC to traverse them, leading to unpredictable latency spikes and "Stop the World" pauses.

The standard alternative is deploying an external cache like Redis. While incredibly powerful, it introduces network latency, serialization/deserialization overhead, and adds another moving part to your deployment stack.

The goal: We want the speed of local memory without the GC overhead, and we want to do it in pure Java.

The Solution: java.lang.foreign and Memory Layouts

By utilizing Java’s FFM API, we can allocate memory completely outside the JVM’s view. Our custom NativeOffHeapCache achieves this by using memory-mapped files via FileChannel and Arena.ofShared(). This means the OS handles paging the memory, and as a massive bonus, the cache can technically survive application restarts since it is backed by disk.

1. Defining the Memory Layout

Unlike a typical Java object where the JVM dictates the memory layout, we must define exactly how our bytes are structured. We use a flat array of “slots” to represent our hash table.

SLOT_LAYOUT = MemoryLayout.structLayout(
    ValueLayout.JAVA_LONG.withName("key"),          // 8 bytes (Hash of the string key)
    ValueLayout.JAVA_LONG.withName("value"),        // 8 bytes (The actual payload)
    ValueLayout.JAVA_LONG.withName("stringOffset"), // 8 bytes (Pointer to the string pool)
    ValueLayout.JAVA_INT.withName("stringLength"),  // 4 bytes (Length of the string)
    ValueLayout.JAVA_INT.withName("status")         // 4 bytes (Concurrency state)
);

Using VarHandle, we can perform raw, lightning-fast memory reads and writes directly to these specific offsets.

2. The String Pool Problem

Off-heap memory loves fixed-size data. But cache keys are usually strings of variable lengths.

For that we have a saparate memory segment for string keys and instead of allocating a massive fixed-size block for every possible key, the cache employs a separate String Pool (strings.dat). When a new string comes in, we atomically grab an offset using a metadata segment:

final long stringOffset = (long) OFFSET_H.getAndAdd(metadataSegment, 0L, keyLength);

STR_OFFSET_H.set(tableSegment, offset, stringOffset);
STR_LENGTH_H.set(tableSegment, offset, keyLength);

MemorySegment.copy(MemorySegment.ofArray(keyBytes),
        0,
        stringMemorySegment,
        stringOffset,
        keyLength);

We copy the string bytes into the pool, and then store only the stringOffset and stringLength in our fixed-size hash table slot. This keeps the main hash table tightly packed and cache-line friendly.

3. Collision Resolution: The Math of Linear Probing

Hash collisions are inevitable. When two keys hash to the same index, we need a strategy to find an empty slot. We use Open Addressing with Linear Probing.

Here is how the index calculation and probing look in the code:

// 1. Calculate initial index
// Bitwise AND with 0x7FFFFFFF strips the negative sign from the hash
long index = (storageKey.hashCode() & 0x7FFFFFFF) % capacity;
long offset = index * byteSize;
int searched = 0;

while (searched < capacity) {
    // ... [Attempt to write or read at current offset] ...

    // 2. Linear Probe: Move to the very next slot
    index = (index + 1) % capacity;
    offset = index * byteSize;
    searched++;
}

Why Linear Probing? While techniques like separate chaining (using linked lists) are common in standard libraries, they are terrible for off-heap memory. Linear probing places colliding items in contiguous memory slots. Modern CPUs love contiguous memory. When the CPU fetches a slot from RAM, it pulls the surrounding bytes into its ultra-fast L1/L2 cache. Because we probe the very next slot (index + 1), it is highly likely that the next slot is already in the CPU cache, resulting in blazing-fast lookups even during collisions.

4. Lock-Free Concurrency via CAS

Locks kill performance. Instead of using synchronized blocks, the cache manages concurrency through a clever state machine using Compare-And-Swap (CAS) operations on the status field.

The states are simple:

• 0: Empty
• 1: Valid / Ready to read
• 2: Currently being written to

When a thread wants to put data, it finds an index and attempts to atomically claim the slot:

if (STATUS_H.compareAndSet(tableSegment, offset, (int) 0, (int) 2)) {
    // Slot claimed! Write the key bytes to the pool, set the values...
    STATUS_H.setRelease(tableSegment, offset, (int) 1); // Publish the data
}

If another thread is reading (get), it simply checks if the status is 1. If it is 2, it treats it as empty or skips it, preventing dirty reads without ever blocking.

The Road Ahead: From Concept to Production

It is important to note that the implementation above is a proof-of-concept. It brilliantly demonstrates the core mechanics of FFM, CAS, and memory layouts, but a production-ready cache needs a few more features:

• Eviction and TTL (Time-To-Live): Currently, the cache just fills up. To make it a true cache, you would add an expirationTime (another 8 bytes) to the SLOT_LAYOUT. During a get, you check if the current time exceeds the expiration; if so, you treat it as a cache miss and reset the status to 0.
• Deleting Keys (Tombstoning): With linear probing, you cannot simply delete a key and set its status back to 0. Doing so would break the probe chain for other keys that collided and were placed further down the line. You must introduce a new "Tombstone" status (e.g., 3), which tells the get operation to keep searching, but tells the put operation that the slot is safe to overwrite.
• Storing Complex Objects: Right now, the cache stores a primitive long value. To store complex Java objects or JSON payloads, you would treat the values exactly like the String keys: serialize the object into a byte array (using Jackson, Protobuf, or Kryo), write it to a dedicated "Value Pool" memory segment, and store the valueOffset and valueLength in the main hash table slot.
• Load Factor Management: Linear probing degrades sharply if the cache gets too full (clusters merge). In production, you would monitor the load factor and potentially trigger a background thread to allocate a larger memory segment and rehash the data.

Why Own Your Infrastructure?

Writing your own infrastructure components isn’t about reinventing the wheel; it’s about building exactly the vehicle you need.

By understanding and implementing a system like NativeOffHeapCache, you achieve:

1. Zero GC Pressure: Millions of records can sit in memory, and the JVM won’t even blink.
2. Zero Network Latency: No HTTP or TCP round trips. It’s as fast as an L3 cache hit.
3. Absolute Control: You dictate the exact memory footprint and features without external dependencies.

In an era where we are quick to outsource our problem-solving to massive enterprise frameworks, there is still immense value in getting close to the metal. Understanding memory layouts, concurrency primitives, and OS-level memory mapping makes you a better system architect. Sometimes, the most efficient solution is the one you craft yourself.

Here is the code if you wanna check it out fully: Off-Heap-Caching