Shrinking the LLM KV Cache by 8x on CPU: The Engineering Behind AdapTQ

If you’ve ever tried running a Large Language Model (LLM) locally on a MacBook, a Raspberry Pi, or a standard consumer CPU, you’ve probably hit the exact same wall I did.

It’s not the model weights that kill your performance. It’s the KV Cache.

Every time an LLM generates a word, it caches the Key (K) and Value (V) representations of that word. This avoids recomputing the entire context window from scratch on the next step. It’s a brilliant optimization — until your conversation reaches 8,000 tokens. Suddenly, that “short-term memory” balloons into Gigabytes of data.

At that point, inference stops being bottlenecked by math (compute) and starts being completely bottlenecked by your RAM speed (memory bandwidth). Your CPU is starving, waiting for data to arrive from memory.

To fix this, I built AdapTQ (Adaptive Streaming Vector Quantization). It’s an open-source, production-grade C++ engine that shrinks the KV cache by up to 8x while actually increasing token generation speed.

Here is the story of how I built it, the mathematical tricks that made it possible, and the low-level C++ engineering that made it fast.

The Problem: The Tyranny of Outliers

When you have too much data, the obvious solution is to compress it. If we can squeeze 16-bit floats (FP16) down to 4 bits, we instantly cut our memory bandwidth requirements by 75%.

But if you blindly apply simple scalar quantization to LLM hidden states, you destroy the model’s intelligence. Why? Outliers.

LLMs naturally develop massive outlier features. A handful of dimensions will have values 100x larger than the rest. If you quantize that vector, those massive outliers stretch the “dynamic range” so wide that all the normal, subtle features get crushed into zero.

The traditional fix is Vector Quantization (VQ) — matching chunks of data to a pre-defined “codebook” of shapes. But traditional VQ is mathematically expensive and far too slow for real-time inference.

The Math Fix: Hadamard Accelerated Rotation (HAR)

I needed a way to tame the outliers before quantizing, without adding massive computational overhead. Enter the Fast Walsh-Hadamard Transform (FWHT).

Before quantizing a vector, AdapTQ rotates it. Don’t let the math intimidate you. Think of this operation like taking a clump of peanut butter (the outlier) and spreading it perfectly evenly across a piece of toast.

By applying a random sign flip and running the FWHT, the massive spike of energy in one dimension is mathematically smeared across all dimensions. The resulting distribution looks beautifully Gaussian.

Because the FWHT requires zero multiplications (only simple additions and subtractions), it runs blisteringly fast on a CPU. Once the outliers are flattened, we can safely apply aggressive 4-bit scalar quantization. The result? We get the high fidelity of Vector Quantization at the speed of simple scalar math.

Engineering the Hot Path: The SIMD Aha! Moment

Having a great mathematical theory is only half the battle. Implementing it in C++ is where things usually break down.

My first implementation was terrible. I compressed the cache successfully, but during the attention calculation, I had to decompress the 4-bit vectors back into 32-bit floats before I could multiply them. The CPU spent more time unpacking data than it saved by fetching less of it.

The Insight: Never dequantize inside the inner loop.

Instead of decompressing the Keys, I pre-computed a Look-Up Table (LUT).

When a new Query token arrives, I rotate it (using the FWHT) and immediately multiply it against the 16 possible values (centroids) of our 4-bit codebook. This creates a tiny LUT.

Inside the actual attention loop, the Key vectors remain deeply compressed in 4-bit chunks. Using AVX2 SIMD intrinsics, I read 8 indices at a time. The CPU simply looks at the 4-bit index, reaches into our LUT, and grabs the pre-computed answer.

By writing a quad-unrolled, fully branchless AVX2 kernel, 2-bit, 3-bit, and 4-bit decoding all share the exact same ultra-fast execution path. Zero heap allocations. No scalar fallbacks.

Hybrid Execution: The Best of Both Worlds

There’s a catch to all of this. If your context window is very short (say, 100 tokens), your KV cache is tiny. At that size, memory bandwidth isn’t your bottleneck — compute is. The overhead of rotating and quantizing vectors actually slows you down.

To fix this, I implemented a Hybrid Execution Engine.

When a conversation starts, AdapTQ stores the tokens in raw, contiguous FP32 memory. If the sequence length is below a certain threshold (e.g., 256 tokens), AdapTQ bypasses quantization entirely and runs standard, hyper-optimized FP32 attention.

The moment the cache crosses that threshold, it seamlessly routes the execution to the quantized SIMD pathway. Because the memory is stored in a flat buffer, there is zero data copying when this switch happens. You get peak performance at 10 tokens, and peak performance at 10,000 tokens.

The Results

I benchmarked AdapTQ on consumer hardware, maintaining a sequence of 4,096 tokens across 4 attention heads.

At these lengths, slashing the memory footprint to 64 bytes per token radically reduced the strain on the RAM.

Metrics Comparison (AdapTQ 4-bit vs FP16 Baseline):

Memory per Token: 64 bytes vs 512 bytes (8.0x smaller)

Throughput (>2k tokens): ~1,139 tok/s vs ~720 tok/s (1.5x faster)

Cosine Similarity: 0.947 vs 1.000 (High Fidelity)

Notice how the latency remains bounded and stable as the sequence length explodes, eventually crossing over and beating FP16 at around 2,000 tokens.

V2: Instant Branching and Deterministic Replay

If you’ve ever built a real-world LLM app, you know users love to “undo” and change a previous prompt. Normally, this forces the LLM to completely recompute the entire conversation up to that point.

Because AdapTQ runs on the CPU and strictly manages its own memory, I was able to build the Session Snapshot (.aqss) format.

In milliseconds, AdapTQ can serialize its exact, bit-packed internal state to disk. Using the Python API, you can instantly load a conversation and branch it from any specific token, with zero context-recomputation overhead.

Python Snippet:

from adaptq import ReplayEngine

engine = ReplayEngine()

Instantly load a 5,000-token conversation and branch generation starting from token 128!

result = engine.replay(“chat_session.aqss”, from_token=128)

Getting Started

A low-level C++ library is useless if nobody can use it. So, I spent weeks building robust Python wrappers using pybind11 and injecting them into the ecosystem.

Today, you can drop AdapTQ into a standard Hugging Face pipeline with just two lines of code. It intercepts the model’s internals and manages the KV state completely invisibly.

Python Snippet:

from transformers import AutoModelForCausalLM

from adaptq import create_adapter

model = AutoModelForCausalLM.from_pretrained(“Qwen/Qwen2–0.5B”)

Wrap the model. The KV cache is now fully managed by AdapTQ!

adapter = create_adapter(“transformers”, model=model, bits=4)

Generate normally

model.generate(…)

It also natively supports llama-cpp-python, allowing you to compress the KV cache of ultra-fast GGUF edge models even further.

Conclusion

Building AdapTQ was a masterclass in C++ memory management, AVX2 SIMD optimization, and mathematical transforms. It proved that with careful memory layout and algorithm design, we can push the boundaries of what’s possible on consumer hardware.

The project is fully open-source (MIT Licensed) and published on PyPI.

Try it out:

pip install adaptq

Check out the code on GitHub:

https://github.com/l3tchupkt/adaptq

If you’re working on local AI, edge inference, or C++ performance engineering, I’d love to hear your thoughts. Feel free to open an issue or drop a star on the repo!