TurboQuant KV Cache Benchmark on AMD Strix Halo

*TL;DR: TurboQuant works great on Gemma 4 (ISWA model), enabling ~4–5× KV cache compression with minimal speed impact. MoE models like GPT-OSS 120B still have a known crash bug with TurboQuant, but standard quantization (q8_0, q4_0) works perfectly.**

What is TurboQuant?

TurboQuant is a new vector quantization technique from Google Research that can compress high-dimensional vectors with near-zero quality loss. Published at ICLR 2026 arXiv:2504.19874, it achieves 3.5 bits per channel with “absolute quality neutrality” and only marginal degradation at 2.5 bits.

The key innovation is a two-stage approach:

1. Random rotation + PolarQuant: First, input vectors are randomly rotated to simplify their geometry. This allows applying optimal scalar quantizers to each coordinate independently. The rotation is data-oblivious (doesn’t depend on the actual data), making it fast and practical.
2. Quantized Johnson-Lindenstrauss (QJL): A second stage uses just 1 bit to apply QJL on the residual error, eliminating bias in inner product estimation. This is crucial for attention mechanisms in transformers.

Why KV Cache Compression Matters

If you’ve ever run a large language model and watched your GPU memory vanish as the context window grows, you understand the KV cache problem. The key-value cache stores “attention” data that the model needs to reference during inference, and it grows linearly with context length.

For a model like Gemma or Llama, a single token in the KV cache can consume:

• f16: 2 bytes × head_dim × 2 (K+V) = significant memory
• TurboQuant 3-bit: ~3× smaller, with theoretically near-zero quality loss

With TurboQuant, you can run context lengths that would normally OOM your GPU. A 120B model that needs ~26GB for KV cache at 80K context can suddenly fit in your 24GB GPU.

AMD ROCm Support

Big thanks to domvox who implemented the full HIP/ROCm port of TurboQuant for AMD GPUs.

The port includes:

• All GPU kernels (WHT rotation, quantization, flash attention)
• Seamless integration with llama.cpp’s existing KV cache system
• Benchmark support via llama-bench

The implementation is available at [github.com/domvox/llama.cpp-turboquant-hip]. See also the Reddit discussion where this was originally shared.

Test Setup

Hardware:

• AMD Ryzen AI MAX+ 395 (Strix Halo)
• 112 GB VRAM (shared with system)
• gfx1151 architecture

Environment variables (important for Strix Halo):

export GGML_HIP_NO_VMM=1
export HSA_ENABLE_SDMA=0

Benchmark command used:

./build-rocm/bin/llama-bench \
 -m <model.gguf> \
 -ngl 999 -fa 1 \
 -b 2048 -ub 512 \
 -p 512,2048,4096,8192 \
 -n 128 \
 -d 0,8192,32768,65536 \
 -mmp 0 -dio 1 \
 -t 16 -r 3

[embed]

Summary Table

Gemma 4 26B MoE — Prefill @ 8192 context

| KV Type | pp8192 | vs f16 |
| - - - - -| - - - -:| - - - -:|
| f16 | 908 t/s | baseline |
| q8_0 | 840 t/s | -7.5% |
| turbo3 | 844 t/s | -7.1% |
| turbo4 | 830 t/s | -8.6% |

Gemma 4 26B MoE — Long Context @ 65536

| KV Type | pp8192 @ 65K | vs f16 |
| - - - - -| - - - - - - - :| - - - -:|
| f16 | 333 t/s | baseline |
| q8_0 | 330 t/s | -0.9% |
| turbo3 | 328 t/s | -1.5% |
| turbo4 | 326 t/s | -2.1% |

Conclusion

TurboQuant works on ISWA models (like Gemma 4) with only ~7–9% prefill slowdown 2. Long context performance is excellent: at 65K context, TurboQuant shows only 1–2% slowdown vs f16 3. MoE models need work: TurboQuant crashes on GPT-OSS 120B (known upstream issue)

The real win here is VRAM. With 4–5× KV cache compression, you can run significantly longer context windows. A model that would OOM at 32K context with f16 KV cache can run at 128K+ with turbo3.