KVQuant: How I Got 2-Bit KV Cache Quality to Match 3-Bit Five Extensions to Near-Optimal Vector…

If you’re running large language models locally, the KV cache is probably
your biggest memory problem. At long contexts it dominates VRAM…