The KV Cache — The one data structure that decides how many people your AI can serve at once

If you have ever wondered why AI companies spend billions on GPUs, the answer is not what you think. The biggest bottleneck in production AI systems is not the model itself — it is the model’s memory. Specifically, something called the KV cache.

In this article, I will break down what the KV cache is, why it eats so much GPU memory, and how engineers solved one of the most expensive problems in AI infrastructure — using an idea that is over 50 years old.

How AI Actually Writes: One Word at a Time

Here is something most people do not realize about AI. When you ask ChatGPT a question, it does not think about the full answer and type it all out at once. It writes one word at a time.

You send your question. The AI picks the first word. Then it reads your question plus that first word, and picks the second word. Then it reads everything again — your question, word one, word two — and picks the third word. And so on, until the response is complete.

Every single time the AI picks a new word, it has to look back at everything that came before — your original question and every word it has generated so far.

Think of it like writing an essay where, before every new sentence, you have to re-read the entire essay from the beginning. For a short paragraph, no problem. For a 10-page essay, you are spending most of your time re-reading instead of actually writing.

That is exactly what happens inside an AI model. And it creates a massive problem at scale.

The KV Cache: Saving Work So You Do Not Repeat It

Here is the insight that makes AI actually usable in real time.

When the model processes each word, it creates two pieces of information:

1. A Key — a label that says: “Here is what this word is about.”
2. A Value — the actual content: “Here is the useful information this word carries.”

There is also a Query — that is the current word asking: “What do I need to know right now?” The model compares the current word’s Query against the Keys of all previous words. High match? Pull in that word’s Value. That is how the AI decides which earlier words matter for the next word it writes.

The critical insight: once a word’s Key and Value are calculated, they never change. Word 5’s Key and Value are the same whether you are generating word 6 or word 5,000. So why recalculate them every time?

The fix is simple: save them. The first time the model processes a word, store its Key and Value. Every time after that, just look them up.

That saved storage is the KV cache. Without it, generating a long response would be so slow the system would be unusable.

But there is a catch. All those saved Keys and Values have to live somewhere — in GPU memory. And at the scale that companies like OpenAI and Google operate, that storage becomes the single biggest engineering challenge in AI infrastructure.

Why the Memory Cost Is So Brutal

AI models run on GPUs — specialized chips with their own memory. A top-of-the-line GPU has about 80 GB of memory. Here is how that gets used:

1. The model itself takes around 35 GB. That is the AI’s brain — its learned knowledge.
2. Everything else goes to the KV cache — the memory of ongoing conversations.

For a short conversation — a few hundred words — the KV cache is small. A gigabyte or two. No problem.

But as conversations get longer, the cache grows fast. A long conversation — the kind where you paste in a whole document and ask questions about it — can eat 40 GB or more of KV cache. For a single conversation. On a GPU that only had 45 GB of free space after loading the model.

One long conversation can take over the entire GPU. There is no room left for anyone else.

Now multiply this by millions of users chatting at the same time. Each conversation needs its own KV cache. Each cache takes GPU memory. And GPUs cost tens of thousands of dollars each.

The question becomes: how do you fit as many conversations as possible into limited GPU memory?

The Old Way: Wasteful and Broken

Before anyone built a smarter system, memory management was crude.

When a user sent a message, the system would reserve memory for the maximum possible response length — say, 4,000 words — all upfront. Even if the actual answer was only 200 words, all 4,000 words worth of memory stayed locked.

It is like booking an entire restaurant for a party that might have 100 guests, but only 5 show up. Ninety-five empty seats, locked up, and meanwhile other people are being turned away because the restaurant looks full.

95% of reserved memory sitting empty.

But it got worse. As conversations started and finished at different times, the freed memory ended up scattered in tiny gaps across the GPU — like a parking lot where the open spots are spread everywhere instead of next to each other. A new conversation that needs a big chunk of memory might find enough total free space, but none of it in a usable piece.

When researchers measured real systems, they found that 60 to 80 percent of KV cache memory was being wasted at any given moment. Companies were buying extra GPUs at enormous cost to compensate for what was fundamentally a software problem.

The Fix: A 50-Year-Old Idea Applied to GPUs

The solution came from an unexpected place — operating systems from the 1970s.

Your computer runs dozens of programs at the same time. Each program needs memory. But they do not each get one big reserved block. Instead, the operating system chops memory into small, equal-sized pages. Programs get pages as they need them. Done with a page? It goes back to the pool.

The genius part: pages do not have to be next to each other in physical memory. Your browser might be using pages scattered all over the place. The operating system keeps a map that makes everything look organized to the program, even though the actual pieces are spread out.

Engineers applied this exact idea to GPU memory for the KV cache using a technique called PagedAttention. Instead of reserving one giant contiguous block of memory for each conversation, the KV cache is split into fixed-size blocks that can be allocated dynamically and scattered across GPU memory, just like virtual memory pages in operating systems.

The results were dramatic:

1. Memory waste dropped from 60–80% to under 4%.
2. The same GPUs could serve 2–4x more users.
3. Not because the AI got faster — because the memory was no longer being wasted.

Prefix Caching: Stop Redoing the Same Work

The paged approach solved memory waste. But there is another kind of waste: repeated computation.

Think about a customer service chatbot. Every conversation starts with the same 2,000-word company guidelines. The paged system can share the stored Keys and Values in memory. But the very first conversation still had to compute all those Keys and Values. What happens when the cache gets cleared, or a new server comes online? The next conversation recomputes everything from scratch.

Prefix caching fixes this. The system keeps a fingerprint of each block of words. When a new conversation arrives, it checks: “Have I already computed the Keys and Values for these starting words?” If yes — skip all that work and jump straight to the user’s actual question.

Prefix caching is especially useful in systems where many requests share the same starting context, such as system prompts, documentation, or company knowledge bases. By reusing previously computed KV states, providers can reduce both latency and GPU compute costs.

For readers interested in a deeper technical walkthrough, I’ve also put together a companion YouTube video covering prefix caching and other inference optimization techniques — including offloading, compression, and smart eviction — used in large-scale AI systems

[embed]

If you are interested to know how different distributed systems, like TinyURL, Twitter, Distributed Cache, Uber, Whatsapp, Netflix, Dropbox, Stripe, etc., can be designed then please checkout our course:“The Bible of Distributed System Design Interviews”.

[embed]The Bible of Distributed System Design Interviews The Best Distributed System Design Interview Preparation Course. This course is the single best resource to learn not…www.thinksoftwarelearning.com

Regarding our course “The Bible of Distributed System Design Interviews”, this course is a great resource to not just improve your distributed system design skills but also to ace your distributed system design interviews and targeting level L5+ in companies like Facebook, Google, etc. The book/course is still in-progress work. There are already quite a few chapters discussing the design of TinyURL, Twitter, Distributed Cache, Uber, Whatsapp, Netflix, Dropbox, Stripe etc.

The course also includes a section on mock system design interviews. This is a great resource because it will provide you the experience of an actual distributed system design interview. It will enable you to go through real-life examples of system design interviews. You will see how an experienced interviewer can ask different questions to evaluate a candidate while designing a simple distributed system, such as TinyURL service, and how a candidate tackle questions asked by interviewers. What are the different mistakes that candidates make during the system design interviews and how you can avoid those mistakes in your interview. Also, more mock system design interviews are being added.