Decoding LLM Inference: The Compute and Memory Paradigm Part 1

LLM inference is a hot topic these days, and with recent advancements, it will only become more important. When working with LLMs, two key metrics are crucial: TTFT (Time to First Token) and ITL (Inter-Token Latency). I have been studying the mechanics of inference and this article shares some learnings and experiences along the way. This will be a series of articles each going in more depth on the concepts.

Attention Layers and QKV Matrix

I will try to simplify the concepts to keep the content short and understandable but I would suggest reader to dig in more details

LLMs are essentially a series of neural network layers, with some special layers called attention layers. In modern LLMs, the attention layer determines which part of the input is relevant for generating the output token. You can think of the attention layer as answering the question: “What part of the input sequence does this particular output token need to look at?” During the decoding phase, these models can look back at the entire input sequence and see what part of the input matters to the current output token.

These layers work by creating three matrices: Query, Key, and Value (which we’ll call Q, K, and V). To give you more intuition about what these three matrices store, you can think of them as follows: Q represents what you need to know about the input sequence, K represents where you can find those answers, and V represents the actual values themselves.

A single set of Q, K, and V matrices creates one attention head. Modern LLMs have multiple attention heads, which intuitively allows them to analyze the same input sequence from different perspectives.

Inference Mechanism

Inference or a forward pass to the LLM, is a lot of linear layers, attention layers or multi-head attention, and activation functions. As the input sequence is passed through the LLM, it creates this Q, K and V transformations of input sequence.

#Linear transforms for a single attention layer
q = self.Q(input)
k = self.K(input)
v = self.V(input)

From these simple equations we see that the input sequence is passed from these matrices. The matrix operations can be performed in parallel, and it’s where the GPU shines.

Continuing to this story, the LLMs are autoregressive — ie they predict token one by one and how does this work — the last predicted token becomes the part of input for the next generated token. This is done in loop until the end of sequence token is produced.

If you observe here, the input grows during the decoding process — one token after another. The LLM has to process the same prefix sequence multiple times, which is inefficient. To prevent this from happening repeatedly, we use a KV Cache. For the next token prediction, instead of passing in the full input sequence, only the last predicted token is passed. This way, the GPU does not have to do all the recomputation again and again.

Phase 1: Prefill

So we know what is KV cache, the LLM inference can be broken down into two steps. First is the prefill, where the model fills up the KV cache. By the process we discussed, as a result of matrix multiplications the prefill step is compute heavy.

Phase 2: Decode

After the KV cache is populated, the first output token is predicted by the model, this is called decoding — from the hidden state of model the model predicts the next token, that token again goes to the input of the model and next token is predicted. The KV cache grows in size here, since the entire output and input sequence becomes part of KV cache. As a result the decode phase is memory bound.

Two phases, the prefill is compute bound the decode phase is memory bound.

All the inference optimizations pin down to these concepts and different libraries handle these differently.

Experiments

So too much theory, yeah it all looks and sound good but we can actually test this. I did some benchmarking and will share the results to share how this shows up in practice.

I used a cloud machine with these specifications

Nvidia L4 GPU — 24 GB GDDR6 Memory,

Used vLLM to serve locally, Qwen/Qwen2.5–7B-Instruct model.

Used vllm bench utility to run the experiments

I used different length of input prompt 512, 1024, 2048 and 4096 tokens and tracked different metrics TTFT, ITL, Throughput.

[embed]

Graphs added,

TLDR

Across all experiments, we observed that as the input sequence length increases, the Time to First Token (TTFT) grows rapidly, while the Inter-Token Latency (ITL) remains consistent.

The longer the input, the more computation time the prefill step requires, which directly drives up the TTFT. In contrast, the decoding phase is memory-bound; because memory access speeds remain constant regardless of the input sequence length, the ITL stays stable. Additionally, if the input sequence becomes too long, the expanding KV cache will exceed available memory, resulting in an Out of Memory (OOM) error during inference.

Ultimately, balancing these compute and memory constraints is the key to scaling LLMs effectively. My goal here was to demystify that process and provide a better intuition for how inference actually works. I hope you found this article helpful!