My Understanding of vLLM and PagedAttention

I wish this is the last major topic I need to deeply understand in the LLM and fine-tuning space for this year, before I move back to creating YouTube videos on RAG and Fine-Tuning in my YouTube Channel “Praveen Reddy Learnings”.

For the past few months, I have been stuck trying to properly understand some core LLM inference concepts. One of those topics recently led me to the PagedAttention paper while exploring how KV-Cache actually works internally.

Before understanding this paper, I strongly recommend to check out my KV-Cache blog first, because this entire paper is built on top of the KV-Cache bottleneck problem.

[embed]Understanding KV Cache in LLM’s We all know that Large Language Models (LLMs) are autoregressive models.medium.com

As we already discussed in the KV-Cache blog above, KV-Cache helps LLMs avoid recomputing attention for previously generated tokens again and again. Since autoregressive models take the entire sequence generated so far as input to predict the next token, the model would normally have to recompute the Keys and Values for all previous tokens at every generation step.

To avoid this repeated computation, KV-Cache stores the previously computed Keys and Values in memory and reuses them during next-token generation, significantly improving inference speed and efficiency.

But they mentioned in the paper that, the moment LLMs starts serving multiple users simultaneously, KV-Cache itself became one of the biggest bottlenecks in production systems.

Challenges with Traditional KV-Cache Systems

1. KV-Cache Consumes Huge GPU Memory

For every generated token:

• Key vectors and Value vectors are stored across every transformer layer and across every attention head present inside those across all transformer layers and attention heads

Since each transformer layer contains multiple attention heads, every head computes its own separate Query, Key, and Value vectors for every token.

For example, if a single transformer layer has 10 attention heads, then for a single word/token:

• 10 different Query vectors, 10 different Key vectors and 10 different Value vectors are generated inside that layer itself, because each head learns different relationships and attention patterns from the same token.

This means KV-Cache grows continuously as tokens increase.

2. Sequence Length is Unpredictable

We never know how long a user will generate text. User A may stop conversation with LLM after 30 tokens, User B may continue conversation for 4000 tokens. So inference systems cannot accurately preallocate memory.

To stay safe: systems reserve much larger memory regions for storing KV Cache than actually needed. This creates memory waste.

3. Internal Fragmentation

Let’s assume, A system allocates memory for: 2048 tokens. But if the user generates only: 300 tokens. Remaining memory stays unused. This unused reserved memory is called: Internal Fragmentation

4. External Fragmentation

In the realtime, many requests will be arriving and finishing dynamically. GPU memory starts looking like:

[A][free][B][free][C]

Here, even though the total free memory available might actually be enough for a new request, the problem is that the free memory is split into multiple smaller chunks instead of one large continuous block. Traditional memory allocation systems often require one large contiguous memory region to allocate KV-Cache for a new sequence. Since such a continuous block may not exist anymore, the allocation fails even though enough total memory is technically available. This problem is called External Fragmentation.

They mentioned in the paper that, This is exactly the same memory management problem the traditional operating systems faced before the concept of Virtual Memory Paging.

As LLM serving systems(like ChatGPT) have started handling:

• Longer conversations
• More concurrent users
• Larger KV-Caches
• Dynamic batching
• Advanced decoding strategies

Traditional contiguous memory allocation started becoming highly inefficient.

Why are we managing KV-Cache like rigid continuous tensors instead of managing them like operating system memory?

and that single idea eventually led to the introduction of:

• PagedAttention
• and the vLLM serving architecture.

Instead of allocating one large continuous memory region for every token sequence, vLLM treats KV-Cache memory similarly to virtual memory paging in operating systems by dividing KV-Cache into smaller fixed-size blocks that can be allocated dynamically across GPU memory. This reduces memory fragmentation, improves GPU utilisation, and enables efficient large-scale LLM serving.

To solve these KV-Cache memory challenges, the paper introduces a new memory management mechanism called PagedAttention.

Core Idea Behind PagedAttention

In operating systems:

• Memory is divided into pages, pages are mapped logically & physical memory does not need to be contiguous.

Similarly:

vLLM divides KV-Cache into:

• Fixed-size blocks/pages. Instead of allocating one huge continuous memory chunk, KV-Cache grows block-by-block dynamically.

Traditional KV-Cache Layout

Older systems used like:

[Sequence A Continuous Memory]

If sequence grows:

• Larger memory allocated, old cache copied, previous memory freed.

This creates:

• Fragmentation
• Copying overhead
• Allocation inefficiency.

PagedAttention Layout

Each block can exist anywhere in GPU memory. No need for contiguous allocation. This completely changes memory management.

Here we will see on how Paged Attention is implemented:

1. Logical Blocks

Instead of treating the entire KV-Cache as one huge continuous memory region, PagedAttention divides the sequence into smaller fixed-size chunks called logical blocks. These blocks are created based on the token order in the sequence.

Example:

Tokens 1–16  -> Belongs to Logical Block 0
Tokens 17–32 -> Belongs to Logical Block 1

2. Physical Blocks

Physical Blocks represent the actual locations where KV-Cache data is stored inside GPU memory. While Logical Blocks maintain the sequence order of tokens, Physical Blocks are the real memory blocks allocated on the GPU hardware.

Example:

Logical Block 0 -> Physical Block 7
Logical Block 1 -> Physical Block 21

3. Block Table

In PagedAttention, the Block Table is responsible for tracking where each part of a sequence is actually stored inside GPU memory. Since KV-Cache is divided into smaller blocks and these blocks can be placed anywhere in memory, the system needs a way to identify which physical memory block belongs to which part of the sequence.

The Block Table maintains this mapping information. For every Logical Block in a sequence, it stores the corresponding Physical Block location where the actual KV data exists.

The block table connect: Logical Block to the Physical Block

Example:

Logical Block 0 may point to Physical Block 7
Logical Block 1 may point to Physical Block 21
Logical Block 2 may point to Physical Block 3

Because of this block-based memory design, KV-Cache can now grow dynamically as new tokens are generated, instead of requiring one huge preallocated continuous memory region from the beginning. As blocks can now be allocated independently anywhere in GPU memory, the system no longer depends on finding large contiguous free spaces for every sequence. This dramatically reduces both internal and external memory fragmentation.

Now, let’s understand the overall vLLM System Architecture and how different components work together to efficiently serve Large Language Models at scale.

vLLM System Architecture

1. Request Scheduler

The Request Scheduler is responsible for managing incoming user requests and efficiently organising them for GPU execution. In real-world LLM serving systems, users continuously send prompts at different times, and every request may have completely different sequence lengths and generation speeds.

The request scheduler handles:

• Accepting new requests
• Batching multiple requests together
• Deciding the execution order
• Dynamically inserting new sequences and removing completed sequences

One of the most important capabilities enabled by the request scheduler is Continuous Batching.

Batch means: Group of multiple user requests processed together on the GPU at the same time, because GPUs are designed for parallel processing and can handle many computations simultaneously much more efficiently than processing requests one-by-one sequentially.

Instead of waiting for an entire batch to finish before processing new requests, vLLM continuously updates active batches by inserting new requests whenever space becomes available. This keeps the GPU busy almost continuously and significantly improves GPU utilisation and serving throughput.

2. KV-Cache Manager

The KV-Cache Manager acts as the core memory management engine behind PagedAttention. Since KV-Cache is now divided into smaller blocks that can dynamically grow and move across GPU memory, the system needs a dedicated component to manage all these operations efficiently, which is done by KV Cache manager.

The KV-Cache Manager is responsible for:

• Allocating new blocks
• Freeing unused blocks
• Maintaining block tables

This component is one of the main reasons why vLLM can handle large numbers of concurrent requests efficiently without excessive memory fragmentation.

So overall, the KV-Cache Manager is the component that intelligently handles all these memory operations efficiently behind the scenes.

3. PagedAttention CUDA Kernel

A kernel in GPU programming is a small highly optimized program that runs directly on the GPU to perform a specific computation in parallel across thousands of GPU cores.

Traditional attention kernels are usually designed with the assumption that KV-Cache exists in one continuous memory region. However, in PagedAttention, KV blocks can be scattered across different physical memory locations on the GPU.

Because of this non-contiguous memory layout, traditional attention kernels have become inefficient. To solve this problem, vLLM introduces custom PagedAttention CUDA kernels specifically optimized for block-based KV access.

These kernels are responsible for:

• Gathering scattered KV blocks
• Reconstructing the correct logical token order
• Computing attention efficiently and maintaining high GPU throughput.

4. GPU Workers

GPU Workers are responsible for performing the actual transformer computations during inference. Once requests are scheduled and KV blocks are managed properly, GPU workers execute the model operations required for token generation.

Their responsibilities include:

• Transformer layer execution
• Attention computation
• Feed-forward network computation and next-token generation.

These workers continuously process active sequences during inference while interacting with the PagedAttention kernels to fetch the required KV blocks dynamically.

5. CPU Coordination Layer

While GPUs perform the heavy tensor computations, the CPU manages the control flow and coordination across different system components.

This layer handles:

• Request management
• Scheduling decisions
• Worker coordination
• Memory management coordination and communication across different execution components.

It acts as the central controller that ensures all parts of the vLLM serving pipeline work together efficiently during large-scale LLM inference.

Now, Let’s understand How Decoding Works in vLLM step by step:

Step 1 — User Request Arrives

Example:

"Explain transformers"

When a user sends a prompt, the Request Scheduler first accepts the request and creates a sequence for it. A sequence simply represents the complete token flow associated with that request, including:

• The input prompt tokens, and all future generated output tokens.

At this stage, the sequence currently contains only the input prompt tokens: "Explain transformers". As the model starts generating new tokens one-by-one, those generated tokens also become part of the same sequence.

Step 2 — KV Blocks Allocated Dynamically

Once the input prompt tokens are processed, the model generates the corresponding Keys and Values for those tokens. Instead of allocating one huge continuous memory region for storing the entire KV-Cache, vLLM allocates smaller fixed-size KV blocks dynamically and incrementally as needed.

For smaller prompts, only a small number of KV blocks may be allocated initially. For example, if the prompt contains only a few tokens and fits within the configured block size, a single KV block may be sufficient.

Example:

Block 1
Block 2

As more tokens are generated during inference, additional KV blocks are allocated dynamically whenever required.

Step 3 — Tokens Generated Sequentially

As the model starts generating new tokens one-by-one, new Key and Value entries are continuously added to the KV-Cache. Additional KV blocks are allocated dynamically only when the existing blocks become full. This incremental block allocation helps avoid unnecessary memory reservation and reduces memory over-allocation.

Step 4 — Attention Computation

During token generation, the PagedAttention CUDA kernel fetches the required KV blocks from GPU memory, gathers the corresponding Keys and Values, and performs the attention computation needed to predict the next token. Even though the KV blocks may be physically scattered across different GPU memory locations, the Block Table ensures that the logical token ordering of the sequence remains correct during computation.

Step 5 — Sequence Completion

Once the request generation finishes, the KV blocks associated with that sequence are immediately freed and become available for future requests. This dynamic allocation and deallocation mechanism helps vLLM utilize GPU memory very efficiently while supporting large numbers of concurrent requests.

Now that we understood how a single request gets processed internally in vLLM using dynamically allocated KV blocks, the next important question is: How does vLLM efficiently handles thousands of user requests simultaneously while keeping GPUs continuously utilized?

This is where one of the most important optimizations in modern LLM serving systems comes into the picture:

Continuous Batching in vLLM

vLLM introduces Continuous Batching. Instead of waiting for the entire batch to finish, vLLM continuously updates the active batch dynamically. As soon as one request finishes, a new incoming request is immediately inserted into the batch if GPU space becomes available.

This keeps the GPU busy almost continuously and significantly improves:

• Throughput
• GPU utilization and request latency.

Why PagedAttention is Critical for Continuous Batching

PagedAttention solves this problem by allowing KV-Cache blocks to be allocated dynamically and independently across GPU memory instead of requiring one large continuous memory region for every sequence. This makes dynamic insertion and removal of requests much more efficient and enables continuous batching to scale effectively.

Performance Impact of vLLM and PagedAttention

The paper demonstrated that PagedAttention significantly improves GPU memory utilization and overall LLM serving throughput compared to traditional inference systems. Since memory fragmentation becomes extremely small, vLLM can fit a much larger number of active sequences inside GPU memory simultaneously.

This directly improves:

• Concurrent request handling
• Batching efficiency
• Long-context inference and overall GPU utilization.

Since vLLM can dynamically insert and remove requests efficiently without expensive memory reallocations, GPUs remain busy almost continuously instead of waiting for entire batches to finish.

Because of all these optimizations together, vLLM achieved significantly higher serving throughput while drastically reducing KV-Cache memory waste, which is one of the main reasons why vLLM became one of the most widely adopted LLM serving systems today.

One important thing to understand here is that vLLM is an inference and serving engine that sits between the model and the GPU to make LLM inference much more efficient.

Normally, during inference, people directly load models using frameworks like HuggingFace Transformers and generate outputs. But as the number of concurrent users increases, KV-Cache memory usage, fragmentation, and GPU inefficiencies start becoming major bottlenecks.

This is where vLLM comes into the picture.

As discussed, Instead of manually handling KV-Cache and batching logic, vLLM automatically manages: PagedAttention, Dynamic KV-Cache allocation, Continuous batching, GPU memory optimization and high-throughput request serving internally.

vLLM is installed on the inference server or GPU machine and acts as the runtime responsible for serving the model efficiently.

Another important point is that vLLM works perfectly even with fine-tuned models such as LoRA-based models. Helps the fine-tuned model to benefit from all the optimizations provided by vLLM, including efficient KV-Cache management, PagedAttention, continuous batching, and better GPU utilization.

Below is a simple code comparison between traditional HuggingFace inference and vLLM inference.

Traditional HuggingFace inference:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
prompt = "Explain transformers"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_new_tokens=100
)
print(tokenizer.decode(outputs[0]))

In this approach:

• HuggingFace directly handles inference
• KV-Cache management is traditional
• Batching optimizations are limited
• GPU utilization may become inefficient at scale

Now the same idea using vLLM:

from vllm import LLM, SamplingParams
llm = LLM(
    model="meta-llama/Llama-2-7b-chat-hf"
)
sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=100
)
#SamplingParams is used to define how the model should generate text during inference, such as controlling randomness, output length, and decoding behavior.
outputs = llm.generate(
    "Explain transformers",
    sampling_params
)
print(outputs[0].outputs[0].text)

Here, vLLM automatically handles:

• PagedAttention
• KV-Cache block allocation
• Continuous Batching
• GPU memory optimization
• Efficient concurrent request servin

without anyone manually implementing any of those optimizations.

Similarly, even LoRA fine-tuned models can be served through vLLM while still benefiting from all these inference optimizations.

That’s it….

Thank You !! Happy Learning !!