vLLM: Anatomy and Mechanisms of High-Throughput LLM Inference

Executive Summary

The vLLM inference engine represents a significant advancement in the deployment of Large Language Models (LLMs), specifically addressing the critical bottleneck of GPU memory management. By introducing PagedAttention, an algorithm inspired by virtual memory paging in operating systems, vLLM achieves a 2–4x improvement in throughput compared to state-of-the-art systems like FasterTransformer and Orca.

The system’s core innovation is the decomposition of the KV (Key-Value) cache into fixed-size, non-contiguous blocks, virtually eliminating memory waste caused by internal and external fragmentation. This architectural shift allows for significantly larger batch sizes and efficient memory sharing for complex decoding algorithms such as beam search and parallel sampling. Beyond memory management, vLLM incorporates advanced features including continuous batching, prefix caching, speculative decoding, and disaggregated prefill/decoding to optimize both latency and throughput across distributed hardware environments.

The Core Challenge: KV Cache Fragmentation

In LLM inference, the generation process is autoregressive, requiring the system to store the Key-Value (KV) vectors of all previous tokens to compute the next one. This “KV cache” is massive and dynamic.

Inefficiencies in Traditional Systems

Existing systems (e.g., Orca, FasterTransformer) suffer from severe memory waste because they treat the KV cache as contiguous tensors:

• Internal Fragmentation: Systems pre-allocate memory for a request’s maximum possible length (e.g., 2048 tokens), even if the actual output is much shorter.
• External Fragmentation: Variability in request lengths leads to unusable gaps between allocated memory chunks.
• Over-reservation: Memory is reserved for future tokens during the entire lifetime of a request, preventing other requests from utilizing that space.
• Waste Quantification: Data indicates that in traditional systems, only 20.4% to 38.2% of the KV cache memory actually stores token states. In contrast, vLLM increases this utilization to approximately 96.3%.

PagedAttention: The vLLM Innovation

PagedAttention allows KV cache vectors to be stored in non-contiguous physical memory blocks, moving away from the requirement of contiguous tensor storage.

Block-Level Management

• Logical vs. Physical Blocks: A request’s KV cache is divided into logical blocks. These are mapped to physical blocks on the GPU via a Block Table.
• On-Demand Allocation: Blocks are allocated as needed. When a logical block is filled, the scheduler assigns a new physical block from a “free block pool.”
• Block Size: While larger blocks increase hardware utilization, they can increase internal fragmentation. vLLM identifies 16 tokens as the optimal default block size.

Memory Sharing via Copy-on-Write (CoW)

PagedAttention enables multiple sequences to share the same physical memory blocks:

• Parallel Sampling: Multiple outputs for one prompt share the prompt’s KV blocks.
• Beam Search: Different “beams” share a common history. As they diverge, vLLM only copies the specific block where the divergence occurs (Copy-on-Write), significantly reducing memory usage.
• Efficiency Gains: Beam search can see memory savings of up to 55.2% (Alpaca dataset) to 66.3% (ShareGPT dataset) through block sharing.

Engine Architecture and Scheduling

The vLLM engine core is composed of a Scheduler, a KV Cache Manager, and a Model Executor.

Iteration-Level Scheduling and Continuous Batching

Traditional “naive” batching waits for an entire batch to finish before starting a new one. vLLM utilizes continuous batching (iteration-level scheduling):

• After every iteration (forward pass), the system can remove finished requests and insert new ones into the batch.
• This minimizes queuing delays and maximizes GPU utilization.

Prefill vs. Decode Operations

The scheduler manages two distinct types of workloads:

1. Prefill: Processes the initial prompt. It is compute-bound (limited by the GPU’s TFLOPS).
2. Decode: Generates one token at a time. It is memory-bandwidth-bound (limited by the speed of moving weights and KV cache into on-chip memory).

Preemption and Recovery

When VRAM is exhausted, vLLM employs a First-Come-First-Served (FCFS) policy and handles preemption via two methods:

• Swapping: Evicted KV blocks are moved to CPU RAM.
• Recomputation: The system simply re-executes the prompt to regenerate the KV cache when the request is rescheduled. Recomputation is often more efficient for small block sizes, whereas swapping excels for larger blocks.

Advanced Optimization Features

Prefix Caching

This feature avoids recomputing the KV cache for prompts that share common beginnings (e.g., “system prompts” or instructions).

• vLLM hashes chunks of 16 tokens. If a new prompt’s hash matches an existing cached block, the system reuses the KV vectors directly.
• This drastically reduces Time-To-First-Token (TTFT) for repetitive workloads.

Chunked Prefill

Very long prompts can monopolize the engine, causing high latency for other requests. Chunked prefill splits these long prompts into smaller segments, allowing the scheduler to interleave prefill and decode tasks more effectively.

Speculative Decoding

To overcome the bottleneck of single-token generation, vLLM uses a smaller, faster “draft” model to propose k candidate tokens.

• The large model then verifies all k tokens in a single forward pass.
• vLLM implements proposal schemes including n-gram, EAGLE, and Medusa to accelerate generation without losing statistical accuracy.

Disaggregated Prefill/Decoding (P/D)

This architecture separates prefill and decode tasks onto different GPU instances.

• Prefill workers compute the initial state and write it to a KV-cache service.
• Decode workers read from the service to continue generation.
• This isolation prevents bursty prefill tasks from interrupting the steady latency required for decoding.

Distributed Execution and Scalability

vLLM supports models that exceed the VRAM of a single GPU through several parallelism strategies:

Strategy

Description

Tensor Parallelism (TP)

Shards model weights across multiple GPUs on the same node. Preferred for high intranode bandwidth.

Pipeline Parallelism (PP)

Distributes different layers across different nodes.

Data Parallelism (DP)

Replicates the entire model to handle multiple requests concurrently.

Multi-Process Execution

The MultiProcExecutor coordinates multiple GPU workers. A driver worker (Rank 0) broadcasts tasks to children via shared memory message queues. This abstraction allows the engine to treat a multi-GPU setup as a single unit.

Performance Metrics and Benchmarking

System performance is measured through a trade-off between Latency and Throughput.

Key Metrics

• TTFT (Time to First Token): Critical for user-facing interactivity.
• ITL (Inter-Token Latency): The time between consecutive tokens.
• Goodput: Throughput that successfully meets specific Service Level Objectives (SLOs).

The Roofline Model

Performance is constrained by either memory bandwidth or compute power:

• Bandwidth-Bound: At low batch sizes, the GPU spends most of its time loading weights.
• Compute-Bound: At high batch sizes (saturation point), the GPU’s processing power (TFLOPS) becomes the limiting factor.

vLLM provides CLI tools (vllm bench) to simulate real-world workloads, using Poisson distributions for request arrivals to measure how different configurations impact these metrics.