Your LLM Server Is Wasting 80% of Its GPU Memory — Here’s How vLLM Fixes That

Imagine a restaurant where every table is reserved the moment a customer walks in — even before they’ve ordered. The table stays locked until they finish, pay, and leave. Meanwhile, there’s a queue outside, and half the tables are technically “occupied” but nobody’s eating.

That’s exactly how most LLM inference systems manage GPU memory — and studies suggest up to 80% of reserved space sits empty. You’re paying for a full restaurant and filling 2 tables.

vLLM (virtual LLM) is an open-source inference engine built at UC Berkeley that fixes this with a new memory management approach called PagedAttention. It’s now one of the most widely used LLM serving frameworks in production — and by the end of this post, you’ll understand exactly why, and have it running on your own machine.

TL;DR

Traditional LLM servers waste 60–80% of GPU memory on pre-reserved KV cache

vLLM introduces PagedAttention — borrowed from OS virtual memory — to fix this

Result: up to 24x higher throughput without any change to the model itself

Ships with an OpenAI-compatible API — drop-in replacement for most setups

Full setup guide for Linux (native) and Windows (via WSL2) included below

Why LLM Serving Is Harder Than It Looks

Every time you send a prompt to an LLM, the model generates tokens one at a time. Each new token needs to “see” all previous tokens to maintain context. Rather than recomputing that attention from scratch at every step, the model stores intermediate results — called the KV cache — in GPU memory.

This is the right call. Recomputing would be catastrophically slow. But the way most frameworks allocate and manage this cache is deeply wasteful — and that waste is what kills throughput in production.

The Problem: Your GPU Memory Is Being Held Hostage

Here’s what happens with a naive LLM server.

Suppose Alice sends a prompt that might generate anywhere from 10 to 500 tokens. The server doesn’t know the final length — so it reserves the maximum possible KV cache space upfront, just in case. Meanwhile Bob’s request queues. So does Carol’s.

The GPU is technically “full,” even though most of that reserved memory holds nothing yet. It’s locked out for requests that are still mid-generation. This is called memory fragmentation.

In practice, studies show 60–80% of GPU memory in traditional setups is wasted on pre-reserved but unused KV cache space. You’re paying for 100% of the GPU and getting 20–40% of its capacity. For a team running inference on an A100, that’s $30,000 of hardware delivering $6,000 of output.

The usual fix is static batching — wait for 16 requests, run them all together. But sequences vary in length. Request 1 finishes in 50 tokens. Request 8 is still going at 450 tokens. Request 1’s allocated GPU capacity sits idle while the batch waits. You get slow throughput and high latency. The worst of both worlds.

How PagedAttention Solves This

vLLM takes a page — quite literally — from operating system design.

When your laptop runs low on RAM, the OS doesn’t crash. It uses virtual memory: it maps logical addresses to physical memory pages, allocates pages only when needed, and shares pages between processes via copy-on-write. This idea is from the 1960s. It’s battle-tested.

vLLM applies the exact same logic to GPU memory. Its core innovation, PagedAttention, changes how the KV cache is allocated, stored, and shared across concurrent requests.

What is the KV cache, exactly?

In each transformer layer, the attention mechanism computes a “query” for the current token and looks it up against “keys” and “values” from all previous tokens. Instead of recomputing those keys and values for every token in history on each forward pass, the model caches them. That’s the KV cache.

In a traditional framework, you pre-allocate one big contiguous block of memory per sequence at the maximum sequence length. If your max is 2048 tokens and the actual sequence uses 300 tokens, you’ve wasted 85% of that allocation. Multiply across a batch of 32 concurrent requests and the waste becomes staggering.

How PagedAttention manages memory differently

PagedAttention splits the KV cache into fixed-size blocks — say, 16 tokens per block. Instead of one giant reservation per sequence, it maintains a block table: a mapping from logical block indices (what the model sees) to physical blocks (where data actually lives in GPU memory).

When Alice’s sequence needs its first 16 tokens cached, vLLM allocates one physical block. Tokens 17–32 get another. If Alice’s sequence ends at token 47, only a fraction of the third block is wasted — never a full sequence-length worth.

Compare: Alice generates 340 tokens, Bob generates 12. Traditional setup reserves 2048 each = 4096 token-slots, 3744 wasted. With vLLM, Alice uses 22 blocks and Bob uses 1. Near-zero waste.

Memory sharing via copy-on-write

If two requests share the same system prompt, vLLM points both to identical physical memory blocks — no duplication at all. Those blocks are only copied when the sequences diverge and write new content. For parallel sampling (generating 5 completions from one prompt), this alone cuts memory usage by over 55%.

This is copy-on-write, straight from OS memory design. For production chatbots with a fixed instruction prefix, or few-shot prompting setups where every request starts the same way, the savings stack up fast.

Continuous batching: keeping the GPU fully busy

Traditional batching is static. The server picks a batch size, waits for that many requests, runs them together, and returns results when the entire batch finishes. The problem? Sequence lengths vary wildly. Request 1 finishes in 50 tokens, request 8 is still going at 450. Request 1’s capacity idles for 400 token-steps of wait time.

vLLM uses continuous batching — also called iteration-level scheduling. After each forward pass, the scheduler checks which sequences just finished, removes them, and immediately slots in waiting requests. The GPU never waits for stragglers. Every iteration, the batch is as full as it can be.

The full request lifecycle in vLLM

1. Your request hits the vLLM API server (OpenAI-compatible REST endpoint)
2. The scheduler checks available physical blocks and admits or queues the request
3. A block table is initialized — a small set of physical KV cache blocks are pre-allocated
4. Prefill phase: the entire prompt is processed in a single forward pass
5. Decode phase begins: tokens are generated one at a time, each allocating a new block as needed
6. Each decode step, the scheduler removes finished sequences and slots in new waiting ones
7. When a sequence finishes, its blocks are freed and returned to the global block pool
8. Your response streams back, token by token

Real-World Benefits

Higher throughput at scale. The vLLM paper reports up to 24x higher throughput compared to HuggingFace Transformers on the same hardware. For a team serving thousands of daily users, that’s the difference between needing 10 A100s and needing 1.

Lower cost per token. If you’re self-hosting an open-source model like LLaMA 3 or Mistral 7B, throughput directly determines your cost per response. More requests per GPU-hour means a lower bill at the end of the month.

Better latency for real users. Continuous batching means users aren’t blocked waiting for someone else’s long sequence to finish. Their request gets slotted in the moment capacity opens.

Drop-in deployment. vLLM ships with an OpenAI-compatible REST API. You point your existing openai client at your vLLM server URL and it just works. No application code changes needed.

Multi-GPU support out of the box. vLLM supports tensor parallelism across multiple GPUs, and PagedAttention’s block design scales naturally with distributed setups.

Setting Up vLLM: Step-by-Step Guide

vLLM is natively built for Linux. On Windows, the easiest path is WSL2 (Windows Subsystem for Linux) — it gives you a real Linux environment with full CUDA access, and setup takes about 10 minutes. Pick your platform below.

🐧 Linux

Prerequisites: NVIDIA GPU (Ampere/A10/A100/H100 recommended), CUDA 11.8 or 12.1+, Python 3.9–3.12, and at least 16 GB VRAM for a 7B model.

1. Check your NVIDIA drivers and CUDA version

Before installing anything, confirm your GPU and CUDA are visible to the system.

# Check GPU and driver version 
nvidia-smi 
# Check CUDA version 
nvcc - version

You need CUDA 11.8 or higher. If nvcc is not found, install the CUDA Toolkit from NVIDIA's developer site.

2. Create a Python virtual environment

Always use a virtual environment to keep dependencies isolated.

# Create and activate a virtual environment 
python3 -m venv vllm-env source vllm-env/bin/activate # Confirm Python version (3.9–3.12 required) python - version

3. Install vLLM

Install via pip. vLLM will pull in PyTorch and all CUDA dependencies automatically.

# Upgrade pip first 
pip install - upgrade pip 
# Install vLLM (this pulls PyTorch + CUDA wheels automatically) 
pip install vllm

This takes 2–5 minutes depending on your connection. The download is ~2–3 GB including PyTorch.

4. Run your first model

Spin up the OpenAI-compatible API server. vLLM will auto-download the model from HuggingFace on first run.

# Serve Mistral 7B (good starting point - needs ~16 GB VRAM) 
python -m vllm.entrypoints.openai.api_server \ - model mistralai/Mistral-7B-Instruct-v0.2 \ - port 8000 
# For a smaller model on less VRAM (6–8 GB), try: 
python -m vllm.entrypoints.openai.api_server \ - model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ - port 8000

The server is ready when you see: INFO: Application startup complete.

5. Test it with curl

Open a new terminal and fire a test request.

# Test with a simple curl request 
curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mistralai/Mistral-7B-Instruct-v0.2", "messages": [{"role": "user", "content": "What is PagedAttention?"}], "max_tokens": 200 }'

6. Call it from Python (OpenAI SDK)

Since vLLM is OpenAI-compatible, your existing Python client just works — point it at localhost.

# pip install openai (if not already installed) 
from openai import OpenAI 
client = OpenAI( api_key="not-needed", # vLLM doesn't require a key by default 
      base_url="http://localhost:8000/v1" ) 
response = client.chat.completions.create( model="mistralai/Mistral-7B-Instruct-v0.2", 
      messages=[{"role": "user", "content": "Explain vLLM in one paragraph"}], 
      max_tokens=200 ) 
print(response.choices[0].message.content)

🪟 Windows (via WSL2)

Important: vLLM does not run natively on Windows. The supported path is WSL2 (Windows Subsystem for Linux), which gives you a real Ubuntu environment with direct GPU access. Windows 10 (build 19041+) or Windows 11 required.

1. Enable WSL2 and install Ubuntu

Open PowerShell as Administrator and run:

# Install WSL2 with Ubuntu in one command (Windows 11 / Win10 build 19041+) 
wsl - install # If you already have WSL1, upgrade to WSL2: 
wsl - set-default-version 2

Restart your machine when prompted. On first launch, Ubuntu will ask you to create a username and password — set something simple like sumit.

2. Install NVIDIA CUDA drivers for WSL2

This is the step most people get wrong. Do not install CUDA inside WSL — install the Windows NVIDIA driver only. WSL2 will automatically expose your GPU.

# Inside WSL2 Ubuntu - verify the GPU is visible 
nvidia-smi

If nvidia-smi works and shows your GPU, you're good. If not, update your NVIDIA drivers on the Windows side from nvidia.com (version 510+ required for WSL2 CUDA support).

3. Install Python and pip inside WSL2

# Update apt and install Python 3.11 
sudo apt update && sudo apt upgrade -y sudo apt install python3.11 python3.11-venv python3-pip -y 
# Confirm 
python3.11 - version

4. Create a virtual environment and install vLLM

# Create and activate virtual environment 
python3.11 -m venv vllm-env source vllm-env/bin/activate 
# Upgrade pip and install vLLM 
pip install - upgrade pip 
pip install vllm

Sit back — this downloads PyTorch and CUDA wheels and will take 3–8 minutes.

5. Launch the server

# Start the OpenAI-compatible vLLM server 
python -m vllm.entrypoints.openai.api_server \ - model mistralai/Mistral-7B-Instruct-v0.2 \ - port 8000

The server binds on localhost:8000 inside WSL2 — but Windows can access it at the same address from your browser or Python scripts on the Windows side.

6. Call it from Windows Python or browser

Open a Windows terminal (cmd or PowerShell — not WSL) and test:

# From Windows PowerShell or cmd - localhost routes to WSL2 automatically 
curl http://localhost:8000/v1/models

Or from a Windows Python script:

from openai import OpenAI 
client = OpenAI(api_key="not-needed", 
        base_url="http://localhost:8000/v1") 
response = client.chat.completions.create( model="mistralai/Mistral-7B-Instruct-v0.2", 
        messages=[{"role": "user", "content": "Hello from Windows!"}], 
        max_tokens=100 ) 
print(response.choices[0].message.content)

Where Do We Go From Here?

The bottleneck in LLM serving was never the model weights themselves. It was memory management — specifically, the assumption that you have to reserve space for the worst case before you know the actual case.

vLLM showed that if you apply the same thinking that made operating systems efficient in the 1970s, you can squeeze dramatically more out of the hardware you already have. No new model architecture. No bigger GPU. Just smarter bookkeeping.

The bigger question I keep coming back to: what else in the ML stack is still stuck with naive memory management? Embedding caches? Retrieval indices? Speculative decoding buffers? My guess is quite a lot of it.

If you’re running vLLM in production — or tried it and went back to something else — tell me what you ran into. I’m especially curious whether anyone has seen the copy-on-write benefit materially in real workloads. Drop it in the comments.