Getting Started with VLLM — Installation, Setup & Inference (Online & Air-Gapped)

Post 1 of vLLM blog series

VLLM has quickly become one of the most popular and efficient inference engines for serving Large Language Models (LLMs). Its ability to deliver high throughput, low latency, and GPU-efficient memory management makes it an ideal choice for enterprise deployments, including fully air-gapped environments such as BFSI, telecom, defense, and government.

This blog post is your end-to-end guide for:

• What VLLM is and why it matters
• Prerequisites
• Installing VLLM in online environments
• Installing VLLM in completely offline air-gapped environments
• Running local models in production
• Validating VLLM with multimodal inference using curl

What is VLLM?

VLLM (Virtual Large Language Model Server) is a high-performance inference engine built by UC Berkeley. It solves common LLM serving bottlenecks using:

1. PagedAttention

A breakthrough mechanism that avoids GPU memory fragmentation by storing the KV-cache in a paged format — similar to virtual memory. This enables:

• Longer context lengths
• More parallel requests
• Zero memory fragmentation issues

2. Continuous Batching

VLLM automatically merges incoming requests to maximize GPU utilization and throughput — ideal for real-world production workloads.

3. OpenAI-Compatible REST API

Applications originally built for OpenAI can work with VLLM with minimal or zero code changes.

4. Flexible Model Loading

Supports models from:

• Hugging Face
• NVIDIA
• Local disk (offline/air-gapped mode)

Prerequisites

To install and run VLLM smoothly, ensure:

• Python 3.10 or above
• pip ≥ 23
• CUDA 11.8 or 12.1
• Supported NVIDIA GPUs (A10, A40, L40, A100, H100, etc.)
• Optional helpful package:

pip install num2words

Installing VLLM (Internet-Enabled Environments)

When the machine has internet access, installation is simple.

Step 1 — Reset GPU (Optional but Recommended)

sudo nvidia-smi --gpu-reset -i 1

Step 2 — Install VLLM

pip install vllm

Step 3 — Verify Installation

pip show vllm

Step 4 — Ensure vllm CLI is in PATH

ls ~/.local/bin/vllm
export PATH="$PATH:$HOME/.local/bin"
echo 'export PATH="$PATH:$HOME/.local/bin"' >> ~/.bashrc
source ~/.bashrc

Step 5 — Run a HuggingFace Model via VLLM

vllm serve "HuggingFaceTB/llama3.2-70B-Instruct" \
  --gpu-memory-utilization 0.6 \
  --dtype float16

The server now runs at: **http://10.10.10.100:8000**

Installing VLLM in Air-Gapped Environments (Offline)

Air-gapped deployments require zero internet access. VLLM fully supports this mode.

Step 1 — Download Wheels in Online Zone

Run on a machine with internet:

pip download vllm --dest vllm_offline_packages

This folder now contains:

• vllm wheels
• All dependencies

Step 2 — Securely Transfer the Wheels

Using:

• Encrypted SSD
• USB with checksum validation
• IT-approved air-gap transfer procedure

Step 3 — Install VLLM Offline

python3 -m pip install --no-index --find-links=vllm_offline_packages vllm

Step 4 — Download the Model Offline

On a machine with internet:

hf download HuggingFaceTB/llama3.2-70B-Instruct --local-dir llama3_70b_offline

hf download HuggingFaceTB/SmolVLM2-2.2B-Instruct --local-dir SmolVLM2-2.2B-Instruct

Transfer to the air-gapped host.

Step 5 — Run VLLM Offline with Local Model

vllm serve "./llama3_70b_offline" \
  --gpu-memory-utilization 0.6 \
  --dtype float16

Offline Safety Tip

Prevent any outbound Hugging Face requests:

export HF_HUB_OFFLINE=1

Running Local Models in Production (Recommended for Offline Use)

Example: serving llama3.2–70B-Instruct locally:

vllm serve /home/ai/models/vllm_vlm_models/llama3.2-70B-Instruct \
  --gpu-memory-utilization 0.6 \
  --dtype float16

The model is served entirely from disk, ideal for air-gapped and secure deployments.

Validating VLLM Using curl (Multimodal Text + Image)

Use this API call to ensure all components work end-to-end:

curl -X POST "http://10.10.10.100:8008/v1/chat/completions" \
  -H "Content-Type: application/json \
  --data '{
    "model": "/home/ai/models/vllm_vlm_models/llama3.2-70B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image in one sentence."
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
            }
          }
        ]
      }
    ]
  }'

This test verifies:

✔ API reachability ✔ Local model loading ✔ Image-to-text inference ✔ Multimodal pipeline health

Conclusion

You now have a complete, streamlined guide for:

• Installing VLLM online
• Installing VLLM offline in air-gapped environments
• Running local models securely
• Validating inference via API calls

VLLM is production-ready, enterprise-friendly, and ideal for regulated sectors requiring secure LLM serving.

Bonus Tip :

Install CUDA Toolkit 12.x (matching driver 550)

Sincedriver supports CUDA 12.4, install either CUDA 12.4 or 12.1 (both compatible with driver 550).

I recommend CUDA 12.4, latest & ideal for H100, H200, GH100

Install CUDA Toolkit 12.4 (Ubuntu)

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600

wget https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda-repo-ubuntu2204-12-4-local_12.4.1-550.54.15-1_amd64.deb

sudo dpkg -i cuda-repo-ubuntu2204-12-4-local_12.4.1-550.54.15-1_amd64.deb

/var/cuda-repo-ubuntu2204-12-4-local/

sudo cp /var/cuda-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/

sudo apt update
sudo apt install -y cuda-toolkit-12-4

Set environment variables

echo 'export CUDA_HOME=/usr/local/cuda-12.4' >> ~/.bashrc
echo 'export PATH=$CUDA_HOME/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Verify installation

nvcc --version
ls $CUDA_HOME/include