vLLM Internalised: The Mechanics of Modern LLM Inference

Previously, we have explored about llama.cpp. The llama.cpp is agile, works on almost any hardware and is perfect for single-user exploration. It does not care about high concurrency. It just needs to get us to our destination.

vLLM is designed to move hundreds of concurrent users simultaneously on the same tracks (GPU infrastructure). We build the tracks and manage the schedule.

In llama.cpp, memory is often pre-allocated in a contiguous block for the KV Cache, the LLM short-term memory of the conversation. If the conversation ends early, or if we do not fill the context window, that GPU memory is wasted. However, vLLM treats GPU memory like virtual memory in an OS. It breaks the KV Cache into small, non-contiguous “pages”. This eliminates memory fragmentation, allowing us to fit many more concurrent conversations on the same GPU.

vLLM is purpose-built to sit behind an OpenAI-compatible API to handle:

• High-traffic spikes;
• Stable latency (Time-to-First-Token) for multiple concurrent users;
• Observability by giving us the telemetry of memory and network.

When we are building an AI solution on Amazon Bedrock, we are actually consuming a managed version of something like vLLM. (Image Credit: Amazon News)

The scheduler.py

In the v1 architecture of vLLM, the Scheduler is the "Traffic Controller". Its job is to decide, for every single step of the GPU, which request gets to generate a token next.

Before doing anything, the scheduler looks at a token_budget. It calculates how many tokens the GPU can handle in this specific step. If we have 100 requests waiting, it will not try to process them all at once. It will pack as many as possible until the token_budget is full.

def schedule(self) -> SchedulerOutput:
  self.current_step += 1
  ...

  preempted_reqs: list[Request] = []
  ...

  token_budget = self.max_num_scheduled_tokens
  if self._pause_state == PauseState.PAUSED_ALL:
    # Do not schedule any requests when paused.
    token_budget = 0
  ...

The max_num_scheduled_tokens is the hard limit. If we have 50 customer requests hitting our AI solution at once, the scheduler subtracts from this token_budget until it hits zero. Once it hits zero, even if our chatbot is ready to answer, the engine tells it to wait.

Notice the variable preempted_reqs. This is where the code handles the "overflow". If the engine runs out of room, it moves requests into this list, essentially telling the customer: "I am sorry, I have to pause your session to prioritise others."

So in vLLM system, latency stays flat until we hit the token_budget limit. Then, it hits a cliff.

In the same file, we can see the loop which is the physical manifestation of our AI solution performance.

# First, schedule the RUNNING requests.
req_index = 0
while req_index < len(self.running) and token_budget > 0:
  request = self.running[req_index]
  ...
  # Schedule newly needed KV blocks for the request.
  with record_function_or_nullcontext("schedule: allocate_slots"):
    while True:
      new_blocks = self.kv_cache_manager.allocate_slots(
        request,
        num_new_tokens,
        num_lookahead_tokens=self.num_lookahead_tokens,
      )

      if new_blocks is not None:
        # The request can be scheduled.
        break

...

vLLM is not checking for a big chunk of RAM. Instead, it is asking the cache manager: "Can I have a few more small pages?" Thus, if our individual prompts are too long and require large context window, they will cause the memory to fragment and the scheduler to struggle to find free blocks.

Birth of the Request

When a customer interaction for our AI solution reaches our vLLM, it triggers the following specific method.

def add_request(self, request: Request) -> None:
  existing = self.requests.get(request.request_id)

  if existing is not None:
    update = StreamingUpdate.from_request(request)

    if existing.status != RequestStatus.WAITING_FOR_STREAMING_REQ:
      assert existing.streaming_queue is not None, "duplicate request id"
      # Queue next input chunk (or finished sentinel).
      existing.streaming_queue.append(update)
    elif update is not None:
      # Commence next input chunk.
      self._update_request_as_session(existing, update)
    else:
      # Streaming-input session finished.
      self.finish_requests(request.request_id, RequestStatus.FINISHED_ABORTED)
  else:
    if request.resumable:
      request.streaming_queue = deque()

    self._enqueue_waiting_request(request)
    self.requests[request.request_id] = request

    if self.connector is not None:
      self.connector.on_new_request(request)
    if self.log_stats:
      request.record_event(EngineCoreEventType.QUEUED)

vLLM first checks if this request_id already exists. This is how vLLM handles Streaming. If our AI solution is sending a response chunk-by-chunk, the engine needs to know if this is a new question or just the next piece of an existing one.

The line self._enqueue_waiting_request(request) moves our AI solution user request from our API into the waiting queue we looked at in schedule. This is the exact moment your user goes from "I sent a prompt" to "I am now being managed by the Traffic Controller".

We now understand the full path of a token:

1. Interface Layer (add_request): Our prompt arrives, gets an ID, and is marked as "QUEUED";
2. Traffic Controller (schedule): The prompt waits in the queue until the token_budget allows it to be processed;
3. Memory Manager (allocate_slots): The prompt is broken into pages in the KV Cache;
4. GPU Execution: The token is generated.

The Law of Architectural Convergence

Whether we are running vLLM, or the proprietary engines inside Amazon Bedrock, Microsoft Foundry, or Gemini Enterprise Agent Platform, they are all solving the exact same physical problem: How to serve transformers efficiently on GPUs.

Since the physics of the KV Cache and the necessity of high-concurrency throughput are universal, all major LLM serving engines inevitably adopt these three patterns:

• Logical Paging: They all must move away from contiguous memory blocks to avoid fragmentation, such as the vLLM approach we have seen above;
• Continuous Batching: They all must interleave requests to keep the GPU utilization near 100%, such as the Traffic Controller approach;
• Token Budgeting: They all must have a mechanism to queue requests once the compute capacity of the hardware for a single step is reached;
• Observability: All enterprise-grade serving engines must expose telemetry. Without these, the system is a black box, and we cannot govern the performance of our AI solutions effectively.

By learning vLLM, we do not just learn a vendor-specific tool, but we have learned the Standard Model of LLM Inference.