LLM Inference Engineering Room — Part 3: The Orchestration Layer

This is Part 3 of “The LLM Engine Room.” Part 1 introduced vLLM and the inference stack. Part 2 went deep on the algorithms — quantization, attention architectures, scheduling, and distributed computing. This final part zooms out to the layer above the engine: how you route, scale, observe, and operate LLM serving in production.

Parts 1 and 2 of this series lived inside the engine room — the GPU, the model, the kernels, the scheduler. All of that is about making a single inference server fast.

But a single fast server is not a production AI system.

Production means handling traffic spikes at 3am. It means routing a request to the right model variant without the user knowing multiple exist. It means knowing — before your users do — that latency just doubled on the EU cluster. It means autoscaling down at 2am and back up at 9am without a human touching anything.

That is the orchestration layer. And it is where most production AI systems either mature into reliable infrastructure or quietly collapse under their own complexity.

The Stack, Revisited

Before going further, here is the full picture of where everything sits:

┌─────────────────────────────────────────────────┐
│              User / Application                  │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│           Inference Gateway (IGW)                │
│   AI-aware routing · load balancing · auth       │
│   prefix-cache affinity · rate limiting          │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│         Orchestration (Kubernetes)               │
│   KServe · OpenShift AI · llm-d                  │
│   autoscaling · scheduling · lifecycle           │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│          Inference Engine (vLLM)                 │
│   PagedAttention · continuous batching           │
│   multi-LoRA · prefix cache · TP/PP/EP           │
└─────────────────────┬───────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────┐
│              Hardware (GPUs)                     │
│   H100 · MI300X · NVLink · NVSwitch              │
└─────────────────────────────────────────────────┘

Parts 1 and 2 covered the bottom two layers. Part 3 covers the top two.

The Inference Gateway

What It Is

The Inference Gateway (IGW) is the traffic layer that sits between your application and your model servers. Every request passes through it. It is the nervous system of your serving infrastructure — deciding, in milliseconds, where each request goes and why.

A naive implementation is just a load balancer: round-robin traffic across model replicas. That works at small scale. It falls apart as soon as you have multiple models, multiple hardware types, variable request sizes, prefix cache state distributed across nodes, and SLA requirements that differ by customer.

An AI-aware gateway understands the semantics of LLM serving. It makes routing decisions based on information that a generic load balancer doesn’t know exists.

AI-Aware Routing

Prefix cache affinity is the most important routing feature that generic load balancers miss entirely.

Recall from Part 2 that prefix caching stores the computed KV-Cache for common prompt prefixes. If a request with a known shared prefix gets routed to a replica that doesn’t hold that cache, the cache miss means a full prefill — wasting the optimization entirely.

AI-aware routing tracks which replicas hold which cached prefixes and routes requests accordingly. A request with a 2,000-token system prompt gets directed to the replica that already has those tokens computed. TTFT drops from seconds to milliseconds.

Load-aware routing goes beyond simple request counts. Two requests with the same token count are not equivalent: a request decoding 4,000 tokens holds KV-Cache memory far longer than one decoding 50 tokens. An intelligent gateway weights routing decisions by estimated memory consumption and decode time, not just queue depth.

Model-aware routing handles fleets with multiple model variants — different sizes, different fine-tunes, different quantization levels. The gateway can route based on request complexity (simple queries to smaller/cheaper models, complex ones to larger), customer tier (enterprise customers get the flagship model), or latency budget (if the large model is saturated, fall back to a faster smaller one).

Rate limiting and quota enforcement happen at the gateway, not inside vLLM. Per-customer token budgets, per-API-key rate limits, and burst handling are all gateway responsibilities.

Envoy and AI Gateway Implementations

The leading open-source AI gateway implementations are built on or inspired by Envoy Proxy — the same battle-tested data plane used in Istio and the broader service mesh world.

Envoy AI Gateway adds LLM-specific filters to Envoy: token counting, streaming response handling, model routing rules, and observability hooks that understand the difference between a prefill and a decode phase.

llm-d’s routing layer is a Kubernetes-native implementation that tightly integrates gateway routing decisions with the scheduler’s knowledge of per-replica cache state. This is the tightest coupling between gateway and engine available in the open-source ecosystem today.

For teams already running on cloud providers, managed options exist: AWS Bedrock has model routing built in, Azure AI Studio has a similar concept, and GCP’s Vertex AI handles routing as part of its serving infrastructure. These abstract away the gateway entirely in exchange for less control.

Orchestration

Kubernetes as the Foundation

Everything at scale runs on Kubernetes. This is not a bold claim — it is simply the state of the industry. The question is not whether to use Kubernetes for LLM serving, but how to extend it to handle the specific demands of GPU workloads and inference lifecycle management.

Vanilla Kubernetes was designed for stateless, CPU-bound web services. LLM serving is stateful (KV-Cache, LoRA adapter state), GPU-bound, and has dramatically different scaling characteristics. Running LLM workloads on Kubernetes requires several additional components.

KServe and the LLMInference CRD

KServe is a Kubernetes-native model serving framework that extends Kubernetes with custom resource definitions (CRDs) purpose-built for ML inference. It is part of the CNCF ecosystem and runs on any Kubernetes cluster.

Instead of writing raw Deployments and Services to run a vLLM server, you declare an InferenceService:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: llama3-70b
spec:
  predictor:
    model:
      modelFormat:
        name: vllm
      storageUri: s3://models/llama3-70b
      resources:
        limits:
          nvidia.com/gpu: "8"

KServe handles the rest: pulling the model, starting vLLM with the right configuration, exposing an OpenAI-compatible endpoint, setting up health checks, and integrating with the cluster’s autoscaler.

The newer LLMInference CRD (emerging from the Kubernetes community) goes further — it is purpose-built for LLM workloads and includes first-class concepts like model sharding configuration, LoRA adapter manifests, and per-replica cache size declarations. It represents the direction the ecosystem is moving as LLM serving becomes a first-class Kubernetes workload type.

llm-d — Disaggregated Serving on Kubernetes

llm-d (from Red Hat / IBM) implements disaggregated prefill-decode serving at the Kubernetes level. It is the most architecturally ambitious open-source project in this space.

The core idea: prefill and decode are fundamentally different workloads.

Prefill is compute-bound — it processes many tokens in parallel and loves high FLOP/s GPUs. Decode is memory-bandwidth-bound — it generates one token at a time and is bottlenecked by how fast weights and KV-Cache can be streamed through the GPU.

In a standard vLLM deployment, both phases run on the same GPU. They compete for resources. Decode requests slow down prefill; long prefills block decode queues.

llm-d separates them onto different hardware. Prefill nodes handle initial prompt processing. Decode nodes handle token-by-token generation. KV-Cache state is transferred between them over high-speed interconnects (NVLink, RDMA, or InfiniBand depending on the hardware topology).

The result: prefill latency and decode throughput are independently tunable. You can scale decode capacity without touching prefill nodes, and vice versa. For high-traffic deployments this is a significant operational and cost advantage.

llm-d also implements prefix-cache-aware routing, tracking cache state across replicas and feeding that information to the gateway routing layer.

Red Hat OpenShift AI

OpenShift AI is Red Hat’s enterprise-grade ML platform built on top of Kubernetes and OpenShift. For teams in regulated industries — financial services, healthcare, government — it adds the operational maturity that raw Kubernetes lacks:

RBAC and multi-tenancy — Fine-grained access control over which teams can deploy which models, with audit logging of every inference request. Necessary for SOC2, HIPAA, and FedRAMP compliance.

Model registry and governance — A versioned catalog of approved models with lineage tracking. You know exactly which model checkpoint is running in production, who approved it, and when it was deployed.

Pipeline integration — Model deployment connects to MLflow, Kubeflow Pipelines, or Tekton for automated promotion from training to serving, with validation gates between environments.

Operator-managed lifecycle — The OpenShift AI operator manages the entire LLM serving stack — GPU node provisioning, vLLM configuration, networking, and monitoring — via declarative CRDs. Day-2 operations (upgrades, scaling, failover) are automated.

For enterprise deployments, OpenShift AI is the fastest path from “we have a trained model” to “we have a compliant, monitored, production inference system.”

Autoscaling

Why LLM Autoscaling is Hard

Standard Kubernetes autoscaling (HPA — Horizontal Pod Autoscaler) scales on CPU or memory utilization. Neither is the right signal for LLM serving.

A vLLM replica can be at 10% CPU utilization and 95% GPU memory utilization simultaneously. CPU metrics say scale down. The system is actually at capacity.

LLM autoscaling needs GPU-aware signals:

KV-Cache utilization — When cache utilization crosses a threshold (say, 80%), requests start being preempted. This is the best leading indicator that a replica needs help before users see latency spikes.

Queue depth — Requests waiting to be processed. A growing queue means the current replica count is insufficient for current load.

TTFT and TPOT percentiles — When p95 TTFT starts climbing, that is a user-visible signal that the system is under pressure. Scaling on latency percentiles directly ties autoscaling to user experience.

Token throughput — Total tokens generated per second across the fleet. For cost optimization, you want to minimize idle GPU time; for performance, you want enough headroom that throughput never saturates.

Scale-to-Zero and Cold Start

For workloads with predictable off-hours — internal tools, batch processing, non-consumer products — scale-to-zero is attractive. Shut down all replicas when idle and spin them up on demand.

The challenge: LLM cold start is expensive. Loading a 70B model from object storage to GPU memory takes 3–5 minutes depending on storage throughput. A user request arriving during cold start waits that entire time.

Mitigation strategies:

Model caching on node-local storage — Keep model weights on NVMe SSDs attached to GPU nodes so loading takes seconds rather than minutes. This requires nodes to be pre-provisioned rather than truly ephemeral.

Minimum replica count — Keep one small replica warm during low-traffic periods. Not zero, but cheap.

Predictive scaling — Use historical traffic patterns to pre-warm replicas before demand arrives. Kubernetes KEDA (Kubernetes Event-Driven Autoscaling) supports cron-based scaling alongside metric-based scaling for exactly this pattern.

Knative-based serving — Knative’s serverless model includes built-in cold-start handling with configurable timeout behavior — requests queue during startup rather than failing immediately.

Observability

The Three Signal Tiers

Good observability for LLM serving requires tracking three categories of signals simultaneously. Missing any one leaves you blind to a class of failure.

Tier 1: Infrastructure Metrics

These are GPU and system-level signals that tell you whether the hardware is healthy and utilized:

GPU utilization — Both compute (SM utilization) and memory utilization are distinct numbers. A GPU at 100% memory and 20% compute is memory-bound on decode. A GPU at 80% compute and 40% memory is doing heavy prefill. The ratio tells you what the bottleneck is.

KV-Cache hit rate — What percentage of incoming requests hit the prefix cache. Low hit rates on a workload with shared system prompts indicate the cache is being evicted too aggressively or routing is not cache-aware.

Batch size distribution — The average number of requests batched per forward pass. Consistently low batch sizes mean the system is underloaded or the scheduler is inefficient. Consistently at maximum means you are at capacity.

Queue depth and wait time — How many requests are waiting and for how long. This is the leading indicator for throughput collapse before users feel it.

GPU memory fragmentation — PagedAttention reduces fragmentation dramatically, but monitoring it confirms the allocator is behaving as expected.

Tier 2: Inference Quality Metrics

These measure whether the model is producing outputs at the speed and reliability users expect:

TTFT (p50, p95, p99) — The distribution matters. A p99 of 8 seconds while p50 is 0.8 seconds indicates occasional very bad outliers, often caused by long prefill requests starving the queue.

TPOT (p50, p95) — Decode speed distribution. Should be relatively stable; spikes indicate decode contention.

Request success rate — What fraction of requests complete vs. time out, error, or get dropped. Preempted requests that exceed retry limits show up here.

Output token length distribution — Understanding whether users are generating short or long outputs helps tune batch size limits and cache allocation.

Speculative decoding acceptance rate — If running speculative decoding, the acceptance rate tells you whether the draft model is well-matched to your traffic patterns.

Tier 3: Business and Safety Metrics

These connect infrastructure to what the system is actually for:

Token consumption by customer/team — Enables cost allocation, quota enforcement, and capacity planning. Without this, you cannot answer “how much did the legal team’s chatbot cost this month?”

Latency by model tier — If routing different customer tiers to different model sizes, per-tier latency confirms the routing is working as intended.

Safety filter hit rate — If running content moderation before or after inference, the rate of filtered requests is a signal about user behavior and potential abuse patterns.

Audit log completeness — In regulated deployments, every inference request must be logged with inputs, outputs, model version, and timestamp. Gaps in the audit log are a compliance failure.

Telemetry-Driven Routing

The most sophisticated production deployments close the loop between observability and routing — the gateway makes routing decisions based on live telemetry from the inference layer.

Instead of routing based only on queue depth, the gateway might:

• Route away from a replica whose KV-Cache hit rate has dropped (its cache has been polluted by a long outlier request)
• Prefer a replica whose p95 TTFT is trending down (it just freed up after a heavy batch)
• Avoid a replica that just logged three consecutive timeout errors (possible GPU health issue)

This transforms the gateway from a static router into an adaptive control loop. llm-d implements exactly this pattern — its routing layer subscribes to per-replica telemetry and adjusts weights in near-real-time.

MLPerf and Benchmarking

Before going to production, you need to know what your system can actually do. MLPerf Inference is the industry-standard benchmark suite for LLM serving, covering:

Scenario: Server — Sustained throughput at a target query rate with latency constraints. The closest to real production traffic patterns.

Scenario: Offline — Maximum throughput with no latency constraint. Useful for batch workloads.

Metrics measured: TTFT, TPOT, and tokens per second per GPU — normalized so results are comparable across hardware configurations.

Running MLPerf before launch tells you your actual throughput ceiling, your latency profile under load, and where the bottlenecks are. It also gives you a baseline to regress against when you upgrade vLLM, change quantization settings, or add more LoRA adapters.

Beyond MLPerf, teams typically build custom benchmarks for their specific traffic shape — the token length distributions, concurrency levels, and prefix sharing patterns that match their actual users.

Putting the Full Stack Together

Here is what a mature, production LLM serving deployment looks like end to end:

A request arrives at the Inference Gateway. The gateway authenticates it, checks the customer’s token quota, and identifies the prompt prefix. It looks up which vLLM replica holds that prefix in cache and routes accordingly. If no replica has it cached, it routes to the least-loaded replica.

The request enters vLLM’s scheduler. The continuous batching scheduler checks KV-Cache availability. If cache pressure is high, it preempts a low-priority request (swapping its state to CPU). The new request joins the active batch.

Prefill runs — on a disaggregated prefill node if using llm-d, or on the same GPU otherwise. FlashAttention handles the attention computation. The KV-Cache for this request is allocated in pages by PagedAttention. If the prompt prefix was cached, only the novel suffix is processed.

The KV-Cache state is transferred to a decode node (in a disaggregated deployment). Decode begins, generating tokens one at a time. If the model is MoE, the router selects experts at each layer; Expert Parallelism distributes the computation across GPUs. If the request matches an active LoRA adapter, it is applied transparently.

Tokens stream back through the gateway to the user. The gateway timestamps each token for TPOT measurement.

Telemetry flows to the observability layer — cache hit/miss, TTFT, TPOT, batch size, GPU utilization — all captured per request and per replica. Dashboards show fleet state in real time. Alerting fires if p95 TTFT exceeds SLA thresholds.

The autoscaler watches queue depth and KV-Cache utilization. If either crosses a threshold, a new vLLM replica is scheduled. Kubernetes pulls the model from node-local NVMe cache and starts serving within seconds.

The audit log records every request — model version, input hash, output, latency, customer ID — to object storage, satisfying compliance requirements.

That is production LLM inference at maturity.

The Series in Summary

Three articles, one complete picture:

Part 1 answered: What is the inference engine and why does it exist? vLLM, PagedAttention, prefill vs. decode, parallelism, the basic ecosystem.

Part 2 answered: How does it work under the hood? Quantization, FlashAttention, kernel fusion, MHA/MQA/GQA/MLA, TTFT/TPOT, throughput collapse, prefix caching, multi-LoRA, MoE, ZeRO, distributed communication.

Part 3 answered: How do you operate it at scale? Inference gateways, AI-aware routing, llm-d disaggregated serving, KServe, OpenShift AI, autoscaling, observability, telemetry-driven routing.

The GPU is not the hard part. The algorithms are not the hard part. The hard part is building a system where all of it works together reliably, observably, and efficiently — under real production load, at 3am, without anyone watching.

That is inference engineering.

Complete Series Glossary

Inference Gateway (IGW)

Traffic layer between application and model servers that handles routing, authentication, and rate limiting.

AI-Aware Routing

Routing decisions based on LLM-specific signals such as cache state, token cost, model tier, and replica health.

Prefix Cache Affinity

Routing requests to the replica that already holds the prompt prefix in its KV-Cache to maximize cache reuse and reduce latency.

KServe

Kubernetes Custom Resource Definition (CRD) framework for managing the lifecycle of machine learning model serving.

LLMInference CRD

A Kubernetes custom resource specifically designed for configuring and managing LLM inference workloads.

llm-d

An open-source Kubernetes-based serving system that disaggregates prefill and decode phases of LLM inference.

Disaggregated Serving

An architecture that separates prefill and decode operations onto different hardware resources to improve utilization and scalability.

OpenShift AI

Red Hat’s enterprise AI platform that provides governance, RBAC, model serving, and operator-managed infrastructure.

HPA (Horizontal Pod Autoscaler)

Kubernetes-native autoscaling mechanism based on resource metrics. While useful, it is generally insufficient by itself for GPU-intensive LLM workloads.

KEDA

Kubernetes Event-Driven Autoscaling framework that supports custom metrics, event-based scaling, and scheduled scaling policies.

KV-Cache Utilization

The percentage of allocated KV-Cache memory currently in use. Often the most important signal for LLM-serving autoscaling decisions.

Scale-to-Zero

The practice of shutting down all serving replicas during idle periods to reduce costs. Challenging for LLMs because of model loading and cold-start latency.

SM Utilization

Streaming Multiprocessor utilization on GPUs, representing actual compute activity and hardware efficiency.

MLPerf Inference

The industry-standard benchmark suite used to evaluate AI and LLM inference performance, throughput, and latency.

Telemetry-Driven Routing

A routing strategy that dynamically adjusts traffic distribution using real-time performance telemetry from serving replicas.

Audit Log

An immutable record of inference requests, responses, and system actions used for compliance, governance, and traceability.

Quantization

A model optimization technique that stores weights in lower-precision formats (such as INT8 or FP8) to reduce memory consumption and improve inference speed.

FlashAttention

A highly optimized attention algorithm that reduces memory movement by leveraging on-chip SRAM, improving throughput and lowering latency.

Kernel Fusion

Combining multiple GPU operations into a single execution kernel to reduce memory transfers and improve overall efficiency.

TTFT (Time to First Token)

The elapsed time between submitting a request and receiving the first generated token. One of the most important user-perceived latency metrics.

TPOT (Time Per Output Token)

The average time required to generate each subsequent token after the first token has been produced.

Throughput Collapse

A sharp reduction in serving throughput caused by resource saturation, often driven by excessive KV-Cache pressure or scheduling bottlenecks.

Prefix Caching

The reuse of previously computed KV-Cache entries for prompts sharing common prefixes, significantly reducing prefill computation.

Multi-LoRA Serving

Serving multiple LoRA fine-tuned adapters from a single base model, allowing efficient multi-tenant deployments.

MoE (Mixture of Experts)

A sparse neural network architecture that activates only a subset of expert subnetworks for each token, improving scalability and efficiency.

PagedAttention

A memory management technique that treats KV-Cache similarly to virtual memory pages, enabling efficient allocation and reduced fragmentation.

ZeRO

A distributed training optimization technique that partitions model states, gradients, and optimizer data across devices to eliminate redundancy.

MHA / MQA / GQA / MLA

A family of attention mechanisms that trade off model quality, memory consumption, KV-Cache size, and inference efficiency:

• MHA (Multi-Head Attention) — Traditional attention mechanism with separate key-value pairs for each attention head.
• MQA (Multi-Query Attention) — Shares key-value pairs across heads to reduce memory usage.
• GQA (Grouped-Query Attention) — A compromise between MHA and MQA, balancing quality and efficiency.
• MLA (Multi-Head Latent Attention) — Advanced attention design focused on reducing KV-Cache requirements while maintaining model performance.



