Routing LLM Inference in Production at OpenAI

This document synthesizes a presentation given by the OpenAI Inference team regarding the evolution and architecture of their inference load balancer. The system handles routing requests from CPU clusters to GPU engine clusters in production.

Introduction to the Inference Load Balancer

The inference load balancer sits between incoming requests processed by frontend CPU clusters and the backend GPU clusters hosting various inference engines. When a request arrives, the CPU cluster processes it and forms an inference request. The load balancer is then responsible for selecting the best available inference engine to serve the request. A single model may have multiple engines behind it, and the selection must consider factors such as Time to First Token (TTFT), overall load, network overhead, error rates, and key-value (KV) cache matching. The goal is to proxy the request through the internal network to the optimal backend engine.

Routing back to the same engine for a similar prompt without re-transferring data improves efficiency and reduces latency, taking advantage of cache utilization. The load balancer must manage a combination of performance, reliability, and mechanical variance associated with the application.

The Early Days: Periodic Feedback Loop

In its early iterations, the load balancer relied on a periodic feedback loop. This architecture filtered inference requests based on eligibility (e.g., engine capability) and used weighted consistent hashing to select the destination inference engine.

The weights for this hashing were generated dynamically. Inference engines emitted various signals, which were periodically updated by a controller using Exponential Weighted Moving Average (EWMA) smoothing. The controller computed a performance goal, compared it against a base average, and updated the weights in a manner conceptually similar to a Proportional-Integral-Derivative (PID) controller. If an engine’s performance was better than average, its weight increased; if worse, its weight decreased.

Pros and Cons of the Feedback Loop

This early approach had several positive attributes. It naturally balanced out routing restrictions, considered many signals simultaneously, was self-adaptive like a PID controller, and generally “just worked” without requiring extensive manual intervention.

However, the adaptability came with significant trade-offs:

•Opacity: Because the system combined so many signals, it was very difficult to reason about specific routing decisions or control the system. Fine-tuning one aspect almost inevitably impacted another.

•Uneven Distribution: Load was not always evenly distributed, particularly when dealing with a mixed fleet where models were served by engines running on different GPU SKUs with varying characteristics.

•Oscillation: The feedback loop created slow, detrimental oscillations. When traffic was shifted away from a heavily loaded engine, the engine would cool down. The controller would interpret this new state as available capacity and shift traffic back. This back-and-forth shifting disrupted the KV cache hit rate and overall efficiency.

These limitations motivated the team to rethink the architecture and develop a new, globally optimized solution.

New Architecture: Separated Control and Data Planes

To address the shortcomings of the periodic feedback loop, OpenAI introduced a new architecture that separates the control plane from the data plane. This separation answers the fundamental question: for each request from a single CPU cluster, which GPU engine should serve it?

A naive approach of simply picking the server with the lowest load requires cross-engine lookups, which is impractical when signals are constantly moving. Furthermore, engines are heterogeneous, possessing different hardware, capacities, health statuses, and physical distances from the CPU clusters. Keeping all routing decisions strictly local to a single CPU cluster is also flawed; multiple CPU clusters might independently decide to route traffic to the same optimal engine, overloading it while leaving others underutilized.

The solution requires a global view to produce a globally optimized routing answer, paired with a data plane capable of making rapid routing decisions based on that answer.

Three Paths Through the System

The new architecture defines three distinct paths through the system: one synchronous request path and two asynchronous control paths.

1. Inference Request Path (Synchronous, per request)This is the fast, local path where the actual routing happens. A request arrives from a CPU cluster, the local data plane uses its current routing state to select an engine and route the request, and the GPU engine serves the request. No request waits on the control plane during this synchronous operation.

2. Engine Signal Path (Asynchronous feedback)The system continuously collects real-time signals from the GPU engines, such as TTFT, Time Between Tokens (TBT), queue depth (regular tasks), and overall engine health. These signals are aggregated and ingested by both the control plane (to build globally optimized weights) and the data plane (to serve as fast local routing state).

3. Routing Weight Path (Asynchronous pull)The control plane computes new routing weights and publishes them. The local data planes asynchronously pull these updates to refresh their local routing state. The control plane continuously computes the next globally optimized snapshot while the data plane operates on the latest installed snapshot.

Case Study: Why Nearest-Only Routing Fails

The presentation provided a case study illustrating why routing traffic exclusively to the geographically nearest engine is suboptimal. The core issue is that traffic demand and GPU capacity are rarely balanced geographically.

Consider three regions:

•Region 1: CPU Cluster A generates 90 Requests Per Second (RPS), and nearby Engine A has a capacity of 100 RPS. Local routing works perfectly here.

•Region 2: CPU Cluster B generates 120 RPS, but nearby Engine B only has a capacity of 100 RPS.

•Region 3: CPU Cluster C generates 40 RPS, but nearby Engine C has a capacity of 80 RPS, leaving 40 RPS of spare capacity.

If the system insists on nearest-only routing, the extra 20 RPS from CPU Cluster B will be forced to wait on the overloaded Engine B, resulting in significant engine-side latency. However, if the system routes that extra 20 RPS across regions to Engine C, it incurs a network distance penalty (e.g., +4ms latency) but avoids the much larger engine-side waiting time. Therefore, a further engine can provide a faster end-to-end experience.

The Optimizer: From Signals to Weights

The control plane relies on an Optimizer to turn various signals into actionable routing weights. The Optimizer determines what fraction of a CPU cluster’s traffic should be directed to each GPU engine.

The Optimizer takes four primary inputs:

1.Demand (requests from the CPU cluster).

2.Network latency to each engine.

3.Engine capacity and health.

4.TTFT / TBT latency profiles (which describe how an engine’s latency degrades as load increases).

Using these inputs, the Optimizer performs traffic allocation with a specific optimization goal: to minimize the expected end-to-end latency across all routed traffic. This requires balancing network distance with engine-side latency. A nearby engine is attractive when it has room to serve traffic, but a further engine becomes the better choice if nearby engines are nearing full capacity.

The Optimizer must also respect several hard constraints during its calculations. It must route all traffic demand, ensure that all engines stay within their effective capacity limits, and keep all resulting routing weights non-negative.

Protection Mechanisms

Because production environments are unpredictable — capacities degrade, nodes fail, and networks deteriorate — the load balancer incorporates three key protection mechanisms to maintain system health under heavy load.

1. Penalties (Failover)When an engine begins to fail or behave anomalously, the system detects the issue and reduces the routing weight to that specific engine. This penalty gives the engine a chance to recover autonomously from a transient issue, or provides time for human engineers to intervene and resolve the problem.

2. Retry with CapWhile retrying failed requests is a standard mitigation technique, it can exacerbate problems when a system is heavily utilized, potentially causing a retry storm that brings the system down. To prevent this, the load balancer implements dynamic caps or budgets on retries. During normal operation, the system can tolerate more retries, but when the system is under heavy load, the cap is tightened to keep retry volume within an acceptable, safe region.

3. Load SheddingAs a last resort, when production capacity simply cannot meet the increasing volume of request demand, the system employs load shedding. Rather than allowing the entire system to become overwhelmed and fail catastrophically, the load balancer proactively drops a portion of the traffic. This ensures that the system degrades gracefully and continues to serve as much traffic as possible reliably.

Ref: Speakers: Qianru Lao & Lu Zhang (Inference Team) OpenAI