Routing LLM Inference in Production at OpenAI

This document synthesizes a presentation given by the OpenAI Inference team regarding the evolution and architecture of their inference…