How We Built Kargo’s Real-Time ML Platform to Serve Predictions at Massive Scale

Inside our Contextual Bandit Pipeline for Dynamic Product Ads

Latency comparison (p50 vs p99) across inference architectures. Service-based deployments introduce significantly higher tail latency, while embedded ONNX and native C++ avoid network overhead and remain consistently fast.

The advertising ecosystem runs at millisecond scale.

Every time a page loads, an app opens, or a video starts, we have a split second to decide which ad to show — balancing user engagement with advertiser value.

At Kargo, that decision must happen in under 20 milliseconds, across millions of requests per second.

Hitting this SLA required more than better models — it forced us to rethink the entire ML lifecycle. To understand where latency actually comes from, we evaluated multiple inference architectures.

In this post, we’ll walk through how we evolved from brittle lookup tables and notebook-trained models to a real-time inference platform, and how contextual bandits power our Dynamic Product Ads system at production scale.

The Quest

Achieving sub-20ms inference in ad auctions isn’t simply about putting a model behind an API. The entire ML system must operate in real-time.

Features must be fresh and available instantly, models must be retrained continuously and versioned safely, and inference must scale to millions of QPS while maintaining strict latency guarantees. Monitoring must close the loop by tying predictions to both infrastructure health and business outcomes.

Building a platform capable of meeting these constraints required rethinking nearly every part of the ML lifecycle.

The Challenges

Early machine learning systems at Kargo were fundamentally batch-first and siloed.

Models were trained in notebooks, serialized into precomputed lookup tables, and deployed into Brain, our centralized inference service. Each model operated largely in isolation, with its own feature logic and deployment pipeline, limiting reuse and consistency across systems.

While lookup tables enabled fast inference by mapping feature combinations directly to outcomes, they introduced significant constraints. Adding new features required regenerating entire tables offline, slowing iteration and increasing operational overhead. As advertiser catalogs grew, these tables expanded rapidly, eventually hitting memory and scalability limits.

The system also lacked robust monitoring and observability. Because models were trained and deployed via notebooks, there was limited visibility into model performance, feature quality, or prediction behavior in production. This made it difficult to debug issues, validate changes, or tie model outputs to business outcomes.

More critically, this architecture prevented real-time learning. Predictions were derived from static mappings, with no ability to explore alternative actions or adapt based on new feedback. As we began exploring contextual bandits and reinforcement learning, it became clear that this approach could not support dynamic, real-time decision-making.

To meet our latency and scale requirements, we needed a fundamentally different approach.

The Rebuild

To support real-time decisioning across our advertising products, we built an internal ML platform designed for low-latency, production-scale inference.

At a high level, the system is built on AWS SageMaker for training orchestration, and the inference stack is composed of several tightly integrated components:

• Model Registry: Manages model artifacts, versioning, and safe promotion of models from experimentation to production.
• Inference Layer: Central interface for model inference, responsible for routing prediction requests and coordinating execution across multiple backends.
• Monitoring System: Tracks infrastructure health, prediction latency, model performance, and business KPIs.
• Feature Store: Provides low-latency access to online features while maintaining training–serving parity.

Together, these components allow us to move models from training to production quickly while maintaining reliability, observability, and performance.

High Level Architecture of ML Inference System

Kargo’s ML Inference Architecture

The ML Platform Client (MLPClient) is not a standalone inference service. Instead, it serves as the entry point for model inference, routing requests across multiple backends through a unified prediction API. It is the core abstraction that enables flexible execution across both centralized and embedded environments.

When a prediction request is issued, the client determines the appropriate execution path based on model configuration, runtime requirements, and latency constraints. For GPU-accelerated models and high-throughput workloads, the client routes requests to Brain, our gRPC-based inference service. Brain acts as the centralized inference gateway and delegates model execution to NVIDIA Triton Inference Server.

Beyond Triton-based inference, our platform is built on a shared GPU infrastructure layer designed to support multiple inference runtimes, including both Triton and vLLM. This infrastructure is powered by Sleipnir, our GPU-optimized Kubernetes platform, which abstracts GPU scheduling, resource isolation, and workload orchestration across heterogeneous inference services. Sleipnir manages GPU allocation at the cluster level, enabling multiple models and runtimes to co-exist efficiently while enforcing isolation and fair resource sharing.

This unified GPU layer allows us to support different classes of workloads. Triton is primarily used for traditional ML and deep learning models that benefit from batching and concurrent execution, while vLLM is used for large language model workloads that require efficient token generation and KV-cache management. By standardizing both runtimes on the same infrastructure, we achieve consistent deployment, scaling, and observability across diverse inference workloads.

To support specialized runtimes, we build custom Docker images on top of Triton and vLLM base containers, packaging model-specific dependencies alongside the inference runtime , for example, Vowpal Wabbit for Dynamic Product Ads models. This approach ensures reproducible deployments while allowing containers to be tuned for workload-specific characteristics such as memory usage, thread configuration, and GPU utilization.

Case Study: Migrating a Dynamic Product Ads Model

One of the first large-scale migrations to the new platform was Dynamic Product Ads (DPA), a commerce model responsible for selecting which advertiser products should be displayed within an advertisement. The system operates within the ad auction pipeline and must produce predictions within strict latency constraints while evaluating a potentially large set of candidate products from an advertiser’s catalog.

Legacy Architecture: Lookup Tables in Brain

Originally, DPA relied on precomputed lookup tables served from Brain, mapping feature combinations directly to predicted outcomes.

While this approach enabled fast inference, it broke down under the demands of product selection at scale. Each request required evaluating a large set of candidate products, and as advertiser catalogs grew, the number of possible feature combinations became increasingly difficult to manage.

More importantly, the system was entirely static. Predictions were derived from fixed mappings, with no ability to explore alternative actions or adapt based on new feedback. This made it fundamentally incompatible with learning-based approaches.

As we began exploring contextual bandits for product selection, it became clear that the lookup-table architecture could not support the dynamic decision-making required for real-time learning.

Framing Product Selection as a Contextual Bandit Problem

The Dynamic Product Ads problem maps naturally to the contextual bandit framework.

At each ad request, the system observes a context consisting of real-time signals such as:

• page domain
• ad slot identifier
• device characteristics
• user interaction signals
• advertiser catalog metadata

Given this context, the system must choose an action, which corresponds to selecting a specific product from the advertiser’s catalog to display in the ad.

Once the ad is served, the system observes partial feedback in the form of a reward signal (e.g., click or no-click). Importantly, the reward is observed only for the product that was shown, while the outcomes for all other candidate products remain unknown.

This partial-feedback setting makes the contextual bandit framework particularly well suited for the problem. Bandit algorithms allow the system to continuously balance exploration, which gathers information about potentially high-performing products. This prioritizes products that have historically performed well under similar contexts and enables the model to adapt to evolving user behavior, product inventory changes, and shifting advertiser objectives.

Building a Contextual Bandit Pipeline

We rebuilt the training pipeline using Vowpal Wabbit’s Conditional Contextual Bandit (CCB) framework.

The pipeline joins impression logs, click events, auction metadata, and product catalog context to generate training examples.

For the initial version, we focused on contextual signals such as:

• TOP_LEVEL_DOMAIN
• AD_SLOT_ID

Categorical features undergo normalization during preprocessing. String values are lowercased, whitespace is replaced with delimiters, and rare feature values are grouped into an "other" bucket to mitigate sparsity and reduce feature dimensionality.

Each training example is encoded in VW’s CCB format, which represents the contextual bandit interaction as a hierarchical structure consisting of:

• shared contextual features
• slot definitions representing decision points
• candidate actions corresponding to advertiser products
• action probabilities derived from the exploration policy
• observed reward signals (e.g., clicks)

This encoding allows the model to learn a policy over candidate products, optimizing expected click-through rate while accounting for exploration probabilities introduced during data collection.

The training pipeline now runs multiple times per day, enabling rapid policy updates as new interaction data becomes available. Each trained model artifact is registered in the Model Registry, where it is versioned and promoted through the deployment pipeline.

These models are tracked using standardized metadata tags, which allow us to manage versions, promote candidates to production, and ensure safe rollbacks when needed.

For DPA, we label models using standardized metadata to manage versions and deployments. For example, the contextual bandit model was registered under the dpa model family with version v300. Once validated in production, it was promoted as the champion model, while previous versions were retained only as fallback candidates.

Example model metadata:

model_family = dpa
model_version = v300
alias = champion

ML Inference Pipeline (New Vs Old)

The diagram shows how, for each DPA request, features are fetched from the feature store and combined with the latest model from the registry to generate predictions dynamically.

This enables continuous policy updates based on new interaction data, improving product selection and overall campaign performance.

Real-Time Serving with Sleipnir + Triton

In production, inference is executed via a Python sidecar on Sleipnir, packaged as a custom Triton Docker image with VW bindings.

At runtime, Brain forwards bid requests via gRPC to the VW sidecar, which has the latest model preloaded from the Model Registry. Hot reloads are triggered automatically when new champion models are promoted.

Triton then fetches contextual features, executes VW inference with batching and concurrency scheduling, and returns predictions to Brain in under 20 ms. These predictions are logged in the monitoring pipeline.

By containerizing each VW-based model, we ensure isolation, reproducibility, and optimized runtimes, while Sleipnir manages GPU allocation and scaling.

Operating at Real-Time Scale

Running at production scale has validated the platform design. Inference now handles millions of requests per second while meeting a sub-20 ms SLA. Models are retrained and deployed multiple times per day without downtime, and monitoring ties prediction quality directly to business KPIs. What once required static lookup tables now runs as a continuous, automated, observable system.

The platform provides a unified inference layer that supports multiple runtimes and execution patterns while scaling elastically under production workloads. By decoupling infrastructure from model execution, it enables efficient utilization of GPU resources and optimization of inference for both latency-sensitive and high-throughput use cases. The system is inherently future-proof, with support for Vowpal Wabbit, TensorFlow, PyTorch, and ONNX, and the flexibility to integrate emerging runtimes as modeling approaches evolve.

Today, Brain and Triton handle the bulk of our centralized inference. For models that demand even lower latency (sub-10ms), we are extending inference into the client itself. With the MLP Client, AuctionSDK, our client-side execution environment for auction-time decisioning, can load serialized ONNX models directly into memory, refresh them as new versions are released, and execute predictions locally.

The MLP client provides a simple predictor API that hides the complexity of model management. Under the hood, it handles:

• Model Loading & Refreshing: Keeps the latest ONNX models hot in memory, automatically refreshing when new versions are promoted.
• Feature Transformation: Applies the same preprocessing steps as training, ensuring training–serving parity.
• Prediction Execution: Runs models locally in-process (sub-millisecond inference) or routes to a Triton sidecar on Sleipnir when GPU acceleration is needed.
• Failover & Routing: Falls back to cached versions if updates fail, guaranteeing stability.

The client unlocks a dual-mode inference architecture:

• Centralized inference (Brain + Sleipnir): high throughput, observability, monitoring, and best for GPU-heavy models.
• Embedded inference (MLPClient in AuctionSDK): sub-millisecond predictions, no network latency, and ideal for ultra-sensitive use cases like auctions and personalization. The client can execute inference locally using embedded C++ runtimes, typically powered by ONNX Runtime or other optimized native inference libraries.

Together, these modes give us the best of both worlds, scale and stability at the core, speed and flexibility at the edge.

ML Client Predictor workflow illustrating model selection and automatic model updates from S3.

The figure above shows how we can execute two models in parallel using the MLPClient.

When GPU-based processing is required, the MLPClient can also route calls to Sleipnir via a Triton sidecar container. We will dive deeper into this pipeline in one of our upcoming blog posts.

What’s Next

The platform foundations are in place, but we’re only scratching the surface of what real-time ML can unlock. Our roadmap extends in four directions:

• Experimentation at scale: Turning the auction loop into a live experimentation engine, where models, features, and policies are continuously tested and automatically promoted based on performance.
• Personalization and reinforcement learning: Moving beyond CTR optimization toward impression-level personalization by combining contextual bandits with embeddings and inline reinforcement learning to optimize long-term advertiser ROI.
• Platform evolution: Extending the ML Platform Client with a unified feature engineering interface, enabling consistent feature fetching, transformation, and serving across models without bespoke pipelines.
• Pushing latency even lower: Driving toward sub-5 ms inference by expanding embedded ONNX execution in AuctionSDK and optimizing Triton and GPU scheduling for micro-batching.
• LLMOps and model serving infrastructure: Building a unified LLMOps pipeline for large model training, evaluation, and hosting, leveraging shared GPU infrastructure to support both traditional ML models and emerging LLM workloads under a common serving and deployment framework.

Acknowledgment

Special thanks to Simon Critchley, Bradley Harkrader, and the entire Machine Learning and Data Science team for their support, collaboration, and contributions to this work.