Deploying a YOLOv8 ONNX Ensemble on NVIDIA Triton Using Triton Control

See how Triton Control simplifies the path to NVIDIA Triton inference by scaffolding and deploying a YOLOv8 preprocessing → ONNX inference → postprocessing pipeline from one workspace.

Co-written by **Koffi Tino Gnagniko**

In the first article in this series, we moved individual models from an artifact or an existing repository to a running NVIDIA Triton endpoint. Real inference APIs often need more than a single model, however. An object detector, for example, rarely accepts a convenient image and immediately returns final bounding boxes.

The image must be converted into the tensor layout expected by the network. The raw prediction tensor must then be filtered, decoded, and reduced with non-maximum suppression. If clients perform those steps themselves, every application must reproduce the same model-specific logic.

This article packages the complete process as a NVIDIA Triton ensemble:

preprocess -> yolov8_onnx -> postprocess

The public model is yolo_pipeline. A client sends one UINT8 image and receives boxes, scores, and class IDs. Triton routes the intermediate tensors internally, so the client makes only one inference request.

The complete, runnable project is available in the YOLOv8 Object Detection Ensemble example.

What We Are Building

The pipeline uses two execution backends plus Triton’s ensemble scheduler:

• The Python backend normalizes the input image before inference and decodes the predictions afterward.
• The ONNX Runtime backend executes the exported YOLOv8n model.
• The ensemble scheduler connects the three model steps and exposes them as one public model.

The external and internal tensor contracts are:

An ensemble is not another copy of the model and does not have its own execution instance. It is the dataflow definition that tells Triton which child model to run when a tensor becomes available. Each child model still controls its own backend and instance_group.

The Triton ensemble mechanism is not limited to ONNX. At the serving layer, an ensemble can connect models implemented with different Triton backends as long as the mapped tensors have compatible names, data types, and shapes.

The wizard scaffolds standard Triton repository files using Python, ONNX Runtime, TensorRT, and LibTorch templates. Because the generated files remain editable, any child can be replaced as long as its input and output tensor contracts remain compatible with the adjacent steps.

Start in a Triton Control Development Workspace

In Triton Control, open Development and create a workspace with:

The Triton Control workspace form: the Triton 26.06 development image, Development installation disabled, 20Gi of storage, and zero workspace GPUs.

The workspace itself does not need a GPU. It only creates the repository and exports the ONNX artifact. The deployed Triton instance will use the GPU.

When the workspace is ready, open code-server. Triton Control provides its deployment plugin inside this browser-based editor, so repository creation, model export, configuration, and deployment can happen in the same persistent workspace.

Install the Jupyter extension from the code-server extension marketplace if it is not already available. If the notebook then has no Python kernel, install the kernel packages in the code-server terminal:

pip install notebook ipykernel

Scaffold the Repository with the Code-Server Plugin

Open the Triton Control icon in the code-server Activity Bar and select New Model Repository. The command is also available from the command palette.

The plugin can scaffold either a Single model repository or an Ensemble pipeline containing multiple model steps. A single-model scaffold creates one model folder with the selected backend template. For this object-detection workflow, choose the ensemble option so preprocessing, ONNX inference, and postprocessing are created inside one repository.

Choose the following values:

1. Select Ensemble.
2. Enter model as the repository name.
3. Select the Python → ONNX Runtime → Python preset.
4. Enter yolo_pipeline as the ensemble model name.
5. Keep preprocess as the first step name.
6. Enter yolov8_onnx as the second step name.
7. Keep postprocess as the third step name.

The ensemble preset creates the three child-model folders and a separate public ensemble model. This is the bundled plugin running inside a Triton Control-managed code-server workspace.

The generated scaffold gives us the required Triton layout, but it is intentionally only a template. Replace its placeholder Python files and config.pbtxt files with the corresponding files from the complete example directory, or edit them to match the contracts in the next section. Copy the example's two client scripts and export notebook into the model/ repository root as well.

The completed model repository looks like this:

model/
├── preprocess/
│   ├── config.pbtxt
│   └── 1/
│       └── model.py
├── yolov8_onnx/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx
├── postprocess/
│   ├── config.pbtxt
│   └── 1/
│       └── model.py
└── yolo_pipeline/
    ├── config.pbtxt
    └── 1/
        └── .keep

Every served model has its own directory and config.pbtxt. Executable artifacts live below a numeric version directory. The ensemble version directory contains .keep because S3-compatible object storage cannot preserve an empty directory, while Triton still expects a model version folder.

Define the Child Model Contracts

config.pbtxt is more than deployment metadata. In an ensemble, it is the interface between steps. A tensor name, type, or dimension that differs on either side of a connection prevents Triton from loading the pipeline.

Preprocessing

The Python preprocessing model accepts an RGB image in height-width-channel layout:

name: "preprocess"
backend: "python"
max_batch_size: 0

input [
  {
    name: "IMAGE"
    data_type: TYPE_UINT8
    dims: [ 640, 640, 3 ]
  }
]

output [
  {
    name: "PREPROCESSED_IMAGE"
    data_type: TYPE_FP32
    dims: [ 1, 3, 640, 640 ]
  }
]

instance_group [{ count: 1 kind: KIND_CPU }]

Its model.py converts the values to FP32, scales them from 0–255 to 0–1, adds a batch dimension, and transposes the image from HWC to NCHW:

images = pb_utils.get_input_tensor_by_name(request, "IMAGE").as_numpy()
if images.ndim == 3:
    images = images[None, ...]
images = images.astype(np.float32) / 255.0
images = np.transpose(images, (0, 3, 1, 2)).astype(np.float32)

This example expects the incoming image to already be exactly 640 × 640. The optional Python client performs that resize. If your production API must accept arbitrary resolutions, resizing or letterboxing belongs in this preprocessing step, together with the coordinate transformation needed to map boxes back to the original image.

ONNX inference

The ONNX model consumes the preprocessed tensor and produces YOLOv8n’s raw detections:

name: "yolov8_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 0

input [
  {
    name: "images"
    data_type: TYPE_FP32
    dims: [ 1, 3, 640, 640 ]
  }
]

output [
  {
    name: "output0"
    data_type: TYPE_FP32
    dims: [ 1, 84, 8400 ]
  }
]

instance_group [{ count: 1 kind: KIND_GPU }]

The names images and output0 come from the exported ONNX graph. Do not guess them. The export notebook prints the graph inputs and outputs so they can be checked against the configuration.

For a CPU-only deployment, change KIND_GPU to KIND_CPU and set the deployment GPU count to zero.

Postprocessing

The postprocessing Python model receives RAW_DETECTIONS. It applies a confidence threshold, converts center-width-height values into corner coordinates, normalizes the boxes, and applies class-agnostic non-maximum suppression. It returns a variable number of detections:

input [
  {
    name: "RAW_DETECTIONS"
    data_type: TYPE_FP32
    dims: [ 1, 84, 8400 ]
  }
]

output [
  { name: "BOXES"    data_type: TYPE_FP32 dims: [ -1, 4 ] },
  { name: "SCORES"   data_type: TYPE_FP32 dims: [ -1 ] },
  { name: "CLASS_IDS" data_type: TYPE_INT64 dims: [ -1 ] }

The -1 dimensions are important: the number of detections is not known until postprocessing finishes. The class-agnostic NMS keeps this example compact, but a production detector may need class-aware suppression to avoid removing overlapping objects from different classes.

Connect the Models in the Ensemble

The ensemble configuration defines the public API and maps tensors between the child models:

name: "yolo_pipeline"
platform: "ensemble"
max_batch_size: 0

input [
  {
    name: "IMAGE"
    data_type: TYPE_UINT8
    dims: [ 640, 640, 3 ]
  }
]

output [
  { name: "BOXES"     data_type: TYPE_FP32 dims: [ -1, 4 ] },
  { name: "SCORES"    data_type: TYPE_FP32 dims: [ -1 ] },
  { name: "CLASS_IDS" data_type: TYPE_INT64 dims: [ -1 ] }
]

ensemble_scheduling {
  step [
    {
      model_name: "preprocess"
      model_version: 1
      input_map  { key: "IMAGE" value: "IMAGE" }
      output_map {
        key: "PREPROCESSED_IMAGE"
        value: "PREPROCESSED_IMAGE"
      }
    },
    {
      model_name: "yolov8_onnx"
      model_version: 1
      input_map  { key: "images"  value: "PREPROCESSED_IMAGE" }
      output_map { key: "output0" value: "RAW_DETECTIONS" }
    },
    {
      model_name: "postprocess"
      model_version: 1
      input_map { key: "RAW_DETECTIONS" value: "RAW_DETECTIONS" }
      output_map { key: "BOXES" value: "BOXES" }
      output_map { key: "SCORES" value: "SCORES" }
      output_map { key: "CLASS_IDS" value: "CLASS_IDS" }
    }
  ]
}

There are two naming layers in every map:

• key is the input or output name declared by the child model.
• value is the tensor name inside the ensemble.

For example, the ONNX model’s output0 becomes RAW_DETECTIONS inside the pipeline. That internal tensor is then connected to the postprocessing model's RAW_DETECTIONS input.

Reading the file as a dataflow graph is easier than reading it as a list of settings:

ensemble IMAGE
  -> preprocess.IMAGE
  -> preprocess.PREPROCESSED_IMAGE
  -> yolov8_onnx.images
  -> yolov8_onnx.output0
  -> postprocess.RAW_DETECTIONS
  -> ensemble BOXES + SCORES + CLASS_IDS

Export YOLOv8n to ONNX

Place export_yolov8_to_onnx.ipynb in the repository root, /workspace/model/, alongside the four model directories. If you copied the supporting files as described above, the layout at this point includes:

/workspace/model/
├── export_yolov8_to_onnx.ipynb
├── infer_client.py
├── make_request_payload.py
├── preprocess/
├── yolov8_onnx/
├── postprocess/
└── yolo_pipeline/

Open export_yolov8_to_onnx.ipynb from that location and run all of its cells.

It installs the export dependencies, downloads YOLOv8n through Ultralytics, and exports a fixed-shape ONNX graph:

model = YOLO("yolov8n.pt")
export_path = model.export(
    format="onnx",
    imgsz=640,
    opset=12,
    simplify=True,
    dynamic=False,
)

Open export_yolov8_to_onnx.ipynb from that location and run all of its cells.

Because the notebook uses a path relative to its working directory, running it from /workspace/model/ places the result at:

/workspace/model/yolov8_onnx/1/model.onnx

The binary is deliberately not stored in the example repository. Exporting it in the workspace makes the source, export parameters, and resulting artifact explicit.

Before deployment, confirm that:

• model.onnx exists under yolov8_onnx/1/.
• The graph input is images.
• The graph output is output0.
• All four model folders contain config.pbtxt.
• yolo_pipeline/1/.keep is present.
• The ONNX instance_group matches the CPU or GPU resources you will deploy.

Deploy Directly from Code-Server

In the code-server Explorer, right-click the repository root — the folder that contains all four model directories — and select Triton Control: Deploy Model Repository.

Run the deployment action on the repository root so the complete ensemble is uploaded together. This context menu is shown in the managed code-server workspace.

The plugin finds the public ensemble’s config.pbtxt, uses it to detect yolo_pipeline and the ensemble platform, and opens the deployment form. It uploads every file below the selected repository root, but it does not validate all child configurations or tensor mappings.

Review those contracts before deployment. Then select the S3 profile and confirm the upload target, Triton image, model-control settings, and resources.

Before deployment, the managed code-server plugin previews the source folder, Triton image, detected public platform, model-control mode, S3 profile, and final upload target

Important: because the plugin detects the public model as an ensemble, it does not infer the GPU requirement from the yolov8_onnx child configuration. The form therefore defaults to 0 GPUs. Expand Resources and explicitly set GPU count to at least 1. If you changed the ONNX model to KIND_CPU, keep the GPU count at 0.

The plugin performs two related operations: it uploads the complete repository to the selected S3-compatible location, then creates the Triton deployment with that location as its model repository. After deployment, Triton Control opens the new instance and its logs.

Deploy the repository root, not only yolov8_onnx. The public ensemble cannot load unless preprocess, yolov8_onnx, postprocess, and yolo_pipeline are all available in the same repository.

Test the Public Pipeline

The example includes two ways to test the deployment.

For a quick protocol-level check, generate a JSON request containing a zero-filled 640 × 640 × 3 image:

python3 make_request_payload.py

Open the deployed instance in Triton Control, select yolo_pipeline, open Inference, paste the contents of request.json into the manual input view, and run inference.

The same request can be sent to the HTTP endpoint:

curl -X POST "http://localhost:8000/v2/models/yolo_pipeline/infer" \
  -H "Content-Type: application/json" \
  --data-binary @request.json

Replace localhost:8000 with the instance endpoint unless you are using a local port-forward.

The zero image is useful as a smoke test, but it is not a meaningful object-detection test. For a real image, use the Python client:

pip install "tritonclient[http]" pillow numpy
python infer_client.py path/to/image.jpg --url your-triton-host:8000

The client resizes the image to 640 × 640, calls only yolo_pipeline, and prints the first detections as class IDs, confidence scores, and normalized boxes. It never calls the three child models directly.

Why the Ensemble Boundary Matters

That boundary is the main architectural benefit: clients depend only on IMAGE -> BOXES, SCORES, CLASS_IDS, while preprocessing and postprocessing remain versioned and deployed with the network.

The example is intentionally simple. It fixes the input size, disables batching with max_batch_size: 0, and implements postprocessing in NumPy. A production version may add letterboxing, original-image metadata, batch-aware shapes, tuned model instances, richer labels, and performance measurements. Those are changes to the pipeline behind the endpoint, not changes every client must independently implement.

Conclusion: What Triton Control Changes

Triton executes the ensemble; Triton Control standardizes the workflow around it — from repository scaffolding and S3 upload to Kubernetes deployment, logs, and endpoint testing. This makes the path from an exported artifact to a tested service repeatable across models and teams.

It does not replace model engineering. Model export, preprocessing, tensor contracts, runtime dependencies, resource choices, and performance validation remain our responsibility.

In the next article, we will serve Phi-3 Mini with vLLM and show how model.json controls batching, context length, GPU memory, and model loading.

Links

• Complete YOLOv8 ONNX ensemble example
• Triton Control on GitHub
• Triton Control development workspaces
• NVIDIA Triton ensemble models
• NVIDIA Triton model configuration
• NVIDIA Triton model repositories