I Built a Vulkan Inference Engine in Rust — 8.9× Faster, Zero Native Binaries

Pure-Rust Vulkan inference for ONNX models — measured on 7 models, honest about where it loses

ONNX Runtime has execution providers for CUDA, TensorRT, DirectML, and CoreML — but not for Vulkan. That gap has a precise cost: AMD GPUs get nothing, cross-platform means CPU-only, and the Rust binding has been stuck at 2.0.0-rc.12 for twelve release candidates.

I built onnx-vulkan-rs: a standalone pure-Rust engine that loads .onnx files and runs them on a Vulkan GPU — no ONNX Runtime, no C++ binary, no CUDA. The architecture depends on the ONNX standard (stable protobuf schema, additive opsets) instead of ORT's evolving plugin API.

The key design choice: CPU compiles at load-time, GPU executes at run-time, zero readback mid-run. If one op is missing, the engine fails loud — never a silent fallback.

Measured on an RTX 4070 against ORT’s CPU EP:

• RF-DETR → 8.93× (41.1 ms vs 367.1 ms)
• SAM 3 ViT-H → 7.54× (1391.5 ms vs 12678.3 ms)
• Parakeet encoder int8 → 6.32× (46.6 ms vs 294.8 ms)

Every model runs at 1 convex GPU block. Op coverage is 100% on the vision/speech matrix.

The honest part: two models sat below 1× before split-K optimization. Their output tensors were smaller than the GPU — 768 useful threads out of 70,656. No tile enlargement fixes that. Split-K manufactured the missing parallelism by slicing along the reduction axis.

cargo add onnx-vulkan. Run ldd — only libc, libm, libgcc_s. Dual-licensed MIT/Apache-2.0.

Read the full deep-dive with architecture diagrams, kernel optimization details, and the SAM 3 OOM→5.6 GB

URL: https://autognosi.substack.com/p/i-built-a-vulkan-inference-engine