I Shrank a YOLO Model by 10x and Deployed It on a $10 Chip — Here’s Everything I Learned

How knowledge distillation, TensorRT INT8 quantization, and a custom .kmodel conversion pipeline got 22 FPS out of a K230 embedded processor on a real industrial textile line.

I work on a textile manufacturing line in Casablanca. Not glamorous, but the problem is real: track every fabric piece as it moves through operator stations, flag anomalies in real time, and do it all on cheap embedded hardware bolted to a factory wall. No GPU. No cloud. A K230 chip running at the edge.

The original model — YOLOv12-XLarge — was beautiful. 96.2% mAP@50 on our 40,000-frame dataset (annotated with SAM2, which deserves its own article). But it needed a GPU just to breathe. Deploying it on a K230 was a non-starter.

So I spent three months doing what the edge AI world calls the “compression gauntlet”: knowledge distillation → ONNX export → TensorRT INT8 calibration → .kmodel conversion → K230 inference. This article is the one I wish I had before I started.

The Problem With “Just Use a Smaller Model”

The obvious answer is: train a small model from scratch. Use YOLOv12-nano, slap your dataset on it, call it a day.

I tried that. The nano baseline trained from scratch hit 88.4% mAP@50 on our textile dataset. That sounds fine until you realize the teacher sits at 96.2%, and in a factory context, that 7.8-point gap means missed defects, scrapped rolls, real money. The line manager said no.

Knowledge distillation is the less obvious answer — and the one that actually worked. Instead of training the small model to mimic ground truth labels, you train it to mimic the behavior of the large model. The student learns not just “this region is a piece” but the teacher’s confidence distribution over all classes — richer signal than a hard 0/1 label.

The result: distilled YOLOv12-nano hit 91.8% mAP@50. Under 5 points gap — acceptable for production. Then INT8 quantization pushed it to 22 FPS on the K230 with 40%+ latency reduction. That is the number that got the project greenlit.

The Full Pipeline

[40k Textile Frames]     [SAM2 Annotations]
         |                        |
         +------------+-----------+
                      |
          +-----------v-----------+
          |   YOLOv12-XLarge      |   Teacher (frozen)
          |   96.2% mAP@50        |
          +-----------+-----------+
                      |  soft logits + neck features
                      v
          +-----------+-----------+
          |   KD Training Loop    |   L = a*L_task + b*L_kd + c*L_feat
          +-----------+-----------+
                      |
                      v
          +-----------+-----------+
          |   YOLOv12-nano        |   Student (distilled)
          |   91.8% mAP@50        |
          +-----------+-----------+
                      |
                      v
          +-----------+-----------+
          |   ONNX Export         |   opset 17, static shapes
          +-----------+-----------+
                      |
                      v
          +-----------+-----------+
          |   TensorRT INT8       |   MinMax calibration, 1k frames
          +-----------+-----------+
                      |
                      v
          +-----------+-----------+
          |   nncase -> .kmodel   |   K230 native format
          +-----------+-----------+
                      |
                      v
          +-----------+-----------+
          |   K230 Inference      |   22 FPS / 45ms latency
          +-----------------------+

Step 1 — The Distillation Loss

Three components. Most tutorials only mention the first one.

Task loss (L_task): standard YOLO detection loss. Keeps the student honest against ground truth.

Response-based KD loss (L_kd): soft cross-entropy between teacher and student logits. Temperature T controls softness. Higher T = richer gradient signal from low-confidence classes.

Feature-based loss (L_feat): L2 distance between intermediate neck feature maps. Secret ingredient. Without it, the student matches the teacher’s predictions but misses the intermediate representations that make the model robust to lighting variation.

import torch
import torch.nn as nn
import torch.nn.functional as F

class KnowledgeDistillationLoss(nn.Module):
    def __init__(self, temperature=4.0, alpha=0.4, beta=0.4, gamma=0.2):
        super().__init__()
        self.T = temperature
        self.alpha = alpha   # task loss weight
        self.beta = beta     # response KD weight
        self.gamma = gamma   # feature KD weight

    def response_kd_loss(self, student_logits, teacher_logits):
        s = F.log_softmax(student_logits / self.T, dim=-1)
        t = F.softmax(teacher_logits / self.T, dim=-1)
        return F.kl_div(s, t, reduction="batchmean") * (self.T ** 2)

    def feature_kd_loss(self, student_feats, teacher_feats, adapters):
        total = torch.tensor(0.0, device=student_feats[0].device)
        for s, t, adapter in zip(student_feats, teacher_feats, adapters):
            total += F.mse_loss(adapter(s), t.detach())
        return total / len(student_feats)

    def forward(self, s_logits, t_logits, s_feats, t_feats, task_loss, adapters):
        l_kd   = self.response_kd_loss(s_logits, t_logits)
        l_feat = self.feature_kd_loss(s_feats, t_feats, adapters)
        total  = self.alpha * task_loss + self.beta * l_kd + self.gamma * l_feat
        return {"total": total, "task": task_loss, "kd": l_kd, "feat": l_feat}

The adapter trap: 1x1 conv layers that project student channels (64/128/256) up to teacher dims (256/512/1024) need Kaiming normal init. Default PyTorch init causes the feature loss to explode early and the student collapses. Add a 0.01 LR warmup on adapters for the first 5 epochs.

Step 2 — ONNX Export

from ultralytics import YOLO

model = YOLO("runs/distill/best.pt")
model.export(
    format="onnx",
    imgsz=640,
    opset=17,
    dynamic=False,    # nncase requires static shapes — do not skip this
    simplify=True,
    half=False,
)

dynamic=False is the most important flag. nncase fails with a cryptic shape inference error if dynamic axes are present. I lost a day to this.

Step 3 — TensorRT INT8 Calibration

import tensorrt as trt
import numpy as np
import cv2
from pathlib import Path

class TextileINT8Calibrator(trt.IInt8MinMaxCalibrator):
    def __init__(self, calib_dir, cache_file, batch_size=8):
        super().__init__()
        self.cache_file = cache_file
        self.batch_size = batch_size
        self.paths = list(Path(calib_dir).glob("*.jpg"))[:1000]
        self.index = 0
        import pycuda.driver as cuda
        import pycuda.autoinit  # noqa
        self.buf = cuda.mem_alloc(batch_size * 3 * 640 * 640 * 4)

    def _prep(self, p):
        img = cv2.imread(str(p))
        img = cv2.resize(img, (640, 640))
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
        return np.transpose(img, (2, 0, 1))

    def get_batch(self, names):
        if self.index >= len(self.paths):
            return None
        import pycuda.driver as cuda
        batch = np.stack([self._prep(p) for p in
                          self.paths[self.index:self.index + self.batch_size]])
        self.index += self.batch_size
        cuda.memcpy_htod(self.buf, batch.ravel().astype(np.float32))
        return [int(self.buf)]

    def get_batch_size(self): return self.batch_size
    def read_calibration_cache(self):
        if Path(self.cache_file).exists():
            return open(self.cache_file, "rb").read()
    def write_calibration_cache(self, cache):
        open(self.cache_file, "wb").write(cache)

Why MinMax and not Entropy? Fabric has occasional bright specular highlights. Entropy calibration clips tails aggressively — a clipped highlight is an invisible defect.

Mermaid Diagram — Render at mermaid.live

graph TD
    A[40k Textile Frames + SAM2 Annotations] --> B[DataLoader 640x640 augmented]
    B --> C[YOLOv12-XLarge Teacher frozen 96.2 mAP]
    B --> D[YOLOv12-nano Student trainable]
    C -->|soft logits| E[Response KD Loss KL Divergence T=4]
    C -->|neck features| F[Feature KD Loss L2 + adapters]
    D --> E
    D --> F
    D --> G[Task Loss YOLO box+cls+obj]
    E --> H[Combined Loss]
    F --> H
    G --> H
    H -->|backprop| D
    D --> I[best.pt 91.8 mAP]
    I --> J[ONNX opset17 static 640x640]
    J --> K[TensorRT INT8 MinMax 1000 frames]
    K --> L[nncase k230 target]
    L --> M[.kmodel 3.8 MB]
    M --> N[K230 KPU 22 FPS 45ms]

Benchmark Results

Teacher vs. Student

ModelParamsmAP@50FPS K230Latency msSize MBYOLOv12-XLarge (teacher)59.1M96.2% — — 224YOLOv12-nano from scratch2.6M88.4%147110.2YOLOv12-nano distilled2.6M91.8%166210.2YOLOv12-nano distilled + INT82.6M91.1%22453.8

Float32 vs INT8 on K230

PrecisionFPSLatency msmAP@50Memory MBFloat32166291.8%38INT8224591.1%14Delta+38%−27%−0.7pp−63%

Demo GIF — What to Record

Mount a camera above the textile conveyor at ~1.5m. Record 15–20s:

1. Fabric pieces entering frame — bounding boxes appearing in real time
2. A deliberate defect (fold, misalignment) — box color changes to red
3. Bottom-left overlay: FPS ~22, latency ~45ms, piece count
4. Operator hand entering frame — no false positives

30fps screen capture, resize to 640×360, keep under 5MB.

Lessons

Temperature sweep is mandatory. T=2.0 (literature default) was wrong for our dataset (80% clean fabric). T=4.0 gave the student proper gradient signal on minority defect classes. Sweep over {2, 3, 4, 6} before committing.

Feature adapter init. Default PyTorch init → feature loss explodes → student collapses. Kaiming normal + 0.01 LR warmup on adapters for first 5 epochs.

Calibration quality beats volume. 5,000 random frames → 2.1pp drop. 1,000 curated frames (one per lighting condition, one per defect type) → 0.7pp drop.

nncase operator whitelist. Validate your ONNX against the K230 SDK nncase version whitelist before building the TRT engine. A broken .kmodel from an unsupported op is silent and painful.

Key Takeaways

• Distillation closes the gap that architecture scaling alone cannot
• Temperature matters more than architecture choice for KD quality
• Feature-based KD added +1.7pp mAP vs response-only — not optional for detection
• INT8 calibration: quality > quantity. 1k curated > 5k random
• MinMax > Entropy for detection in industrial environments
• The full stack is the product. Validate after every conversion step

The K230 is a $10 chip running a model that learned from 59M parameters. 3.8MB. 22 FPS. That is what taking the full pipeline seriously looks like.

Achraf Lamia — ML & Computer Vision Engineer, Casablanca.