Training Edge-Deployable Models via Collaborative Annotation with Humans and Large Foundation Models

Experiences with OpenVINO in Google Summer of Code 2025

Vision foundation models have demonstrated state-of-the-art performance in computer vision and are increasingly being applied in real-world applications. These large foundation models (LFMs) acquire general image understanding through training on large-scale image datasets, and they can flexibly adapt to new downstream tasks.

However, there are several challenges in deploying such large models. First, state-of-the-art LFMs often contain millions or even billions of parameters, which inhibits deployment in resource-constrained environments such as embedded systems and IoT devices. Second, LFMs are typically adapted to specific downstream tasks through additional training on high-quality annotated datasets. Creating these datasets from scratch is labor-intensive and can become a bottleneck in system development.

A promising approach to address these issues is knowledge distillation. In this paradigm, the knowledge of a large “teacher” model is transferred to a smaller “student” model. Specifically, we focus on an output-based distillation setting in which student models imitate the predictions of the teacher model. In other words, the teacher model assigns labels to a dataset, and the student model is trained on it. This approach can reduce the need for manual annotation of every example while enabling efficient deployment of the student model on edge devices.

Yet this approach raises an important question: what happens if the teacher model is wrong? To address this question, our project enables human annotators to correct labels with minimal effort.

We propose a pipeline in which LFMs provide initial annotations, human annotators efficiently correct them, and lightweight models are trained on the refined data. In this way, the system implements a human-in-the-loop distillation process.

This work was carried out as part of Google Summer of Code (GSoC) 2025 with OpenVINO. In this project, we focused on object detection and incorporated the YOLO model series and Florence-2 into the pipeline.

Resources: **Project Page | Code Repository**

Pipeline Overview:

The pipeline consists of the following five main components (illustrated in Figure 1):

Figure 1. Overview of the human-in-the-loop distillation pipeline. LFMs assign initial annotations, human annotators correct informative samples, and small models are trained on the refined labels.

1. Automatic annotation with LFMs LFMs assign initial labels to the training dataset.
2. Selection of data samples to be reviewed The pipeline identifies which annotations should be prioritized for human correction by selecting data samples that are likely to have a large impact on model performance.
3. Human verification and correction Human annotators either accept or correct the labels generated by the LFMs.
4. Training of lightweight models Lightweight models are trained on the refined dataset. These models act as students in a knowledge distillation framework. They learn from both the LFMs’ outputs and the human-corrected labels.
5. Evaluation Users evaluate the task accuracy of the lightweight models. Based on the results, the pipeline selects additional data for correction and retrains the models.

By repeating these steps, the pipeline continuously improves both the dataset and the student models, while mitigating annotation costs.

Main Features of the Pipeline

Integrating Open-Vocabulary Models

In practical applications, the classes present in the pretraining dataset of foundation models do not always align with the objects that a user wants to annotate. To address this mismatch, the pipeline integrates open-vocabulary detection models such as YOLO-World and Florence-2. These models can detect objects specified by natural-language prompts rather than a fixed label set.

Open-vocabulary foundation models provide several advantages:

• Flexible class definitions The variety of classes is not limited to those that appear in the original training dataset. Instead, users can define task-specific concepts for their own domain.
• Prompt-based specification of targets Users can define what they want to detect using natural language descriptions, without retraining the foundation model. This enables immediate use of the model for annotation by simply providing appropriate prompts.

However, there may be a mismatch between expressions that are suitable for the model and those that accurately reflect user intent. For example, a user may want to distinguish between “ripe grapes” and “unripe grapes”, while the model may rely on visual cues such as color, e.g., “purple grapes” and “green grapes”.

To reconcile these differences, the pipeline introduces an ontology, i.e., a mapping between prompts and class names:

• The model is prompted with labels that focus on visual features (e.g., “purple grapes”, “green grapes”).
• The annotations are mapped to user-preferred class names (e.g., “ripe grapes”, “unripe grapes”, respectively).

This ontology-based mapping effectively leverages the potential of the foundation model while preserving the semantics necessary for downstream tasks.

Selecting Data Points for Human Correction

As human annotation is labor-intensive, the pipeline is designed to minimize manual inspection. To this end, it incorporates the following strategies to select the most informative samples for correction.

1. Missed detections by the LFM If the LFM fails to detect objects that should be annotated, human correction is essential. Identifying such cases ensures that important samples are not ignored.
2. Disagreement between LFMs and lightweight models When the predictions of the teacher and the student differ significantly, the corresponding samples are treated as bottlenecks in the training process. These informative instances highlight cases where the student model is struggling or where the teacher’s predictions may be unreliable. This disagreement is quantified using intersection-over-union (IoU) metrics.

By focusing human effort on reconciling these disagreements, the pipeline can effectively address bottlenecks in learning and improve both label quality and model performance.

Exporting Annotation Results in Various Formats

In practice, different training tools often require datasets in specific formats. To support diverse annotation formats, the project adopts Datumaro, an annotation management library in the OpenVINO toolkit ecosystem.

Datumaro enables users to handle multiple annotation formats easily by providing a unified API over popular formats such as COCO and YOLO. When a model requires a specific dataset format, Datumaro allows users to export the dataset in the required format. Thus, the pipeline smoothly integrates annotation results into various training workflows without manual format conversion.

Future Work

During the GSoC project, the core pipeline was implemented. There are several directions for future extension:

• Model and task expansion We plan to incorporate a wider variety of models for teachers and students. In addition, while the current project focuses on object detection, we aim to extend the pipeline to support image classification and semantic segmentation.
• Implementation of advanced sampling strategies We plan to investigate and integrate more sophisticated sample selection strategies. For example, active learning and uncertainty estimation methods can further reduce the amount of human annotation required while maximizing performance gains.

Conclusion

During GSoC 2025, we designed and implemented an end-to-end pipeline that includes:

• Automatic labeling with large foundation models
• Human-in-the-loop refinement of these annotations
• Training of lightweight, deployment-friendly models based on the refined annotations

The pipeline is integrated with components from the OpenVINO toolkit ecosystem and is designed to be extensible to additional models and tasks.

Acknowledgement

I sincerely appreciate Rajesh Gangireddy and Samet Akcay for their constructive suggestions and dedicated support during the GSoC period.