Paper Review: DINOv3 The New State of the Art in Self-Supervised Vision Models

Recently , Meta AI released DINOv3 a new family of vision foundation models that produce exceptionally clean and robust features(something I have not seen before ) . After exploring the models myself including the powerful 7B parameter version it’s clear that DINOv3 represents a major change in self-supervised learning (SSL) . These models are so powerful , and I believe they could serve as the visual backbone (vision tower) for building a powerful Vision-Language Models (research paper idea). Anyway, let’s deep dive theoretically and do some visualization using C***olab* and [HugingFace](https://huggingface.co/spaces/sayedM/DINOv3-features) **space

DINOv3 Features — a Hugging Face Space by sayedM

The paper at its beginning shows a summarized list of contributions

1. “we leverage the benefit of scaling both dataset and model size by careful data preparation, design, and optimization”.
2. “We introduce a new method called Gram anchoring, which effectively addresses the known yet unsolved issue of dense feature maps degrading during long training schedules”.
3. “we apply post-hoc strategies that further enhance our models’ flexibility with respect to resolution, model size, and alignment with text”

https://ai.meta.com/dinov3/

In this paper they followed same technique by using Self-Supervised learning SSL as they did with previous dino because it demonstrates how great it’s when it comes to huge non annotated dataset which is the common case and anyone has used the DINO models as a feature extractor he would understand how great it was allowing a single frozen backbone to serve multiple purposes by just training few Parmerters on top of it

But SSL is not a heaven it has its own challenges

1. it is unclear how to collect useful data from unlabeled collections.
2. in usual training practice, employing cosine schedules implies knowing the optimization horizon a priori (you have no idea when you should stop).
3. the performance of the features gradually decreases after early training (model neglect pixel/dense level features and focus on high level features) this problem confirmed by visual inspection of the patch similarity maps. This phenomenon appears in longer training runs with models above ViT-Large size (300M parameters) (this is the core of the paper ) <==

Data Preparation

Increasing the dataset is not always the right decision if done randomly and may not reflected into enhancement in the downstream tasks and there are two schools in this area by focusing on improving data diversity and balance to cover all possible tasks with almost the same weight , or data usefulness its relevance to common practical applications eg custom medical field dataset starting the Data Collection and Curation process with about 17B images from Instagram probably your picture in Bali included in the training set They have created 3 data partes out of these full 17B images

First Part : by applying an automatic data curation process using a hierarchical k-means by using Dinov2 to generate images embeddings then use 5 levels of clustering first create 200M clusters then clustering those 200M centers into 8M then clustering → 800K → 100K → finally 25K then applied a balanced sampling -check the image — This results in a curated subset of 1,689M images (named LVD-1689M) that guarantees a balanced visual concepts coverage.

2405.15613

Clustering idea Level 1 (Lowest): The initial images are grouped into 200 million very specific clusters. This is the most detailed level of grouping. (e.g., a cluster for “golden retrievers catching a red frisbee”)

Level 2: The centers of those 200M clusters are then grouped into 8 million broader clusters. (e.g., a cluster for “dogs playing fetch”)

Level 3: The centers of the 8M clusters are grouped into 800,000 even broader clusters. (e.g., a cluster for “domesticated animals outdoors”)

Level 4: The centers of the 800k clusters are grouped into 100,000 general clusters. (e.g., a cluster for “animals”) Level 5 (Highest): Finally, the centers of the 100k clusters are grouped into 25,000 top-level, very broad categories. (e.g., a cluster for “nature and wildlife”)

Second Part: (retrieve relevant images) retrieval based on similarity. they get a dataset called “Seed Datasets” which most probably high-quality dataset used to retrieve similar images from the 17B images so the dino perform well on downstream tasks

Third part: (Opensource / well-known dataset) High quality opensource dataset used to make sure the model is powerful on downstream tasks and to compete with other solutions as well

Training Data Sampling

Training done by using 90% Heterogeneous Batches most of the time and 10% Homogeneous Batches where Heterogeneous Batches contains data from LVD-1689M ,the retrieved images and open source datasets like ImageNet22k and Mapillary while Homogeneous Batches contains images from the clean opensource images ImageNet1k so the model which guarantees the model is remembering the required features (which will be used then with downstream tasks )

Training

The paper main contribution is about training SSL 7B parameters we could summarize it into two main steps based on my understanding

Step 1: Initial Pre-training Phase (L_Pre)

This is the step where trainning happened for 1M iterations with a goal to learn the most important features using 3 losses

• L_DINO​: An image-level (global) objective that helps the model understand the overall scene.
• L_iBOT​: A patch-level (local) objective that reconstructs latent features, teaching the model about finer details.
• L_Koleo​: A regularizer that encourages the learned features to spread out, preventing them from collapsing into a narrow space and improving diversity.

So at this step we are dealing with patch level loss/Features ,image level feature like the previous Dino version and the regulization with Koleo

The Koleo regularizer is like telling everyone in a room to maintain personal space by moving away from their nearest neighbor, causing the whole group to spread out evenly. it was useful to Prevent Representational Collapse: Stops the model from producing same output features for different inputs by forcing rep.. to be distinct.

Enhances Feature Diversity: Encourages features to spread out uniformly in the embedding space Improves Representation Quality: The resulting diverse features are more effective and robust for a wide range of downstream tasks .

DINOv3 applies this loss on small, distributed batches (16 samples) to gain the benefits without excessive computational cost.

some points

• In order to improve the robustness of the model to resolutions, scales and aspect ratios, they employ RoPE-box jittering
• they get rid of all parameter scheduling, and train with constant learning rate, weight decay, and teacher EMA momentum (as you do not know after how many patches you should stop)
• AdamW -4096 images split across 256 GPUs
• they used square images with a side length of 256/112 pixels for global/local crops resulting to a total sequence length of 3.7M tokens per batch “*(2(256/16)² +8(112/16)²)4096= 3502080”**

Step 2: Refinement Phase (L_Ref)

After the initial training, the researchers observed that while performance on global tasks (like classification) kept improving , performance on dense tasks (like segmentation and detection ) started to degrade.

To deal with this issue they added a new objective that guarantees this further training does not hurt the performance on dense tasks this objective called “Gram Anchoring” Now our loss welcoming a new member

Gram Anchoring

it’s about mitigating the degradation of patch-level consistency by enforcing the quality of the patch-level consistency The process works as follows:

1. an early checkpoint where the patch level features still meaningful , then use it as a reference (Teacher)
2. both models generate feature maps for same image we have now two matrices of P *d
3. normalize the features
4. create Gram Matrix for each model
5. get the loss (how far they are) By minimizing this loss, the student is encouraged to produce features that have a similar internal structure to the teacher’s

this will make the features have some sort of freedom because it’s on the gram matrix so it somehow dealing with pixel level /patch losses & features without forcing specific values (as the loss operates on the structure of similarities between patches, not the exact feature values themselves) structure of similarities between patches remains consistent with the healthier teacher model . This approach successfully “repairs” the degraded local features , leading to significant and almost immediate improvements in dense task performance .

Now let’s deal with Higher-Resolution Features

The main Idea : They combined two insights: feeding a model a higher-resolution image produces more detailed and consistent features, and smoothing/averaging features can improve their quality (smart)

Instead of just using the standard-resolution image, they implemented a clever two-step process:

1. High-Res: First, they fed an image to the Gram teacher at twice the normal resolution (e.g., 512x512 instead of 256x256). This created a superior , highly detailed feature map.
2. Down-sample & Smooth: They then down-sampled this high-resolution feature map back to the student’s original size. This process acts as a smoothing filter , preserving the excellent patch consistency from the high res version while creating a cleaner reference map.

This technique effectively “distills” the superior consistency of the high-resolution features into the student model during the refinement phase. This simple trick yielded significant additional gains, for example, boosting performance by an extra +2 mIoU on the ADE20k segmentation benchmark.

They confirmed that using a Gram teacher from an early stage of training (e.g., the 200k iteration checkpoint) is crucial. Using a teacher from a much later stage 1M iterations was actually harmful because its features were already degraded

Post-Training

High-Resolution Adaptation

most of us tried DinoV3 and found that it was able to deal with 4k pixels images with a consistent feature map

The model is trained for a short period (10k iterations) on a mix of higher-resolution images (up to 768 pixels). Crucially, Gram Anchoring is used during this phase to prevent the dense features from degrading when faced with the complexity of high-resolution inputs leading to better performance in dense tasks like segmentation and tracking

Model Distillation

As computer vision engineers we are afraid of the idea of importing the mindset of sub billions parameters models from NLP as our clients will ask for running it using a raspberry pi XD but we still covered by this Distillation as they provided a huge family of models VITs and ConvNext

interesting fact that they even distilled the ViT-7B (a transformer ) into ConvNeXt models ( a CNN architecture ) highlighting the versatility of the process

The paper also introduces an efficient multi-student distillation pipeline, which allows training multiple students simultaneously by sharing the computationally expensive forward pass of the teacher model across all GPUs (without Gram anchoring)

Text Alignment CLIP like model

A key enhancement is that they align text with both the global (CLS token) and local (patch embeddings) features of the image, leading to better performance on dense, localized tasks.

mean pooling the local patch embeddings and concatenate them with the CLS token to get compared to text embeddings

dense feature space into 3 dimensions using principal component analysis (PCA)

Performance on Dense Feature Tasks

DINOv3 shows a significant leap in performance for tasks that rely on high-quality local (dense) features.

Dense Linear Probing (Segmentation & Depth Estimation):

• On the ADE20k semantic segmentation benchmark, DINOv3 outperforms previous self-supervised models by over 6 mIoU points and weakly-supervised models by over 13 points2.
• It surpasses strong specialized models like AM-RADIOv2.5 (distilled from SAM) by nearly 3 mIoU points on ADE20k3.
• For monocular depth estimation on NYUv2 and KITTI, DINOv3 again outperforms all other models by significant margins4.

3D Correspondence Estimation:

• DINOv3 achieves state-of-the-art results in matching keypoints across different 3D views of an object.
• On the NAVI dataset (geometric correspondence), it improves over the previous best model, DINOv2, by 4.3% recall6.

Unsupervised Object Discovery:

• Using the Token Cut algorithm, DINOv3’s clean feature maps significantly outperform all other backbones, including its predecessor DINOv2, which struggled with this task due to feature artifacts.

Video Segmentation and Tracking:

• The model demonstrates strong temporal consistency without any video-specific training8.
• On the DAVIS 2017 dataset, it achieves a J&F score of 83.3, outperforming DINOv2 by a large margin of 6.7 points9.

Performance on Global Feature Tasks

For tasks requiring an understanding of the entire image, DINOv3 is the first self-supervised model to achieve performance comparable to the best weakly and fully-supervised models10.

Image Classification:

• It significantly surpasses previous self-supervised models on out-of-distribution datasets like ImageNet-Rendition (+10%) and ObjectNet (+13%) compared to DINOv2.
• On the challenging iNaturalist 2021 fine-grained benchmark, DINOv3 achieves the highest accuracy (89.8%), outperforming even the top weakly-supervised models like PEcore.

Instance Recognition (Image Retrieval):

• DINOv3 achieves the strongest performance by large margins across several landmark and artwork retrieval benchmarks13. It improves over DINOv2 by +10.8 points on the Met dataset and +7.6 points on AmsterTime.

Performance as a Backbone for Complex Systems

When used as a frozen feature extractor for more complex, state-of-the-art systems, DINOv3 consistently pushes performance boundaries.

Object Detection:

• A lightweight detector (100M trainable parameters) built on a frozen DINOv3 backbone sets a new state of the art on the COCO dataset with 66.1 mAP15. This is the first competitive detection model to use a completely frozen backbone.

Semantic Segmentation:

• When combined with a Mask2Former decoder, the frozen DINOv3 backbone achieves state-of-the-art performance on ADE20k with 63.0 mIoU, matching the best existing models that require full backbone fine-tuning.

Monocular Depth Estimation:

• By simply replacing the DINOv2 backbone with a frozen DINOv3 in the Depth Anything V2 pipeline, the model sets a new state-of-the-art on five real-world depth estimation datasets.

3D Understanding:

• Swapping the DINOv2 backbone with a DINOv3 ViT-L model in the Visual Geometry Grounded Transformer (VGGT) pipeline leads to new state-of-the-art results on tasks like camera pose estimation and multi-view stereo.

Performance on Geospatial Data

DINOv3’s learning recipe proves to be highly effective for specialized domains like Earth observation.

• DINOv3 7B model is pre-trained on SAT-493M, a dataset of 493 millions of 512 × 512 images sampled randomly from Maxar RGB ortho-rectified imagery at 0.6 meter resolution
• A version of DINOv3 trained on satellite imagery sets a new state of the art for canopy height estimation.
• Even the original DINOv3 model, trained on web images, sets new state-of-the-art results on numerous high-resolution segmentation and detection tasks for remote sensing, outperforming specialized models that use more complex data.

Based on its strong performance, I highly recommend leveraging this model for classification and segmentation problems, particularly within domains similar to its training data. The model has demonstrated excellent results on diverse datasets, including industrial and medical imagery.

While it performs well out-of-the-box, its true potential is unlocked through fine-tuning. By adapting the model to your specific dataset and use case, you can achieve even greater accuracy and performance.

Try It Yourself!

• Live Demo: You can test the model live on my Hugging Face Space. Please be patient, as it runs on a CPU and may take a moment to load.
• DINOv3 Features — a Hugging Face Space by sayedM
• sayedmohamedscu/DinoV3-visualization-applications: Dinov3 exploration (use cases & capabilities )

About me

I am a senior Computer Vision Engineer with over 5 years of experience applying deep learning , computer vision techniques and NLP to tackle challenging real-world problems using state of the art vision models. Skilled in machine learning frameworks like PyTorch and TensorFlow. Developed and deployed multiple computer vision and NLP systems including auto-ML , auto-annotation, segmentation ,detection ,classification with 3D &2D medical imaging, license plate recognition,OCR ,facial recognition, object detection ,tracking , VLM , RAG and finetuning LLM . Passionate about using CV and NLP to build next-generation technology. Proven ability to deliver high-quality results in fast-paced environments. seeking to leverage expertise in deep learning and computer vision to improve products and processes as a Machine Learning Engineer.

feel free to connect LinkedIn