Artwork Similarity Search with DINO Embeddings

Shambhavi Adhikari

Introduction

Online art collections contain millions of artworks, yet most systems rely on text-based search, requiring users to know labels like “Impressionism” or “Cubism.” This creates a fundamental mismatch: art appreciation is inherently visual, while discovery remains verbal.

As stated in our proposal: “People just know what they like, even when they can’t say what it is.”

We hypothesized that DINOv2, trained via self-supervised learning on 142 million images, would capture the stylistic cues that define artistic movements, brushstrokes, composition, texture, and color palette. Our goal was to enable a “show me more like this” search based purely on visual similarity.

Our initial goal was to evaluate whether self-supervised DINOv2 embeddings outperform a supervised CNN baseline for artwork similarity retrieval. We chose ResNet-50 pretrained on ImageNet-1K via supervised classification as the baseline, as it represents a standard object-centric visual feature extractor widely used in retrieval tasks.

However, early experiments revealed surprising near-parity between DINOv2 and ImageNet-pretrained ResNet-50 achieved nearly identical retrieval performance (within 1% mAP), prompting us to investigate why self-supervised and supervised pretraining produced such similar results. This led us to systematically isolate the contributions of architecture choice, multi-crop augmentation, dimensionality reduction, and ensemble methods. While our core question remains, “Can DINOv2 enable visual artwork search?” we found that answering it rigorously required understanding which performance gains came from the model versus our preprocessing pipeline.

DINO Flowchart

To evaluate this, we compare it against ResNet-50 (supervised on ImageNet), which serves as our baseline. Our analysis revealed near-identical performance, leading us to investigate a secondary question: Do image embeddings prioritize style-based similarity (brushstrokes, composition, texture) or content similarity (subjects such as landscapes, portraits)? We systematically compare self-supervised vs supervised pretraining to understand what features models learn for artistic retrieval.

Methodology

Dataset Construction

Source: WikiArt dataset (81,444 images, 27 styles) Subset Selection: 12,000 images across 8 art movements Evaluation Protocol: From the 12,000 images, we randomly selected 2,000 as query images and used the remaining 10,000 as the gallery for retrieval. This ensures queries and the gallery are disjoint Content Annotation: We automatically labeled artworks by parsing filenames into five categories: Portrait, Landscape, Abstract, Still Life, and Cityscape. These annotations enabled explicit analysis of content-based vs style-based retrieval behavior.

Models Compared

**DINOv2 (Base and Large):

• **Self-supervised pretraining on 142M diverse web images
• Learns by comparing augmented views of the same image (no labels)
• Base: 768D features, Large: 1024D features

**ResNet-50 (Baseline):

• **Pretrained on ImageNet-1K (1.28M images, 1000 object categories) via supervised classification
• Optimized to discriminate objects (e.g., cats vs dogs), not artistic style
• Extracted 2048D features from the penultimate layer (before final classifier) ResNet-50 serves as a strong supervised baseline representing object-centric visual learning.

Technical Enhancements

We tested three enhancements beyond simple single-crop extraction:

1. Multi-Crop Augmentation: Eight crops per image — 1 center crop, 4 corner crops (top-left, top-right, bottom-left, bottom-right), and 3 random augmented views (random crop scale 0.8–1.0, random horizontal flip p=0.5, color jitter ±0.1). All crops resized to 224×224 with ImageNet normalization, then averaged via arithmetic mean in feature space. This augmentation stabilizes representations against framing, scale, and lighting variability common in artworks.
2. PCA Dimensionality Reduction: To reduce memory usage and computational cost, we applied PCA to compress features to 512 dimensions:

We chose 512 dimensions for three reasons. Firstly, it is close to DINOv2-Base’s natural output size (768D), which allows us to compare models fairly, and it reduces computational cost while keeping enough information. We expected PCA to work well since it typically removes noise by discarding low-variance dimensions. However, our assumption was wrong, and the results show that subtle artistic features like brushstroke texture or color harmony often hide in these low-variance dimensions. PCA threw away exactly what we needed for style matching.

3. Ensemble: Combined DINOv2-Base and Large features via concatenation (768D + 1024D = 1792D), then applied PCA to 512D

Computational cost: Multi-crop augmentation requires eight forward passes per image. When combined with the DINOv2 ensemble (Base + Large), each image is processed by two models across eight crops, resulting in 16 total forward passes per image. Thus, the ensemble configuration incurs approximately a 16× computational cost relative to a single-crop, single-model ResNet baseline.

We initially tested reciprocal re-ranking but encountered implementation inconsistencies that prevented fair comparison with other methods, so we excluded it from this analysis. Future work should evaluate re-ranking with properly standardized preprocessing.

Results and Analysis

To isolate whether improvements came from DINOv2’s architecture or our enhancements, we compared both models with identical preprocessing (8 crops, no PCA, no ensemble).

Architectural Comparison (8-crop, no PCA, no ensemble)

We use each model’s native output dimensionality (no compression) to ensure fair comparison of learned representations. Key Finding: When preprocessing is strictly controlled, DINOv2-Base and ImageNet-pretrained ResNet-50 achieve nearly identical performance: mAP: 0.4016 vs 0.4045 Acc@10: 96.3% vs 94.7% DINOv2 achieves comparable performance to ResNet while using 2.7× fewer dimensions (768D vs 2048D). This shows that self-supervised pretraining can produce more compact representations without sacrificing quality. However, it doesn’t translate to better retrieval performance as the two models perform nearly identically.

Same-Dimension Comparison: To address concerns about comparing different dimensions, we also tested both models at 512D using PCA:

When forced to identical dimensions via PCA, both models degrade significantly and perform nearly identically (0.1599 vs 0.1629 MAP, <2% difference). This confirms that:

1. Native dimensionality comparison is more meaningful than forced compression
2. PCA’s variance-based criterion is misaligned with stylistic semantics
3. Low-variance dimensions can encode critical artistic cues

Ensemble Effect

DINO Similarity Search

ResNET 50 Similarity Search

DINOv2 and ResNet-50 similarity search results showing query image and top-10 retrievals. Top panel: DINOv2 Similarity Search Results with landscape sketch queries. Bottom panel: ResNet-50 Similarity Search Results for the same queries. Performance metrics table shows DINOv2 Ensemble achieves MAP=0.5750 vs ResNet-50 MAP=0.5239 (+9.8% improvement), Accuracy@1 of 62.60% vs 57.45% (+5.15pp), and Accuracy@10 of 96.90% vs 95.40% (+1.50pp). Both models successfully retrieve visually similar artworks, with DINOv2 maintaining slightly better style coherence across all retrievals.

These ensemble results (MAP=0.58) were obtained during early experiments before we standardized our preprocessing pipeline. Our subsequent controlled ablation study (see Ablation Study section, row 5) reveals that when DINOv2-Base and the ensemble use identical preprocessing and evaluation protocols, the ensemble improves MAP by only +0.0088 (+5.4%), not +9.8%. The large discrepancy highlights that our initial comparison inadvertently mixed preprocessing differences with architectural effects. This discovery reinforced our core finding: preprocessing choices can dominate model architecture choices in determining retrieval performance.

**Key observations from retrieval examples:

• **Under identical preprocessing, DINOv2-Base and ImageNet-pretrained ResNet-50 retrieve similarly styled artworks, with DINOv2-Base showing slightly stronger visual coherence in the top-10 results.
• ResNet-50 occasionally introduces greater subject variation within the same style category.
• These qualitative trends align with quantitative results showing comparable overall performance.
• The DINOv2 ensemble (Base + Large) further improves Artist Coherence (75.80% vs. 68.80%), indicating complementary multi-scale features.

Visualization

To understand why DINOv2 and ResNet-50 achieve similar performance despite different pretraining objectives, we visualized the learned embedding spaces using t-SNE dimensionality reduction.

t-SNE Embedding Space

t-SNE visualization comparing DINOv2 ensemble embeddings (left) and ResNet-50 embeddings (right), showing clearer style clustering with DINOv2

Accuracy

Top-K retrieval accuracy for DINOv2 ensemble vs. ResNet-50, showing consistently higher performance from DINOv2 across all K values

Top row: t-SNE projections of 12,000 artwork embeddings colored by art movement. The left panel shows DINOv2-Base embeddings with distinct clusters for Impressionism, Cubism, Abstract Expressionism, and other movements. The right panel shows ResNet-50 embeddings with a similar clustering structure. Bottom panel: Top-K accuracy curves comparing both models across K=1 to K=50, showing convergent performance with both models exceeding 95% Acc@10. The structural similarity of embedding spaces explains the near-identical retrieval performance.

**Key observations:

• **Both embedding spaces show a clear separation of art movements with a similar global structure
• DINOv2 clusters appear slightly more compact, particularly for Impressionism (orange) and Cubism (green)
• ResNet-50 exhibits comparable clustering despite being trained for object classification
• Accuracy curves converge at K≥10, confirming fair comparison results
• Visual and quantitative evidence align: self-supervised and supervised pretraining learn similar feature spaces

The clustering patterns reveal that both supervised (ResNet) and self-supervised (DINOv2) pretraining learn to separate art movements effectively. DINOv2’s slightly more compact clusters suggest marginally better style discrimination — consistent with its +1.5pp Acc@10 advantage. This geometric similarity in learned representations explains why performance metrics are nearly identical despite fundamentally different training objectives (object classification vs. instance discrimination).

Ablation Study: Effect of Enhancements

We isolate the source of performance gains through controlled ablations. When both models use identical preprocessing (8 crops, no PCA, no ensemble), DINOv2-Base and ImageNet-pretrained ResNet-50 achieve nearly identical MAP (~0.40). This shows that the primary gains come from preprocessing rather than self-supervised pretraining. The DINOv2 ensemble provides an additional but modest improvement (+0.0088 MAP), indicating complementary scale features rather than a fundamental representational advantage.” We progressively added enhancements to understand their individual contributions:

Ablation Study — Progressive Enhancement Analysis

Analysis

Multi-crop augmentation (row 2): Improved ResNet MAP by +0.0264 (+13.2%), confirming that averaging multiple views stabilizes embeddings. This enhancement is model-agnostic and beneficial for both architectures.

PCA compression (row 3): Unexpectedly decreased ResNet performance by -0.0666 (-29.4%). We reduced dimensions from 2048D to 512D to balance computational efficiency with discriminative power, preserving ~94% variance. However, PCA’s variance-based criterion doesn’t align with semantic similarity for art — high-variance dimensions may encode irrelevant background texture, while low-variance dimensions may encode critical stylistic cues like brushstroke patterns.

The 512D target was chosen to roughly match DINOv2-Base’s native dimensionality (768D) and provide consistent compression ratios. However, this proved too aggressive for ResNet’s 2048D space (4:1 compression) compared to DINOv2’s 768D space (1.5:1 compression). Future work should tune PCA dimensions per-model: 1024D or higher for ResNet, 512D for DINOv2.

PCA Dimensionality Sensitivity Analysis

Note: PCA hurts both models, but ResNet suffers more severely due to a higher compression ratio (4:1 vs 1.5:1). Despite preserving >94% variance, task-relevant features may reside in low-variance dimensions that PCA discards.

DINOv2 vs ResNet with PCA (row 4): Both models suffered from dimensionality reduction. The fair comparison (Table 1) shows they perform equally without PCA, confirming the architectural choice is secondary to preprocessing decisions.

Ensemble effect (row 5): Adding DINOv2-Large to Base improved MAP by +0.0088 (+5.4%), showing that different model scales capture complementary features (Base: local details, Large: global composition).

Content and Style Analysis

We analyzed 2,000 query artworks and their top-10 retrievals (20,000 comparisons) DINOv2: 17.0% style-only, 25.2% content-only, 34.5% both, 23.2% neither ResNet-50: 17.5% style-only, 23.1% content-only, 28.8% both, 30.6% neither Key Takeaway: Both models favored content over style, with DINOv2 showing an even stronger content bias. Style is encoded, but semantic content (such as portraits and landscapes) dominates retrieval.

Content vs Style Retrieval Analysis

For Example:

DINOv2 retrieves content-similar still-life images more strongly than stylistically matching Cubist artworks, demonstrating clear content dominance.

Both models prioritize content over style, with DINOv2 exhibiting an even stronger content bias (25.2% vs. 23.1%). Both architectures learn to recognize semantic subjects (portraits, landscapes) more strongly than artistic techniques (brushstrokes, color palettes).

Why This Happens:

DINO’s pretraining on natural images emphasizes recognizable objects (faces, buildings, natural scenes). Its patch-level attention mechanism appears to focus on identifiable content rather than global stylistic patterns. Practically, this makes both models excellent for subject-based search (“find more portraits”) but less effective for pure style matching (“find more Impressionist paintings regardless of subject”).

The Figure provides compelling visual evidence: when querying with religious figures or still life compositions, both DINOv2 and ResNet-50 consistently retrieve images with matching subjects across diverse artistic movements. A Renaissance portrait query returns modern portraits, Impressionist portraits, and Baroque portraits — all prioritizing the human subject over the artistic style. This pattern persisted across our 2,000-query test set, revealing a fundamental limitation of general-purpose vision models for style-based art retrieval.

Takeaway

Our hypothesis that DINOv2 would clearly outperform ResNet-50 for artwork similarity is only partially supported. While DINOv2 achieves a modest gain in Acc@10 and matches an ImageNet-pretrained ResNet-50 using 2.7× fewer dimensions, overall mAP remains nearly identical, and both models exhibit a strong bias toward semantic content rather than the expected stylistic sensitivity.

Controlled ablation studies show that preprocessing choices dominate performance: when augmentation and normalization are standardized, DINOv2-Base and ResNet-50 achieve nearly identical mAP (0.4016 vs. 0.4045), indicating that supervised versus self-supervised pretraining is less influential than commonly assumed. Multi-crop augmentation is the only modification that consistently improves both models (+13% mAP), producing embeddings robust to viewpoint, lighting, and framing variations. In contrast, aggressive PCA significantly degrades retrieval quality, even when preserving over 94% variance, revealing that low-variance dimensions can encode task-relevant stylistic information. Ensembling DINOv2-Base and Large yields modest but consistent gains, suggesting complementary multi-scale features rather than a fundamental architectural advantage.

Both models prioritize what is depicted over how it is depicted, highlighting a core limitation of general-purpose vision models for pure style-based retrieval. t-SNE visualizations further confirm that DINOv2 and ResNet-50 learn similarly structured embedding spaces with comparable art-movement clusters, explaining their near-identical retrieval performance. Overall, self-supervised DINOv2 matches supervised ResNet-50 with more compact embeddings, but learning style-focused representations will likely require style-aware training objectives and more carefully designed evaluation pipelines.

Conclusion

Self-supervised DINOv2 embeddings do not substantially outperform supervised ImageNet-pretrained ResNet-50 for artwork similarity search. Both models achieve similar performance, both prioritize content over style, and both are strongly affected by preprocessing choices. The primary lesson is not architectural superiority, but the importance of controlled comparisons, careful dimensionality reduction, and understanding the biases of general-purpose vision models when applied to artistic domains.

The goal of this work was not to identify the strongest off-the-shelf model, but to test whether self-supervised pretraining fundamentally changes what image embeddings prioritize in artistic retrieval. DINOv2 represents a state-of-the-art self-supervised alternative to ImageNet-trained CNNs. Showing that both models converge to similar embedding geometry and exhibit the same content bias is a key result, not a failure. DINOv2 was useful because it showed that the limitation is not supervision, but the training data and objective. That insight narrows where future work should focus, on style-aware objectives rather than model scale or pretraining paradigm.

Future works

1. Style-Focused Pretraining: Train DINOv2 specifically on art images to test whether pretraining data determines content vs style priorities 2. Gram Matrix Features: Explore neural style transfer features that explicitly encode texture/brushstroke patterns independent of content 3. Hybrid Retrieval: Combine content-based and style-based rankings with user-adjustable weighting (e.g., “find portraits in Impressionist style”) 4. Computational Optimization: Implement batch-wise similarity computation to enable full-dataset evaluation without memory constraints 5. Cross-Domain Generalization: Test on sculpture, architecture, and non-Western art to verify that findings generalize beyond WikiArt paintings 6. Per-Model PCA Tuning: Systematically evaluate optimal dimensionality reduction for each architecture (e.g., 1024D for ResNet, 512D for DINOv2)

References

Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision transformers. Advances in Neural Information Processing Systems, 34, 21830–21846. https://arxiv.org/abs/2104.14294 Oquab, M., Caron, M., Touvron, H., Bojanowski, P., Joulin, A., & Misra, I. (2023). DINOv2: Learning robust visual features without supervision. https://arxiv.org/abs/2304.07193 Saleh, B., & Elgammal, A. (2015). Large-scale classification of fine-art paintings: Learning the right metric on the right feature. International Journal for Digital Art History, 1, 1–15. https://arxiv.org/abs/1505.00855 WikiArt Dataset. (n.d.). WikiArt dataset for research. GitHub. https://github.com/cs-chan/ArtGAN/tree/master/WikiArt%20Dataset Archive.org. (n.d.). Sample WikiArt images. https://archive.org/details/wikiart-stylegan2-conditional-model PyTorch. (n.d.). An open source machine learning framework. https://pytorch.org Hugging Face. (n.d.). DINOv2 base model. https://huggingface.co/facebook/dinov2-base

Code: *https://github.com/Shambhaviadhikari/Artwork-Similarity-Search-with-DINO-Embeddings/tree/main*