Does Pathology-Specific Pretraining Beat ImageNet? Benchmarking Foundation-Model and CNN Features for Colorectal Cancer Histology

Pitting a pathology foundation model (Phikon-v2) against ImageNet-pretrained CNNs for nine-class tissue classification on H&E-stained colorectal cancer histopathology

H&E-stained cross-section of mouse colon, from my research days in the lab.

Abstract

Automated classification of tissue types in hematoxylin-and-eosin (H&E) stained histopathology images is a foundational task in computational pathology. The field has recently shifted from transfer learning with ImageNet-pretrained convolutional neural networks (CNNs) toward domain-specific “foundation models” pretrained on large quantities of pathology images. This paper asks a practical question: how much does pathology-specific pretraining improve tissue classification over generic ImageNet features when both are used as frozen feature extractors with a lightweight trained head? Using the NCT-CRC-HE colorectal cancer dataset and a held-out test cohort of different patients (CRC-VAL-HE-7K), I compare a pathology foundation model (Phikon-v2) against three ImageNet-pretrained CNN backbones (ResNet50, EfficientNet-B0, MobileNetV3) and a fine-tuned ResNet50. Phikon-v2 outperforms every ImageNet baseline, reaching a macro F1 of 0.935 on the held-out cross-patient test set, compared with 0.897 for the best CNN (a fine-tuned ResNet50). Under a controlled linear-probe protocol the gap is +9.4 accuracy points and is statistically significant (95% CI [+7.9, +10.9]). The foundation model is also considerably more label-efficient: with 10% of the training labels it reaches 0.934 macro F1, which exceeds the ImageNet CNN trained on the full label set. All code is available on GitHub.

1. Introduction

Hematoxylin-and-eosin (H&E) staining is the most widely used tissue-preparation technique in clinical pathology, and virtually every cancer patient has H&E-stained tissue slides on record. These slides contain rich morphological information that pathologists use for diagnosis and grading, but manual assessment is time-consuming and subject to inter-observer variability. Automated tissue-type classification, the task of labeling image regions as tumor, stroma, lymphocytes, and so on, is a building block for computational pathology systems that assist diagnosis, quantify the tumor microenvironment, and extract prognostic biomarkers.

Deep convolutional neural networks (CNNs) are the standard approach to this problem, and a large body of work has shown that they attain high accuracy on histology classification benchmarks. However, training CNNs from scratch requires large labeled datasets and substantial compute. Transfer learning, in which a network pretrained on a large natural-image dataset such as ImageNet is adapted to the histology task, is an attractive alternative that often matches or exceeds from-scratch training at a fraction of the cost. A practical question for practitioners is how far one can get with the cheapest form of transfer learning, a frozen pretrained backbone with only a small trained classifier head, and how much additional performance fine-tuning buys.

This paper addresses that question with a controlled comparison. Using the NCT-CRC-HE colorectal cancer histology dataset, I benchmark three ImageNet-pretrained CNN backbones as frozen feature extractors, a fine-tuned ResNet50, and, as the central comparison, a pathology-specific foundation model (Phikon-v2) used as a frozen feature extractor. All models are evaluated on a nine-class tissue-classification task using a held-out cohort of different patients. The comparison holds the dataset, preprocessing, and evaluation protocol fixed, which isolates the effect of the feature representation itself and quantifies how much pathology-specific pretraining buys over generic ImageNet features.

2. Data and Methods

2.1 Dataset

The NCT-CRC-HE dataset (Kather et al., 2018) consists of 224×224-pixel patches extracted from H&E-stained colorectal cancer and normal tissue slides, labeled into nine tissue classes: adipose (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), normal colon mucosa (NORM), cancer-associated stroma (STR), and colorectal adenocarcinoma epithelium (TUM). The full training set comprises 100,000 patches from 86 patients; a separate cohort, CRC-VAL-HE-7K, provides 7,180 patches from 50 different patients and is commonly used as a held-out test set.

For tractable training, I use a class-balanced subset of 7,200 training patches (800 per class) and evaluate on 2,700 test patches (300 per class). The test patches are drawn from CRC-VAL-HE-7K, a cohort of 50 patients with no overlap with the training cohort, so all reported performance reflects cross-patient generalization rather than memorization of patient-specific staining or morphology.

2.2 Models

I evaluate four ImageNet transfer-learning configurations, all initialized from ImageNet-pretrained weights:

• ResNet50 (frozen): backbone frozen; only a new linear classification head is trained.
• EfficientNet-B0 (frozen): as above, with an EfficientNet-B0 backbone.
• MobileNetV3-Large (frozen): as above, with a MobileNetV3-Large backbone.
• ResNet50 (fine-tuned): the frozen-head model above, followed by unfreezing the final residual block and fine-tuning at a low learning rate.

Against these, I evaluate a domain-specific pathology foundation model:

• Phikon-v2 (frozen): Phikon-v2 (Filiot et al., 2024) is a ViT-L vision transformer pretrained with DINOv2 on more than 450 million histology tiles drawn from over 100 publicly available cohorts spanning more than 30 cancer sites. It is used here as a frozen feature extractor with the same kind of trained linear head as the CNNs. This makes the comparison a direct test of the value of pathology-specific versus natural-image pretraining.

All models use identical input preprocessing (resize to 224×224, ImageNet normalization) and, for the CNNs, light data augmentation (random horizontal and vertical flips) during training. Models were trained with the Adam optimizer using cross-entropy loss. Full hyperparameters are in the repository.

2.3 Evaluation

Models are evaluated on the held-out test set using accuracy, balanced accuracy (the mean of per-class recall), and macro-averaged F1 (the unweighted mean of per-class F1). Per-class F1 scores and a confusion matrix are also reported to characterize class-specific behavior. A fixed random seed (42) was used throughout.

3. Results

3.1 Overall Performance

Table 1 reports test-set performance for all four configurations, ordered by macro F1.

Table 1. Classification performance on the held-out test set

The pathology foundation model leads on every metric. Among the ImageNet models, fine-tuning the final ResNet50 block improves macro F1 from 0.891 to 0.897 over the frozen version, a small but consistent gain, and ResNet50 is the strongest frozen CNN backbone, ahead of MobileNetV3 (0.872) and EfficientNet-B0 (0.832). Phikon-v2, used purely as a frozen feature extractor with no fine-tuning, exceeds even the fine-tuned ResNet50 by nearly four macro-F1 points (0.935 vs. 0.897). Its longer feature-extraction time reflects the fact that a ViT-L is a heavier model than the CNNs.

Accuracy, balanced accuracy, and macro F1 for the four configurations on the held-out test set.

3.2 Per-Class Performance

Per-class F1 scores show where the models differ. Most tissue classes are classified almost perfectly by every model: background, adipose, lymphocytes, and tumor epithelium all exceed 0.98 F1 for the foundation model. The difficulty concentrates in two classes, smooth muscle (MUS) and cancer-associated stroma (STR). These are the hardest classes for every model, but Phikon-v2 handles them substantially better, reaching 0.75 F1 on stroma versus 0.58 for the frozen ResNet50, and 0.995 on normal mucosa versus 0.915. The foundation model’s advantage is therefore not spread evenly across classes. It comes largely from better discrimination of the morphologically ambiguous tissue types, which are also the most clinically consequential to distinguish.

Per-class F1 for each model across the nine tissue types.

3.3 Confusion Structure

The confusion matrix for the best model (Phikon-v2) shows that its residual errors are highly structured and biologically sensible. Almost all mistakes fall along a single axis: smooth muscle and cancer-associated stroma are confused with one another (27% of true smooth-muscle patches are predicted as stroma, and 14% of stroma patches as muscle), with a smaller fraction of stroma confused with tumor (8%). This is expected, because smooth muscle and cancer-associated stroma are both spindle-cell, fibrous tissues that appear similar on H&E staining and are genuinely difficult to separate even for trained pathologists. Every other tissue class is recovered with recall at or above 0.95. The errors therefore reflect real morphological ambiguity rather than arbitrary misclassification.

3.4 Statistical Significance of the Foundation-Model Advantage

To test whether the difference between the pathology foundation model and the best ImageNet CNN is meaningful rather than an artifact of a single training run, I evaluated both as frozen feature extractors with a linear probe across five random seeds, and applied a paired bootstrap test (2,000 resamples) to the per-sample correctness on the held-out test set.

Across the five seeds, Phikon-v2 reaches a mean macro F1 of 0.938 versus 0.844 for ResNet50, with negligible seed-to-seed variance for both. The paired bootstrap test gives an accuracy difference of +9.4 points in favor of Phikon-v2, with a 95% confidence interval of [+7.9, +10.9] that excludes zero. The foundation model’s advantage is therefore statistically significant and substantial.

3.5 Label Efficiency

A central promise of foundation models is that domain-specific pretraining should reduce the amount of labeled data needed for a downstream task. To test this, I train a linear probe on each frozen representation using 10%, 25%, 50%, and 100% of the available training labels, and measure held-out macro F1 at each level.

The label-efficiency curves are the clearest result of the study. Phikon-v2’s performance is essentially flat across label budgets: with 10% of the training labels (720 patches) it achieves 0.934 macro F1, rising only to 0.938 at the full 7,200 patches. The ImageNet ResNet50 improves steadily with more data, from 0.819 at 10% to 0.844 at 100%. The practical consequence is that Phikon-v2 trained on one-tenth of the labels outperforms the ImageNet CNN trained on all of them (0.934 vs. 0.844). In a field where expert annotation is the primary bottleneck, this is a strong argument for pathology-specific pretraining, since it reduces the labeled data required to reach a given level of performance.

Held-out macro F1 versus percentage of training labels used, for the pathology foundation model and the best ImageNet CNN. Tests whether domain-specific pretraining is more label-efficient.

3.6 Why the Representations Differ

To understand why one representation classifies tissue better than another, I projected each model’s frozen embeddings of the test set to two dimensions with UMAP and colored points by their true tissue class. In Phikon-v2’s embedding space, the tissue classes form many small, tightly concentrated groups that are well separated from one another, with little inter-class mixing. In the ResNet50 embedding, several classes remain clumped together in a shared central region; cancer-associated stroma, smooth muscle, and tumor epithelium overlap substantially, which is precisely the region where ResNet50’s per-class F1 is weakest. The qualitative picture matches the quantitative results: the classes that overlap in the ImageNet embedding are the ones the ImageNet classifier confuses, and the foundation model separates them because its pretraining learned features specific to tissue morphology rather than natural-image texture.

UMAP projections of the frozen feature spaces (pathology foundation model vs. ImageNet CNN), colored by tissue class. Cleaner class separation in the embedding explains stronger downstream classification.

4. Discussion

4.1 Interpretation

The results give a clear answer to the paper’s motivating question. For colorectal-cancer tissue classification, pathology-specific pretraining outperforms generic ImageNet features. The advantage is statistically significant (+9.4 accuracy points under a controlled linear probe), concentrated in the hardest and most clinically relevant tissue distinctions, and achieved with a fraction of the labeled data. A frozen Phikon-v2 with a simple linear head beats a fine-tuned ResNet50, which indicates that the gain comes from the quality of the pretrained representation rather than from downstream model capacity or training effort. The practical takeaway is that when a domain foundation model is available, using it as a frozen feature extractor is both cheaper (no fine-tuning) and stronger than adapting a general-purpose CNN, and it is considerably more economical in annotation. The one regime where the models are comparable is smooth muscle, where both hover near 0.73 to 0.77 F1. This suggests that some morphological distinctions remain hard regardless of pretraining and are candidates for higher-resolution inputs or expert-in-the-loop refinement.

4.2 Limitations

This benchmark is scoped to a single dataset and a class-balanced subset for tractable training, and its conclusions should be read with that scope in mind. One methodological caveat specific to the foundation-model comparison is that, for a controlled pipeline, all models including Phikon-v2 received images with the same ImageNet normalization, whereas Phikon-v2 ships its own preprocessing. Using each model’s native preprocessing could shift its absolute performance somewhat, and a stricter comparison would do so. The models evaluated here are also used purely as frozen feature extractors (plus a fine-tuned CNN); fully fine-tuning the foundation model, or comparing against larger gated foundation models such as UNI, Virchow, or Prov-GigaPath, would be a natural and more comprehensive extension. Finally, tissue-type classification on curated patches is an idealized version of the clinical task, which involves whole-slide images with staining variability, artifacts, and ambiguous regions.

4.3 Extensions

Natural extensions include evaluation on the full dataset rather than a subset; fully fine-tuning the foundation model rather than using it as a frozen extractor; and extending the comparison to larger gated foundation models such as UNI (Chen et al., 2024) and Virchow (Vorontsov et al., 2024), which are pretrained on far larger pathology corpora and would be expected to widen any domain-pretraining advantage. The multi-seed, label-efficiency, and embedding analyses reported here provide a template that such extensions can reuse directly.

5. Conclusion

This benchmark compares a pathology-specific foundation model against ImageNet-pretrained CNNs for colorectal-cancer tissue classification, under an identical frozen-feature protocol on a cross-patient test set. Phikon-v2 performs best, reaching 0.935 macro F1 versus 0.897 for the best ImageNet CNN. The advantage is statistically significant, is largest on the hardest tissue classes, and holds with as little as 10% of the training labels. For practitioners choosing between generic and domain-specific pretraining, the evidence here favors the domain foundation model: used as a frozen feature extractor it is more accurate, cheaper to adapt, and more label-efficient than an ImageNet CNN. The code provides a reproducible framework for extending the comparison to additional foundation models and datasets.

All code is available on GitHub.

References

Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., et al. (2024). Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3), 850–862.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, 248–255.

Filiot, A., Jacob, P., Mac Kain, A., & Saillard, C. (2024). Phikon-v2: A large and public feature extractor for biomarker prediction. arXiv preprint arXiv:2409.09173.

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 770–778.

Howard, A., Sandler, M., Chu, G., et al. (2019). Searching for MobileNetV3. IEEE/CVF International Conference on Computer Vision, 1314–1324.

Kather, J. N., Halama, N., & Marx, A. (2018). 100,000 histological images of human colorectal cancer and healthy tissue [Data set]. Zenodo. https://doi.org/10.5281/zenodo.1214456

Kather, J. N., Krisam, J., Charoentong, P., et al. (2019). Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLOS Medicine, 16(1), e1002730.

Paszke, A., Gross, S., Massa, F., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 8024–8035.

Tan, M., & Le, Q. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, 6105–6114.

Vorontsov, E., Bozkurt, A., Casson, A., et al. (2024). A foundation model for clinical-grade computational pathology and rare cancers detection. Nature Medicine, 30(10), 2924–2935.