The Complete Guide to Open-Source Medical Imaging Datasets for AI Segmentation

A researcher’s verified reference for deep learning in radiology — covering 15 benchmark datasets with links, citations, and real academic studies

If you’ve ever spent hours hunting down the right dataset for organ segmentation, digging through Grand Challenge pages and broken links — this guide is for you. I’ve compiled, verified, and expanded the most commonly referenced open-source medical imaging datasets used in clinical AI research, with academic studies that actually used them.

Created by AI

Why This Matters

Medical image segmentation is one of the most active frontiers in clinical AI. Whether you’re building models for surgical planning, radiation therapy, or cancer screening — your dataset choice defines your ceiling. Yet navigating this landscape is surprisingly difficult: dataset names get abbreviated inconsistently, some are buried in challenge archives, and published papers rarely agree on vaka counts.

This guide covers 15 datasets across the following list:

· TotalSegmentator · AMOS · CHAOS · FLARE 22 · HanSeg · HVSMR-2.0

· KiTS · LiTS 2017 · PKG · RAOS · STACOM 2013 · STACOM 2018 · MSD

· BTCV Abdomen · A Comprehensive Dataset

Each entry includes: modality, target structures, case count, official link, primary reference, and real academic papers that used it.

Part 1: Whole-Body & Multi-Organ Datasets

These are the modern benchmarks covering large anatomical regions with many labeled structures.

1. TotalSegmentator

Modality: CT

Target Structures: 104 anatomical structures — 27 organs, 59 bones, 10 muscles, 8 vessels

Case Count: 1,204 CT scans (multi-center, real-world clinical diversity)

Access: GitHub · Zenodo

TotalSegmentator is arguably the most comprehensive publicly available CT segmentation dataset. Scans were randomly sampled from routine clinical studies (2012, 2016, 2020), covering diverse ages, pathologies, scanner types, and acquisition phases. The nnU-Net model trained on this data is available as a plug-and-play tool — a rare example of a dataset that ships with its own production-grade model.

Primary Reference: Wasserthal, J., et al. (2023). *TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images.* Radiology: Artificial Intelligence, 5(5), e230024.

Academic Studies Using This Dataset:

• Tsanda et al. (2024). *On TotalSegmentator’s performance on low-dose CT images.* SPIE Medical Imaging.
• Wasserthal et al. (2024). *TotalSegmentator MRI: Robust Sequence-independent Segmentation.* arXiv:2405.19492.
• Various (2025). *Automatic Segmentation of Cardiovascular Structures — An Update of the TotalSegmentator.* European Journal of Radiology.

2. AMOS (Abdominal Multi-Organ Segmentation)

Modality: CT and MRI

Target Structures: 15 abdominal organs — spleen, kidneys (×2), gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, pancreas, adrenal glands (×2), duodenum, bladder, prostate/uterus

Case Count: 500 CT + 100 MRI (multi-center, multi-vendor, multi-phase)

Access: Grand Challenge

AMOS is one of the most rigorous multi-modal benchmarks available. Its key differentiator is genuine heterogeneity: eight different CT/MRI scanners from two medical centers. It supports two tasks — CT-only segmentation and cross-modality CT+MRI segmentation. Particularly valuable for testing out-of-distribution generalization.

Primary Reference: Ji, Y., et al. (2022). *AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation.* NeurIPS 2022.

Academic Studies Using This Dataset:

• Ji et al. (2022). *AMOS NeurIPS Paper (primary).*
• Wasserthal et al. (2024). *TotalSegmentator MRI:Robust Sequence-independent Segmentation of Multiple Anatomic Structures in MRI — uses AMOS for multi-modal validation.*

3. CHAOS (Combined Healthy Abdominal Organ Segmentation)

Modality: CT and MRI (T1-DUAL, T2-SPIR sequences)

Target Structures: Liver, kidneys (×2), spleen — healthy subjects only

Case Count: 40 CT + 120 MRI (20 patients, each with CT + 3 MRI modalities)

Access: Grand Challenge · Zenodo

CHAOS is specifically designed for cross-modality evaluation — the same patients are imaged with both CT and multiple MRI sequences, enabling true paired analysis. It remains one of the few datasets explicitly designed to test modality-bridging algorithms.

Primary Reference: Kavur, A. E., et al. (2021). *CHAOS challenge — combined (CT-MR) healthy abdominal organ segmentation.* Medical Image Analysis, 69, 101950.

Academic Studies Using This Dataset:

• Chen et al. (2021). *TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation.* arXiv:2102.04306.

4. FLARE 22 (Fast and Low-Resource Abdominal Organ Segmentation)

Modality: CT (contrast and non-contrast)

Target Structures: 13 abdominal organs — liver, spleen, pancreas, kidneys (×2), stomach, gallbladder, esophagus, aorta, inferior vena cava, adrenal glands (×2)

Case Count: 50 labeled + 2,000 partially labeled + 1,800 unlabeled CT images

Access: Grand Challenge

FLARE 22 stands out for its emphasis on efficiency. The challenge specifically targets algorithms that are fast, lightweight, and suitable for low-resource deployment — making it ideal for research on semi-supervised learning, knowledge distillation, and model compression. Its large pool of unlabeled data makes it uniquely suited for self-supervised pretraining.

Primary Reference: Ma, J., et al. (2024). *Unleashing the Strengths of Unlabeled Data in Deep Learning-Assisted Pan-Cancer Abdominal Organ Quantification: The FLARE22 Challenge.* The Lancet Digital Health.

Academic Studies Using This Dataset:

• Ma. et al.(2023). *Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge . * arXiv:2308.05862

5. BTCV Abdomen (Beyond the Cranial Vault)

Modality: CT (abdominal, portal venous phase)

Target Structures: 13 abdominal organs — spleen, kidneys (×2), gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, portal/splenic vein, pancreas, adrenal glands (×2)

Case Count: 50 CT scans (30 train + 20 test) — Vanderbilt University

Access: Synapse

BTCV is arguably the most widely used benchmark for abdominal organ segmentation in the deep learning era. Its modest size (50 scans) makes it accessible, while its standardized evaluation makes results directly comparable across the literature. Nearly every new segmentation architecture (TransUNet, Swin-UNet, nnU-Net) reports results on BTCV.

Primary Reference: Landman, B., et al. (2015). MICCAI Multi-Atlas Labeling Beyond the Cranial Vault — Workshop and Challenge. MICCAI 2015.

Academic Studies Using This Dataset:

• Fang, X., & Yan, P. (2020). *Multi-Organ Segmentation Over Partially Labeled Datasets With Multi-Scale Feature Abstraction. IEEE transactions on medical imaging, 39*(11), 3619–3629.

6. MSD (Medical Segmentation Decathlon)

Modality: CT and MRI (task-dependent)

Target Structures: 10 tasks — brain tumor (MRI), heart (MRI), liver+tumor (CT), lung (CT), pancreas+tumor (CT), hepatic vessels (CT), hippocampus (MRI), prostate (MRI), colon (CT), cardiac structures (CT)

Case Count: 2,633 total 3D images across 10 tasks

Access: Official Site

The MSD’s philosophy is elegant: instead of single-task specialization, it evaluates whether a single model can handle all 10 tasks. It introduced the “decathlon” concept to medical AI — rewarding generalization over task-specific tuning. It remains the definitive multi-task benchmark and is widely used for nnU-Net validation.

Primary Reference: Antonelli, M., et al. (2022). *The Medical Segmentation Decathlon. Nature Communications*, 13, 4128.

Academic Studies Using This Dataset:

• Isensee et al. (2021). *nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.* Nature Methods.

Part 2: Organ-Specific Datasets

These datasets focus on a single organ or disease — ideal for specialized clinical AI development.

7. KiTS (Kidney and Kidney Tumor Segmentation) — KiTS19 / KiTS21 / KiTS23

Modality: CT (contrast-enhanced, portal venous phase)

Target Structures: Kidney parenchyma + kidney tumor + kidney cyst (3 classes from KiTS23 onward)

Case Count: KiTS19: 300 · KiTS21: 300+ · KiTS23: 500+

Access: GitHub

KiTS has evolved through three iterations, progressively adding cyst annotations and expanding case counts. It remains the gold standard for kidney tumor segmentation evaluation. The challenge’s structured leaderboard has driven meaningful performance improvements — the winning KiTS19 submission achieved 0.974 kidney Dice and 0.851 tumor Dice.

Primary Reference: Sathianathen, N. J., Heller, et al. (2022). *Automatic Segmentation of Kidneys and Kidney Tumors: The KiTS19 International Challenge. Frontiers in digital health, 3*, 797607.

Academic Studies Using This Dataset:

• Heller, et al.(2019). *The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes . *arXiv:1904.00445

8. LiTS 2017 (Liver Tumor Segmentation Benchmark)

Modality: CT (portal venous and venous phase)

Target Structures: Liver parenchyma + liver lesions/tumors

Case Count: 131 training + 70 test CT scans (multi-center, 7 clinical sites) Access: Grand Challenge

LiTS is one of the most established organ-specific benchmarks in the field. Its multi-center design (7 institutions) introduces realistic scanner variation, making it a strong test for clinical generalizability. It has been used to benchmark methods from early U-Net variants to modern transformer architectures.

Primary Reference: Bilic, P., et al. (2023). *The Liver Tumor Segmentation Benchmark (LiTS).* Medical Image Analysis, 84, 102680.

Academic Studies Using This Dataset:

• Li et al. (2018).*H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation from CT Volumes.* arXiv:1709.07330.

9. PKG (Polycystic Kidney / Glomeruli — KPIs Challenge)

Modality: MRI (T1/T2-weighted) for kidney volume + Histopathology (PAS-stained WSI) for glomeruli

Target Structures: Polycystic kidneys (total kidney volume measurement) · Glomeruli (in renal biopsy WSI)

Case Count: KPIs 2024: 60+ WSIs with 10,000+ annotated glomeruli · PKD MRI: 400+ cases

Access: KPIs 2024 — arXiv

⚠️ Clarification note: “PKG” does not refer to a single established dataset in the literature. It appears in two distinct contexts: (1) Polycystic Kidney Disease (PKD) MRI databases used to measure total kidney volume, and (2) the Kidney Pathology Image Segmentation (KPIs) Challenge 2024, focused on glomerular segmentation from whole slide images. Both are covered here.

Primary References:

• KPIs 2024: Advancing Glomerular Segmentation from Patch- to Slide-Level. arXiv:2502.07288.
• PKD MRI: *Performance of an AI Multi-observer Deep Neural Network for Fully Automated Segmentation of Polycystic Kidneys.* PMC5537093.

Academic Studies Using This Dataset:

• Various (2023). *Role of Image Acquisition and Patient Phenotype Variations in Automatic Segmentation Generalization (PKD MRI).*

Part 3: Regional & Specialized Datasets

10. HanSeg / HaN-Seg (Head and Neck Organ-at-Risk Segmentation)

Modality: CT and MRI (simultaneously acquired, same patients)

Target Structures: 30 organs-at-risk (OARs) — eyes, lenses, optic nerves, optic chiasm, brainstem, parotid glands, submandibular glands, cochleae, mandible, temporal lobes, inner ear, and more

Case Count: 42 patients (paired CT + MRI, 84 total image series)

Access: Grand Challenge

HaN-Seg is distinguished by its simultaneous CT + MRI acquisition in the same patients — enabling true multimodal OAR segmentation without registration artifacts. It directly supports radiotherapy planning workflows and provides a realistic test bed for approaches that fuse complementary imaging information.

Primary Reference: Podobnik, G., et al. (2023). *HaN-Seg: The head and neck organ-at-risk CT and MR segmentation dataset.* Medical Physics, 50, 1917–1927.

Academic Studies Using This Dataset:

• Podobnik et al. (2023). *Multimodal CT and MR Segmentation of Head and Neck Organs-at-Risk (MICCAI 2023).*

11. HVSMR-2.0 (Whole-Heart Segmentation in Congenital Heart Disease)

Modality: 3D Cardiovascular MRI (CMR)

Target Structures: 8 structures — 4 cardiac chambers (LV, RV, LA, RA) + 4 great vessels (aorta, pulmonary arteries, pulmonary veins, IVC/SVC)

Case Count: 60 CMR scans (20 original HVSMR + 40 new, covering diverse congenital heart defects)

Access: MIT CSAIL

This dataset addresses a critical gap: most cardiac segmentation datasets assume normal anatomy. HVSMR-2.0 specifically targets congenital heart disease — where anatomy deviates significantly from normal, making automatic segmentation substantially harder. Each scan includes detailed diagnosis metadata.

Primary Reference: Pace, D. F., et al. (2024). *HVSMR-2.0: A 3D cardiovascular MR dataset for whole-heart segmentation in congenital heart disease.* Scientific Data, 11, 681.

Academic Studies Using This Dataset:

• PubMed Entry: https://pubmed.ncbi.nlm.nih.gov/38956063/
• Various (2017).* A Combined Random Forests and Active Contour Model Approach for Fully Automatic Segmentation of the Left Atrium in Volumetric MRI.* PMC.

12. STACOM (Left Atrial Segmentation Challenge)

Modality: 3D MRI (Late Gadolinium Enhancement / GE-MRI) and CT

Target Structures: Left atrium (LA) cavity + proximal pulmonary veins

Case Count: 30 MRI + 30 CT scans (King’s College London & Philips collaboration)

Access: Cardiac Atlas Project

STACOM established the benchmark for left atrium segmentation — directly motivated by clinical demand for atrial fibrillation ablation planning. The dataset includes both MRI and CT for the same anatomical target, enabling modality comparison. It has since become a standard reference point for LA segmentation papers.

Primary Reference: Tobon-Gomez, C., et al. (2015). Benchmark for Algorithms Segmenting the Left Atrium From 3D CT and MRI Datasets. IEEE Transactions on Medical Imaging, 34(7), 1460–1473.

Academic Studies Using This Dataset:

• Various (2024). *Left Atrial Segmentation with nnU-Net Using MRI (LASC’13 dataset).* arXiv:2511.04071.
• Various (2017). *CardiacNET: Multi-View CNN for LA and Pulmonary Vein Segmentation.* arXiv:1705.06333.
• Various (2017). *A Combined Random Forests and Active Contour Model Approach for Fully Automatic Segmentation of the Left Atrium in Volumetric MRI.* PMC.

13. STACOM 2018 (Left Atrium Segmentation Challenge — LGE-MRI)

Modality: 3D Late Gadolinium Enhancement MRI (LGE-MRI)

Target Structures: Left atrium (LA) cavity (binary: LA / background)

Case Count: 154 3D LGE-MRI scans (University of Utah + multi-institutional)

Access: arXiv Paper

STACOM 2018 is currently the world’s largest cardiac LGE-MRI dataset for LA segmentation, attracting 27 international teams. The top submission achieved 93.2% Dice — considered near-expert performance.

Primary Reference: Xiong, Z., et al. (2021). *A Global Benchmark of Algorithms for Segmenting Late Gadolinium-Enhanced Cardiac Magnetic Resonance Imaging.* Medical Image Analysis, 67, 101832.

Academic Studies Using This Dataset:

• Vesal et al.(2020). *Fully Automated 3D Cardiac MRI Localisation and Segmentation Using Deep Neural Networks (ACDC + STACOM 2018).* PMC.

ℹ️ Note on STACOM naming: STACOM 2013 focuses on LA + pulmonary vein segmentation using GE-MRI and CT. STACOM 2018 focuses specifically on LA cavity segmentation using LGE-MRI (a different, fibrosis-sensitive sequence). ACDC (Automatic Cardiac Diagnosis Challenge), often associated with STACOM 2017, covers ventricular and myocardial segmentation — a distinct problem.

14. RAOS (Rethinking Abdominal Organ Segmentation)

Modality: CT (+ 9 synthetic MRI contrasts per scan)

Target Structures: 19 abdominal/pelvic organs — liver, spleen, kidneys, gallbladder, esophagus, stomach, pancreas, aorta, inferior vena cava, adrenal glands, duodenum, bladder, rectum, colon, intestine, prostate, seminal vesicles, femoral heads

Case Count: 413 CT scans + 413×9 synthetic MRI images (annotated by senior oncologist, 10 years experience)

Access: GitHub

RAOS is uniquely designed for robustness evaluation rather than standard performance benchmarking. It groups scans into three clinical categories: (1) pre-treatment/radiotherapy cases, (2) partial organ excision, and (3) complete organ excision. This allows systematic evaluation of “organ hallucination” — where models predict organ presence even after surgical removal. It also includes structures absent from other public datasets: prostate, seminal vesicles, rectum, and colon.

Primary Reference: Luo, X., et al. (2024). Rethinking Abdominal Organ Segmentation (RAOS) in the clinical scenario: A robustness evaluation benchmark with challenging cases. MICCAI 2024.

15. AbdomenCT-1K / WORD

Modality: CT (abdominal, multi-center)

Target Structures: AbdomenCT-1K: 4 organs (liver, kidney, spleen, pancreas) · WORD: 16 organs (full abdomen coverage)

Case Count: AbdomenCT-1K: 1,112 CT scans (12 clinical centers) · WORD: 150 CT scans

Access: AbdomenCT-1K GitHub · WORD arXiv

ℹ️ Note: AbdomenCT-1K (Ma et al., 2021) and WORD (Luo et al., 2022) — both large-scale, multi-center CT datasets covering the abdominal region.

AbdomenCT-1K is remarkable for its scale across institutions — 1,112 scans from 12 clinical centers — making it ideal for large-scale generalizability studies. WORD extends this by providing dense 16-organ annotations with both pixel-level and scribble-based sparse annotations, uniquely supporting annotation-efficient learning research.

Primary References:

• Ma, J., et al. (2021). *AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem?* IEEE TPAMI.
• Luo, X., et al. (2022). *WORD: A large scale dataset for abdominal organ segmentation from CT image.* Medical Image Analysis.

Quick Reference Summary Table

Recommended Starting Points by Task

Surgical planning (multi-organ): TotalSegmentator → AMOS → BTCV

Radiotherapy OAR delineation: HaN-Seg → TotalSegmentator → RAOS

Kidney cancer: KiTS23 → AbdomenCT-1K (for transfer pretraining)

Liver cancer: LiTS 2017 → MSD Task03

Cardiac (general): STACOM 2018 → ACDC (for cine MRI)

Cardiac (congenital): HVSMR-2.0

Efficient model development: FLARE 22 → AbdomenCT-1K

Cross-modality research: CHAOS → AMOS

All dataset access links and paper URLs verified as of May 2025. Dataset versions and challenge availability may change — always check the primary platform for the most current access instructions.

If you found this useful, clap and follow for more clinical AI research breakdowns. Have a dataset I missed? Drop it in the comments.