Diving Deep into Complex Datasets: Segmentation, Detection, and Recognition

Phase — 3

At this stage, I truly began to engage with the complexity and diversity of real-world computer vision datasets. While earlier phases were about learning how to load and preprocess datasets, this phase pushed me to tackle datasets that were multi-dimensional, hierarchical, or required pixel-level annotations. It gave me an opportunity to think critically about API design, efficiency, and flexibility for end users.

Oxford-IIIT Pet — First Segmentation Challenge

The Oxford-IIIT Pet dataset marked my first experience creating a segmentation dataset. Unlike classification datasets, it required loading a separate mask variable as a torch boolean tensor, which then had to be compatible with the predefined visualisation function draw_segmentation_masks(). This was a significant technical challenge because I had to ensure that the masks aligned perfectly with the images and that downstream functions could use them seamlessly.

Another innovation in this dataset was full inheritance- creating a parent dataset loader called oxfordiiitpet_segmentation_dataset() and two child loaders: oxfordiiitpet_dataset() for classification into 37 breeds, and oxfordiiitpet_binary_dataset() for cat versus dog classification. Both child loaders fully inherited all functions from the parent, making this one of the most efficient and reusable datasets I had implemented. This idea of using a single dataset structure for multiple purposes was highlighted and appreciated by my mentor, Christophe Regouby. Also, this helped me learn to update the _pkgdown.yml file to reflect segmentation datasets, which was essential for documentation.

📌 Pull Request: #162 — Oxford-IIIT Pet Segmentation Dataset

LFW — Face Recognition and Verification

The LFW dataset was a particularly tricky challenge — one which I had attempted earlier and abandoned due to its complexity. Revisiting it was a personal milestone in overcoming hesitation. The dataset required implementing both multi-class classification for individual identities and face verification for image pairs.

One challenge was downloading the dataset. The original links required a Kaggle sign-in and could not be directly downloaded via R code. To get around this, I uploaded the dataset to my Hugging Face repository and used those links for R-based downloading. For face verification, the images had to be paired correctly to classify them as the same or different person. There were no prepackaged instructions, but some digging unearthed a text file referenced in a research paper and which I used to generate the verification pairs.

This dataset was a lesson in resilience, creative problem-solving, and the importance of reproducible pipelines for datasets that cannot be accessed in standard ways.

📌 Pull Request: #203 — LFW Datasets for Image Classification and Pairs

Pascal VOC — Detection and Segmentation

The Pascal VOC dataset introduced my first object detection task, as well as more advanced segmentation challenges. It was humongous in terms of complexity, supporting six different yearly editions and offering both bounding boxes for detection and per-pixel masks for segmentation. Implementing this dataset required careful handling of folder structures and annotations, particularly because the masks were 3 x H x W tensors with 21 semantic classes. Each class was represented by a unique RGB combination, requiring me to map each pixel to the correct class.

For the detection component, I initially parsed XML files manually for bounding boxes, but my mentor Christophe Regouby suggested a more efficient approach, which greatly simplified implementation. Segmentation was a bigger challenge, as I had to account for 21 different classes in a single image and create masks that worked seamlessly with visualisation and downstream tasks. This PR helped me consolidate my understanding of both detection and segmentation datasets and reinforced best practices in loader design.

📌 Pull Request: #209 — Pascal VOC Datasets for Detection and Segmentation

VGGFace2 — Large-Scale Face Recognition

Finally, VGGFace2 was by far the largest dataset I worked with, totalling around 38 GB. Each identity had multiple images with variations in pose, age, illumination, ethnicity, and profession. I implemented a loader that returned the image array and integer label, along with a class list mapping labels to names and gender.

Further extended the functionality to include detailed attributes like hair colour, hair length, facial hair, eyeglasses, sunglasses, hats, and whether the mouth was open. This was particularly challenging because most images lacked this metadata, resulting in missing values, but the structure was designed to support future attribute-based extensions. Working with VGGFace2 taught me to optimise memory usage, batch processing, and efficient mapping for large-scale datasets. This was a huge step up from Phase 1, where 0.5 GB datasets seemed large, and now handling 38 GB required careful thought about system efficiency and caching.

📌 Pull Request: #238 — VGGFace2 Dataset

Lessons from Phase 3

Oxford-IIIT Pet taught me about segmentation and reusable dataset design with inheritance. LFW challenged me to think creatively about dataset access and verification pair generation. Pascal VOC introduced detection and complex per-pixel segmentation mapping. VGGFace2 pushed my understanding of large-scale face recognition datasets and efficient attribute handling.

This phase cemented my understanding that building dataset loaders is not just about loading images, but designing robust, reusable, efficient, and well-documented APIs. It strengthened my confidence in handling complex datasets and prepared me for the next phase, where collaboration, standardisation, and model integration would take centre stage.