An introduction to Detectron2

Whats Detectron2?

1. Detectron2 is not just a model; it’s a comprehensive framework.
2. Open source Object Detection and Segmentation Framework developed by facebook AI research.
3. Built on top of Pytorch and provides a unified API for variety of tasks, including, detection, instance segmentation, panoptic segmentation.
4. It includes high quality implementations of SOTA algorithms like Mask RCNN, RetinaNet, DensePose.
5. Installing individual models from the R-CNN family, like Mask R-CNN, it’s freaking nightmare. However, Detectron2 simplifies this process by providing a unified solution.
6. It includes Model Zoo (collection of pre-trained models that are readily available for use)

How does Detectron2 work?

1. Detectron2 employs a two-stage approach for object detection and segmentation:
2. Region Proposal Network (RPN):

• In the first stage, Detectron2 uses a Region Proposal Network (RPN) to scan the image and generate a set of candidate regions that are likely to contain objects. This process helps narrow down the areas of interest, reducing the computational load for the next stage.

1. Mask R-CNN:

• In the second stage, these candidate regions are passed to the Mask R-CNN model. Mask R-CNN is responsible for classifying each region into specific object categories and generating precise segmentation masks that delineate the object boundaries. This stage ensures that not only are objects detected, but they are also accurately segmented from the background.

# Note: This is a faster way to install detectron2 in Colab, but it does not include all functionalities (e.g. compiled operators).
# See https://detectron2.readthedocs.io/tutorials/install.html for full installation instructions
!git clone 'https://github.com/facebookresearch/detectron2'

Is Detectron2 worth it?

Absolutely. Detectron2 is a highly valuable tool for anyone working in the field of computer vision, particularly in tasks like object detection and segmentation. Here’s why:

1. Ease of use:

Detectron2 simplifies the often cumbersome process of implementing and integrating state-of-the-art models. With its unified API, you can easily deploy advanced models like Mask R-CNN, RetinaNet, and DensePose without the hassle of configuring each one individually.

1. Comprehensive Model Zoo:

The Model Zoo is a major advantage, providing access to a collection of pre-trained models that can be readily used or fine-tuned for your specific applications. We are going to implement this in next chapter.

In this post, will segment images using pre-trained models. But the real challenge begins when we push these models beyond their comfort zones — specifically, we’ll test how well they handle medical imaging data, a domain they were never trained on.

Next tutorial: Fine-tune Detectron2 for instance segmentation using custom data

# Some basic setup:
import detectron2
import torch

# import some common libraries
import numpy as np
import os, json, cv2, random

# import some common detectron2 utilities
from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog, DatasetCatalog

my_new_image = cv2.imread("data/street_small.jpg")
cv2.imshow(my_new_image)

0. How to use pre-trained model for various segmentation tasks.

1. Configuration Setup:

cfg_keypoint = get_cfg()   # get a fresh new config

get_cfg() is a function from Detectron2 that initializes a configuration object. This object (cfg_keypoint) will hold all the configuration settings for the model, including the model architecture, dataset, hyperparameters, and more.

cfg_keypoint.merge_from_file(model_zoo.get_config_file
("COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml"))

# Here it's keypoint but we can change that to segment/panoptic-segment

• This line loads a pre-defined configuration file for a specific keypoint detection model from the Detectron2 model zoo. The model used here is keypoint_rcnn_R_50_FPN_3x, which is a variant of Mask R-CNN designed for keypoint detection. It uses a ResNet-50 backbone with a Feature Pyramid Network (FPN) and is trained on the COCO dataset.

2. Model Configuration:

cfg_keypoint.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 
 # set threshold for this model

• This line sets the confidence score threshold for the model’s Region of Interest (ROI) heads during inference. Only detections with a confidence score higher than 0.7 will be considered valid.

cfg_keypoint.MODEL.WEIGHTS = model_zoo.get_checkpoint_url
("COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")

• This line sets the model weights to be used during inference. The weights are loaded from a pre-trained model available in the Detectron2 model zoo. The URL provided points to the checkpoint for the keypoint_rcnn_R_50_FPN_3x model trained on the COCO dataset.

3. Predictor Initialization:

predictor = DefaultPredictor(cfg_keypoint)

• DefaultPredictor is a simple wrapper around the model that takes care of running the inference pipeline. It sets up the model according to the configuration and allows you to easily make predictions on new images.

4. Running Inference:

outputs = predictor(my_new_image)

• This line runs inference on my_new_image, which should be a NumPy array representing an image. The model outputs a dictionary that contains the detected instances, including their bounding boxes, keypoints, class labels, and scores.

5. Visualization:

v = Visualizer(my_new_image[:,:,::-1], MetadataCatalog.get
(cfg_keypoint.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))

• Visualizer: Visualizer is a utility from Detectron2 that helps in visualizing predictions.
• Image Conversion: my_new_image[:,:,::-1] converts the image from BGR to RGB format, which is needed for proper visualization since OpenCV uses BGR by default, while most other libraries (including the Visualizer) expect RGB.
• Metadata: MetadataCatalog.get(cfg_keypoint.DATASETS.TRAIN[0]) fetches metadata (e.g., class names, keypoint names) related to the dataset used during training. This information is useful for drawing labels, keypoints, and other annotations on the image.
• Drawing Predictions: draw_instance_predictions(outputs["instances"].to("cpu")) draws the predicted bounding boxes, keypoints, and other annotations on the image. The to("cpu") part ensures that the output is moved to the CPU if it was originally on the GPU, which is necessary for visualization.

6. Displaying the Image:

cv2.imshow(out.get_image()[:, :, ::-1])

• Finally, the visualized image (with predictions drawn on it) is displayed using OpenCV’s imshow function. The [:, :, ::-1] is used again to convert the image back to BGR format for display in OpenCV.

1. KeyPoint Detection Model

• Keypoints are specific locations or landmarks in an image that are distinctive and informative. These key points are selected because they represent significant variations in the local image region and can be reliably detected and matched across different images.

# Inference with a keypoint detection model
cfg_keypoint = get_cfg()   # get a fresh new config
cfg_keypoint.merge_from_file(model_zoo.get_config_file("COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml"))
cfg_keypoint.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7  # set threshold for this model
cfg_keypoint.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg_keypoint)
outputs = predictor(my_new_image)
v = Visualizer(my_new_image[:,:,::-1], MetadataCatalog.get(cfg_keypoint.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
import matplotlib.pyplot as plt

# Convert BGR to RGB for displaying with matplotlib
plt.imshow(out.get_image()[:, :, ::-1])
plt.axis('off')
plt.show()

2. Instance Segmentation

• Instance segmentation is a computer vision task that involves identifying and delineating individual objects within an image by assigning a unique mask to each object instance.

# Inference with instance segmentation
cfg_inst = get_cfg()
cfg_inst.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg_inst.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
# Find a model from detectron2's model zoo.  https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md
cfg_inst.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg_inst)
outputs = predictor(my_new_image)

v = Visualizer(my_new_image[:, :, ::-1], MetadataCatalog.get(cfg_inst.DATASETS.TRAIN[0]), scale=1.0)
out = v.draw_instance_predictions(outputs["instances"].to("cpu"))

# Convert BGR to RGB for displaying with matplotlib
plt.imshow(out.get_image()[:, :, ::-1])
plt.axis('off')
plt.show()

3. Panoptic Segmentation

• Panoptic segmentation is a computer vision task that combines instance segmentation and semantic segmentation to label every pixel in an image with both a class category and a unique instance ID. You segment the entire scene not just the objects.

# Inference with a panoptic segmentation model
cfg_pan = get_cfg()
cfg_pan.merge_from_file(model_zoo.get_config_file("COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml"))
cfg_pan.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-PanopticSegmentation/panoptic_fpn_R_101_3x.yaml")
predictor = DefaultPredictor(cfg_pan)
panoptic_seg, segments_info = predictor(my_new_image)["panoptic_seg"]
v = Visualizer(my_new_image[:, :, ::-1], MetadataCatalog.get(cfg_pan.DATASETS.TRAIN[0]), scale=1.0)
out = v.draw_panoptic_seg_predictions(panoptic_seg.to("cpu"), segments_info)

# Convert BGR to RGB for displaying with matplotlib
plt.imshow(out.get_image()[:, :, ::-1])
plt.axis('off')
plt.show()

4. Custom Data — How Detectron2 fails to segment image of cells

• We need to train a custom model using our own data and labels. (Next tutorial) and will fine-tune Detectron2 for instance segmentation using custom data.

sci_im = cv2.imread("/home/ravina/Desktop/Detectron2/Images/cells.png")
plt.imshow(sci_im)
plt.axis('off')
plt.show()

sci_outputs = predictor(sci_im)
sci_v = Visualizer(sci_im[:, :, ::-1], MetadataCatalog.get(cfg_inst.DATASETS.TRAIN[0]))
sci_out = sci_v.draw_instance_predictions(sci_outputs["instances"].to("cpu"))

plt.imshow(sci_out.get_image()[:, :, ::-1])
plt.axis('off')
plt.show()

Code Link — GITHUB