Semantic vs instance vs panoptic segmentation

The difference between semantic vs. instance vs. panoptic segmentation lies in how they process the things and stuff in the image.

Semantic segmentation studies the uncountable stuff in an image. It analyzes each image pixel and assigns a unique class label based on the texture it represents. For example, in Figure 1, an image contains, three people, a beach, a kite and the sky. The three people represent the same texture.

Figure1

Semantic segmentation would assign unique class labels to each of these textures or categories. However, semantic segmentation’s output cannot differentiate or count the two cars or three pedestrians separately. Commonly used semantic segmentation techniques include SegNet, U-Net, DeconvNet, and FCNs.

Instance segmentation typically deals with tasks related to countable things. It can detect each object or instance of a class present in an image and assigns it a different mask or bounding box with a unique identifier.

For example, instance segmentation would identify the three people in the previous example separately as, let’s say, person1, person2 and person3. Commonly used instance segmentation techniques are Mask R-CNN, Faster R-CNN, PANet, and YOLACT. Figure 2 demonstrates different instance segmentation detections.

The goal of both semantic and instance segmentation techniques is to process a scene coherently. Naturally, we want to identify both stuff and things in a scene to build more practical real-world applications. Researchers devised a solution to reconcile both stuff and things within a scene (i.e., panoptic segmentation).

Panoptic segmentation is the best of both worlds. It presents a unified image segmentation approach where each pixel in a scene is assigned a semantic label (due to semantic segmentation) and a unique instance identifier (due to instance segmentation).

Things and Stuff Classification in Panoptic Segmentation

In panoptic segmentation, objects in an image are typically classified into two main categories: “things” and “stuff.”

• Things: Things in a panoptic image segmentation technique refer to countable and distinct object instances within an image, such as cars, people, animals, furniture, etc. Each object and instance in a scene has well-defined boundaries and is identified and separated as individual instances.
• Stuff: Stuff in panoptic image segmentation refers to amorphous or uncountable regions in an image, such as sky, road, grass, walls, etc. These regions do not have well-defined boundaries and are typically treated as a single continuous segment without individual instances.

Panoptic segmentation assigns each pixel only one pair of a semantic label and an instance identifier. However, objects can have overlapping pixels. In this case, panoptic segmentation resolves the discrepancy by favoring the object instance, as the priority is to identify each thing rather than stuff. Figure 3 demonstrates different panoptic segmentation detections.

• Semantic Segmentation: Labels each pixel by class (e.g., all cars as “car”) without distinguishing between different instances.
• Panoptic Segmentation: Labels each pixel by class and also identifies different instances of the same class (e.g., each car gets a unique label).

Instance vs Panoptic Segmentation with an example

Evaluation metrics

Each segmentation technique uses different evaluation metrics to assess the predicted masks or identifiers in a scene. That is because stuff and things are processed differently.

Semantic segmentation normally employs the Intersection over Union (**IoU**) metric (also referred to as the Jaccard Index), which checks the similarity between the predicted and ground truth masks. It determines how much area overlaps between the two masks. Besides IoU, we can also use the dice coefficient, pixel accuracy, and mean accuracy metrics to perform a more robust evaluation. These metrics do not consider object-level labels.

Instance segmentation, on the other hand, uses Average Precision (AP) as the standard evaluation metric. The AP metric uses the IoU on a pixel-to-pixel basis for each object instance.

AP is averaged over all categories. Traditionally, this is called “mean average precision” (mAP). We make no distinction between AP and mAP (and likewise AR and mAR) and assume the difference is clear from context.

Finally, panoptic segmentation uses the Panoptic Quality (PQ) metric, which evaluates the predicted masks and instance identifiers for both things and stuff . PQ unifies evaluation over all classes by multiplying segmentation quality (SQ) and recognition quality (RQ) terms. SQ represents the average IoU score of the matched segments, while RQ is the F1 score calculated using the precision and recall values of the predicted masks.