Feature Pyramid Networks (FPN)

FPN was proposed in 2017 in a paper titled “Feature Pyramid Networks for Object Detection”. A Feature Pyramid Network, or FPN, is a feature extractor that takes a single-scale image of an arbitrary size as input, and outputs proportionally sized feature maps at multiple levels, in a fully convolutional fashion. This process is independent of the backbone convolutional architectures. It therefore acts as a generic solution for building feature pyramids inside deep convolutional networks to be used in tasks like object detection.

Image1 is architecture when predicting at finest level and image2 is predicting at different levels (FPN)

ARCHITECTURE :

The goal is to leverage ConvNet’s pyramidal feature hierarchy and build a feature pyramid with high-level semantics. We focus on sliding window proposers (RPN) and region-based detectors (Fast R-CNN). Our method takes a single-scale image and outputs proportionally sized feature maps at multiple levels, independent of the backbone convolutional architectures. The construction of our pyramid involves a bottom-up pathway, a top-down pathway, and lateral connections. In this paper we presented the results using ResNets.

1. Bottom Up Pathway

The bottom-up pathway is the feedforward computation of the backbone Convolution Networks, which computes a feature hierarchy consisting of feature maps at several scales with a scaling step of 2. For the feature pyramid, one pyramid level is defined for each stage. The output of the last layer of each stage is used as a reference set of feature maps.

Specifically, for ResNets we use the feature activations output by each stage’s last residual block. We denote the output of these last residual blocks as {C2, C3, C4, C5} for conv2, conv3, conv4, and conv5 outputs, and note that they have strides of {4, 8, 16, 32} pixels with respect to the input image. We do not include conv1 into the pyramid due to its large memory footprint.

2. Top Down Pathway and Lateral Connection

The top-down pathway in the Feature Pyramid Network (FPN) is a process where information flows from higher-level feature maps to lower-level feature maps.

It involves taking semantically meaningful information from the higher pyramid levels (which have coarser spatial resolution) and using it to hallucinate or create more detailed features in the lower pyramid levels (which have finer spatial resolution). This allows for better object detection and recognition at different scales, as the network can combine both low-level details and high-level semantic information to generate a more complete representation of the image.

These features are then enhanced with features from the bottom-up pathway via lateral connections. Each lateral connection merges feature maps of the same spatial size from the bottom-up pathway and the top-down pathway. The bottom-up feature map is of lower-level semantics, but it’s activations are more accurately localized as it was subsampled fewer times.

map is then merged with the corresponding bottom-up map (which undergoes a 1×1 convolutional layer to reduce channel dimensions) by element-wise addition. This process is iterated until the finest resolution map is generated. To start the iteration, we simply attach a 1×1 convolutional layer on C5 to produce the larger resolution map.

Finally, we append a 3×3 convolution on each merged map to generate the final feature map, which is to reduce the aliasing effect of upsampling. This final set of feature maps is called {P2, P3, P4, P5}, corresponding to {C2, C3, C4, C5} that are respectively of the same spatial sizes.

They set d = 256 in this paper and thus all extra convolutional layers have 256-channel outputs. There are no non-linearities in these extra layers, which we have empirically found to have minor impacts.

Simplicity is central to our design and we have found that our model is robust to many design choices. We have experimented with more sophisticated blocks (e.g., using multilayer residual blocks as the connections) and observed marginally better results. Designing better connection modules is not the focus of this paper, so they opt for the simple design described above.

Results :

Results on RPN

Placing FPN in RPN improves AR (average recall: the ability to capture objects) to 56.3, an 8.0 points improvement over the RPN baseline. The performance on small objects is increased by a large margin of 12.9 points.

Results on Faster R-CNN

The FPN based Faster R-CNN achieves an inference time of 0.148 second per image on a single NVIDIA M40 GPU for ResNet-50 while the single-scale ResNet-50 baseline runs at 0.32 seconds. The baseline comparison for FPN using Faster R-CNN. (FPN introduces a small extra cost for those extra layers in the FPN, but has a lighter weight head in the FPN implementation.)

Results on COCO dataset

FPN is very competitive with state-of-the-art detectors. In fact, it beats the winners for the COCO 2016 and 2015 challenge.

CONCLUSION :

1. FPN is not an object detector by itself. It is a feature extractor that works with object detectors.
2. FPN aims to address the problem of feature reuse and resolution degradation by using a top-down architecture that combines high-level, semantically rich features with low-level, semantically poor features
3. FPN introduces lateral connections that allow features from the bottom-up pathway to be combined with features from the top-down pathway, thus enabling the creation of a feature pyramid that maintains both resolution and semantic information.
4. FPN is computationally efficient and can be trained end-to-end with standard backpropagation algorithms.
5. FPN has been shown to improve the performance of object detection models, particularly on small objects and at low resolutions
6. FPN has been adopted in several state-of-the-art object detection models, including Faster R-CNN, RetinaNet, and Mask R-CNN.
7. FPN can also be used in other computer vision tasks, such as semantic segmentation and instance segmentation.

References :

FPN Paper Link : https://arxiv.org/abs/1612.03144

ResNet Paper Link : https://arxiv.org/abs/1512.03385

Implemention in Pytorch

Kaggle Notebook : https://www.kaggle.com/code/qdpatidar687/fpn-on-resnet18