Understanding DETR: When Transformers Decide Bounding Boxes Are Just Another Sequence

Paper: End-to-End Object Detection with Transformers (https://arxiv.org/pdf/2005.12872) May 2020, Facebook AI

DETR: Detection Transformer

An example output. Taken from the paper.

Traditionally, detection models use complex strategies that employ the use of CNNs along with additional non-DL AI algorithms in order to generate bounding boxes with classification for objects of interest. The authors of this paper broke this trend by targeting the same goal but by using a simple transformer-based architecture and a smart method to calculate the loss. The authors strip detection architectures of region proposal networks, anchor boxes, non-max suppression (NMS), etc. and train a model to directly predict a set of bounding boxes. The authors achieve commendable results matching the widely used Faster-RCNN architecture on the COCO dataset.

The Architecture

The DETR Architecture. Taken from the paper.

The DETR architecture is simple. A CNN backbone extracts features from the input image into a smaller latent space. Any CNN can be used, making this method versatile. The authors use ResNet50 and ResNet101 in their experiments. Then a series of transformer encoder layers perform self attention on the tokenized versions of these image features in order to capture global relations in the image features. Following this, a series of transformer decoder layers are used, to which the keys and values are provided by the modified image features. The queries (denoted as “object queries” in the diagram) are learnable vectors, where each query corresponds to one bounding box prediction by the network. Therefore, the number of object queries chosen is the maximum number of objects that the network can predict for any given input image. Each of the tokens of the output of the final transformer decoder layer are passed through a simple FFN to get the final bounding box prediction.

In contrast to traditional detection architectures, the authors rely on the self-attention mechanism in the transformer decoder layers to avoid duplicate bounding box predictions rather than using NMS. Attention mechanisms are also permutation-invariant, and with the lack of any position information embedded into the object queries, this method truly makes it a set prediction problem.

For the FFN, the authors use a simple 3-layer MLP with ReLU activations. The bounding box predictions include the normalized bounding box center coordinates, height, width, “no-object” score, and class scores.

Position embeddings

The authors use an interesting method of encoding position embeddings into their architecture (which I found to be sidelined in their paper). They add fixed sinusoidal embeddings to the queries and keys of all transformer encoder layers. They also add this to the keys of all transformer decoder cross-attention layers. Similarly, the learnable “object queries” are treated as position embeddings too and are added to the queries and keys of all the transformer decoder self-attention layers as well as the queries of all the transformer decoder cross-attention layers.

DETR’s transformer architecture. Taken from the paper.

The authors have provided ablations for different types and ways of position embeddings, please refer to the paper for this. A gist is provided below:

• “spatial pos. enc.” corresponds to the positional embeddings that are added at various stages of the transformer encoder and decoder layers.
• “output pos. enc.” corresponds to the object queries.
• “sine” corresponds to fixed sinusoidal position embeddings; “learned” corresponds to learnable position embeddings.
• “at input” implies that the embeddings are added only to the input of the encoder / decoder; “at attn” implies the embeddings are added at every stage of the attention.

Ablations done on different position embeddings. Taken from the paper.

The authors also explore panoptic segmentation by adding a segmentation head on top of the predicted bounding boxes and taking a pixel-wise argmax of the outputs. We will no go in detail here as this post is mainly geared towards object detection.

The Loss Function

All prior works which forayed into detection methods using transformers predicted bounding boxes autoregressively. In contrast to this, the authors of this paper use parallel decoding of the object queries (as describe above) to predict the outputs.

DETR uses bipartite matching between the set of target bounding boxes and prediciton bounding boxes using the Hungarian algorithm. The cost function for this algorithm involves a simple linear combination of three measures:

• Cross entropy between target “no-object” score + class scores and prediction “no-object” score + class scores.
• L1 distance between target and prediction bounding box parameters.
• Generalized IOU between target and prediction bounding boxes.

The Hungarian algorithm finds that matching that minimizes the overall cost. This cost function is then used as the loss value and is used for backpropagation.

The authors also add auxiliary losses to stabilize training. These losses are the same as bipartite matching loss, but on intermediate layers of the decoder. The matching used for this is still the same one used i.e. from the last decoder layer. The weights of the FFNs are shared in these auxiliary losses. An additional layer-norm is added at the start of the FFN to normalize inputs.

The Data

The authors use the COCO 2017 detection dataset containing 118k training images and 5k validation images. There were 7 objects per image on average, with a maximum of 63 objects.

Challenges with DETR

• No biases — It is worth noting that in addition to the loss of spatial inductive bias that CNNs have, DETR also loses out on estimated shapes and sizes of predicted bounding boxes that are engineered in other detection paradigms. The combination of these makes it slower to train detection models using DETR compared to other methods like Faster R-CNN. However, due to it’s simplicity and elegance, given enough data, I still prefer DETR over traditional CNN methods.
• Sparse gradients — Another reason for slow training is that while the model gets training signals for all predicted bounding boxes when it comes to classification (into the “no-object” class or one of the possible classes), the model only gets gradients from the matched bounding boxes for the bounding box parameters. In datasets with heavy imbalance (of number of objects), which are almost all of them, bounding box parameters may take some more time to learn.
• Bounding box collapse — occurs when bounding box parameters move to 1.0 (to satisfy GIOU loss) or to 0.0 (to satisfy L1 loss when objects in the image are relatively small). This can happen when loss components are not weighed carefully, or in the early stages of training where there is more signal towards collapse. Definitely train for a long time before concluding bounding box collapse as it takes some time for the model to move towards correct bounding box parameters. Increase batch size as that can help. Add an auxiliary loss that pushes the parameters of unmatched bounding boxes towards fixed values (such as 0.5) to provide signal to those parameters too, but make sure to phase it out over time to avoid query collapse.
• Query collapse — try auxiliary repulsion losses (push predicted bounding box centers away from each other using negative of L1 loss / apply negative of generalized IOU loss on predicted bounding boxes such that overlap is discouraged).

Some more resources:

• Hungarian matching: https://www.youtube.com/watch?v=dQDZNHwuuOY
• https://www.youtube.com/watch?v=T35ba_VXkMY

[embed]Glossary: A Reference to All My Articles I wanted to provide a method for readers to be able to find any and all of my articles in one place with some sort of…medium.com

This article presents my interpretation of the research paper presented. The original work and all associated intellectual property belong to the authors. My aim is to highlight the paper’s key contributions and rationale while providing my perspective on its novelty. This analysis does not claim to be exhaustive or definitive, and any misinterpretations are unintentional. I welcome constructive discussion and alternative viewpoints in the comments.