Traffic Light Recognition (TLR) Architecture: 2D Bounding Box Detection

The TLR model is a Fully Convolutional Network (FCN) + FPN + Header model, utilizing an “anchor-free” approach. Instead of guessing thousands of anchor boxes, it treats traffic lights as single center points and regresses all other properties (size, sub-pixel offset, and classifications) directly from that point.

Different Types of Traffic Light

1. The Output Grid

When a 1920x960 image passes through the network, it is downsampled by a down_ratio of 4. The network outputs six massive 240x120 feature maps.

Because the network doesn’t know where the traffic lights are yet, it makes predictions for every single pixel in that 240x120 grid simultaneously.

2. Encoding Ground Truth: Dense Regression & “Smearing”

During training, if the ground truth only penalized the network at the exact 1-pixel dead center of a traffic light, the model would fail to converge. To fix this, the architecture uses Dense Regression to create a “grace area” around the true center.

1. The Blast Radius: For every true traffic light, a radius is calculated based on its actual width and height (gaussian_radius).
2. Patch Generation: A circular patch is generated around the center point.
3. Smearing: Every single pixel inside that patch is filled with the exact same target values: the true wh, the true bulb_cls, and the true nb_cls.
4. Overlap Resolution: If two traffic light patches overlap, a Gaussian bell curve determines which pixel belongs to which object based on proximity to the respective centers.

3. Training & Loss Functions: Erasing the Background

Now that the grid is full of small target patches and tens of thousands of empty “background” pixels, the loss functions must be carefully controlled so the model isn’t trained to predict zeros everywhere.

• Width/Height (wh) & Offset (reg): The pipeline generates a binary mask (dense_wh_mask) from the heatmap. When calculating the Masked L1 Loss, both the predictions and ground truth are multiplied by this mask. This forces all background pixels to zero, effectively erasing them from the math so the network is only penalized for its guesses inside the smeared patches.
• Classifications (bulb_cls, arrow_cls, nb_cls): Instead of a manual mask, the classification matrices are initialized with zeros (where 0 is the background class). The model uses PyTorch's CrossEntropyLoss(ignore_index=0). This natively tells the loss function to completely bypass and ignore any pixel labeled as 0, achieving the same background-erasing effect natively.

(Note: The heatmap (hm) itself is trained using a Focal Loss penalty, which handles the massive imbalance between the few positive center points and the thousands of negative background pixels).

4. Inference & Post-Processing: Plucking the Truth

While the model trains on smeared patches, inference relies entirely on finding the single, perfect center point. Post-processing strips away the grid to build final bounding boxes.

Step A: Finding the Peaks (NMS)

The predicted heatmap is passed through a 3x 3 max-pooling operation. This acts as highly efficient Non-Maximum Suppression (NMS), wiping out overlapping predictions and leaving only the sharpest peaks. The _topk function grabs the coordinates (xs, ys) and grid indices (inds) of the top-K highest confidence peaks.

Step B: Collapsing the Classes

The raw classification feature maps contain probabilities. The pipeline slices off the background index 0 [:,1:,:,:] and applies an argmax to collapse the probabilities into hard integer guesses for every pixel on the grid.

Step C: The “Gather” Operation

Using the inds extracted from the heatmap peaks, custom functions acts as a lookup table. It goes into the massive wh, reg, and class grids and plucks out the predictions strictly at those exact top-K center coordinates. The rest of the 240x120 grid is completely discarded.

Step D: Reconstructing the Final Boxes

1. Sub-pixel Correction: The plucked integer center coordinates (xs, ys) suffered from quantization error when the image was scaled down by 4. The model adds the plucked fractional offset reg to the integers to recover decimal precision (e.g., 101 + 0.25 = 101.25).
2. Applying Dimensions: The plucked width and height are divided by 2 and added/subtracted from the center point to generate the [x1, y1, x2, y2] bounding box corners.
3. Rescaling: The coordinates are multiplied by the resize ratio to scale them from the 240x120 feature map back to the original 1920x960 image resolution.
4. Class Mapping: The plucked class IDs are shifted by +1 to restore the original ground-truth mapping, reversing the slice performed during Step B.