Scaling POI Relocalization with Mapillary

The Next Challenge

In our last post we took a swing at relocalizing POIs in Overture using our Zehpr pose app to collect high accuracy geo-positions and photos. The results were quite encouraging, but there was a problem. Our pose app doesn’t have millions of users, so we have a scaling challenge to get the data we need. Fortunately there is the wonderful source of open street view imagery we could tap, Mapillary.

Intentional vs. Serendipitous

There is one key difference between how we use our pose app and how Mapillary collects imagery. For the Zephr pose app we intentionally took photos of each POI. In the case of Mapillary it is not taking intentional photos of POIs. Instead it is taking regularly timed photos of the street as a vehicle or pedestrian moves down it.

To overcome this challenge we need to figure out which Mapillary images might have an Overture POI in the image. Then confirm it in a QA process. The first step of selecting Mapillary image candidates we can do with some classic API queries, field of view calculations and geoprocessing to get a candidate set of images. Then comes the more challenging task of confirming which Mapillary image contains the POI. For this task we use a multi-modal LLM to look for signage that matches the name of the POI in Overture.

POI Hunting in Mapillary

In order to find our candidate images in Mapillary we used the Graph API with a radius and a closeto (latitude, longitude) filter. This allowed us to set a radius and collect all the Mapillary images within a set distance from an Overture POI.

Then came the more challenging task. From our candidate pool of Mapillary images we needed to determine which image was taken with a camera view of the POI. Fortunately Mapillary exposes the camera parameters for each image they collect:

Mapillary Camera Parameters

Then if computed_focal_length_px is provided and the image width is known we can calculate a two dimensional field of view:

Now we can look for Overture POIs that intersect our field of view. This give us a pretty solid set of candidate images. This graphic sums up the approach visually:

Matching Mapillary Images to Overture POIs

While effective this approach can create many false positives. If the Overture POI’s location if off of reality by more than our search radius we might not get images that match. Also the accuracy of the Mapillary GPS coordinates can have positional errors that cause false positives. Often the GPS coordinates and the FOV met the criteria, but the image doesn’t capture the POI because or errors in the sensor data. Here is an example where the criteria are met but the image is past the POI and doesn’t capture it:

Mapillary False Positive for “The Huckleberry”

If we look at previous images collected from the same camera we see a solid image of “The Huckleberry”.

Mapillary True Positive for “The Huckleberry”

These false positives can easily throw off the relocalization algorithm resulting in a bad location for the POI. We found this in our first experiments, which were exceedingly mediocre.

AI to Save the Day

To solve the problem we needed to QA our Mapillary data to ensure that the images we curated had the POI in the photograph. To help solve this problem we went to our friends at Lazarus AI. They integrated the geospatial metadata from Overture with visual data from street-level imagery, applying transformer-based vision-language models to identify key visual indicators, such as signage or building features. By combining geographic context and visual analysis, we were able to automates the accurate confirmation of real-world POIs directly from images.

The Relocalization Results

This gave a great set of confirmed Mapillary images of POIs in Overture. We then used that pool of Mapillary images for each POI in our study area to run our relocalization routine. The results are shown in the map below where the “red” pin is the original Overture location and the “green” pin is the relocalization result. We also connect a blue polyline between the positions to calculate the distance of the location change from the relocalization model.

Results of Relocalizing Overture POI Positions with Mapillary Images

If we chart the distances of the POI location changes we can see several shifts were small, but there are meaningful number of bigger shifts improving POI positioning.

The average improvement in distance for the POIs we relocalized was 13.72 meters.

Shortcomings and Solutions

It should be noted that there were shortcomings to this approach. For several POI we got no results because there were not enough images of the POI to run the model or we couldn’t confirm that the image had the POI in the frame. Since some POIs have no signage or street address or they are occluded we can end up with false negatives. The other problem is the POI’s location in the database can be so far off it prevents an image of it in the real world from being captured. In this scenario we are simply querying the wrong location.

Both of these problems can be potentially addressed by increasing the radius of our query and/or increasing our field of view. For this test we cut off images at ten candidates. With the process now automated it is feasible to increase both parameters to give us a bigger pool of Mapillary images to QA.

The second problem is the geographic location of the Mapillary images often has significant errors because of poor GPS positioning. This can cause the relocalization model to not find the correct real world location for the POI. This is probably the most persistent shortcoming we see in the results. The location of Overture POI is improved but often is not aligned fully with the building footprint. Our last test will be to augment the Mapillary images with the Zephr positioning SDK to see if we can get the last meters of improvement for full “POI to building” conflation. Stay tuned :-)