Paper reading: GeoChat

Large Vision-language Models (VLMs) have shown great success in natural image domains. However, there is a performance drop in remote sensing image domains due to the domain shifts. To address this issue, the paper “GeoChat: Grounded Large Vision-Language Models for Remote Sensing”, accepted by CVPR 2024, proposed the first versatile remote sensing VLM that offers multitask conversational capabilities. They also generated a novel RS multimodal instruction-following dataset. In this story, we will provide an overview of this paper. The code is available here.

GeoChat: Grounded Remote Sensing VLM

GeoChat is designed not only for the task of visually grounded conversations but also for other spatial reasoning tasks with varying levels of granularity. The tasks GeoChat can achieve are listed below:

Image-Level Conversation Tasks Conservation-based tasks at a holistic level of image. For example, visual question answering (VQA), scene classification, and image captioning.

Region-Level Conversation Tasks Providing spatial box locations b in the image. For example, region-level captioning, region-specific VQA, and multi-turn conversation.

Grounded Conversation Tasks With the spatial task tokens t, provide object locations at different granularities. For example, grounded image captioning/conservation, object grounding, and referring expression detection.

Architecture

The figure above shows an overview of GeoChat. The architecture consists of three components: Global Image encoder (pretrained CLIP-ViT), MLP Adaptor, and LLM (Vicuna-v1.5(7B)). They also added a specific task prompt that indicates the type of task. Additionally, as well as spatial positions within both inputs and outputs to enable region inputs and object grounding.

RS Multimodal Instruction Dataset

As summarized in the table below, they utilized multimodal instruction-following data with multi-round conversations regarding remote sensing images. The total dataset size is 306k for training and 12k for testing.

In the compilation, they incorporated three different types of datasets: object detection (DOTA, DIOR, FAIR1M), scene classification (NWPU-RESISC-45), and VQA (LRBEN, Floodnet).

Result

Scene Classification

The classification accuracy on UCMerced and AID is shown in the table below. GeoChat significantly outperformed existing models.

VQA

The VQA accuracy for each class and overall on RSVQA-LRBEN is shown in the table below. GeoChat outperformed other zero-shot models and performed close to the supervised SOTA models, despite not being finetuned.

The qualitative result is shown in the figure below.

Visual Grounding

The detection accuracy@0.5 IoU on the proposed benchmark is shown in the table below. The model performance is low on small objects or multiple boxes prediction, but GeoChat performed better than MiniGPTv2 on medium size images.

Conclusion

In this paper, GeoChat, the first unified remote sensing VLM was presented. GeoChat not only answers image-level queries but also engages in region-specific dialogues, grounding responses with precise spatial coordinates. A novel RS multimodal instruction-following dataset was also created to train the model. GeoChat achieved robust zero-shot performance across various RS tasks.