Abstract
Recent advancements in Large Vision-Language Models (VLMs) have shown great promise in natural image domains, allowing users to hold a dialogue about given visual content. However, such general-domain VLMs perform poorly for Remote Sensing (RS) scenarios, leading to inaccurate or fabricated information when presented with RS domain-specific queries. Such a behavior emerges due to the unique challenges introduced by RS imagery. For example, to handle high-resolution RS imagery with diverse scale changes across categories and many small objects, region-level reasoning is necessary alongside holistic scene interpretation. Furthermore, the lack of domain-specific multimodal instruction following data as well as strong backbone models for RS make it hard for the models to align their behavior with user queries. To address these limitations, we propose GeoChat - the first versatile remote sensing VLM that offers multitask conversational capabilities with high-resolution RS images. Specifically, GeoChat can not only answer image-level queries but also accepts region inputs to hold region-specific dialogue. Furthermore, it can visually ground objects in its responses by referring to their spatial coordinates. To address the lack of domain-specific datasets, we generate a novel RS multimodal instruction-following dataset by extending image-text pairs from existing diverse RS datasets. We establish a comprehensive benchmark for RS multitask conversations and compare with a number of baseline methods. GeoChat demonstrates robust zero-shot performance on various RS tasks, e.g., image and region captioning, visual question answering, scene classification, visually grounded conversations and referring detection. Our code is available at this https URL.
Abstract (translated)
近年来，在自然图像领域，大型视觉语言模型（VLMs）的进步已经表现出很大的潜力，使用户可以就给定的视觉内容进行对话。然而，在遥感（RS）场景中，这种通用域的VLM表现不佳，当面对RS领域特定的查询时，呈现出的信息往往不准确或捏造。这种行为是由RS图像独特带来的挑战所引起的。例如，为了处理具有不同尺度变化跨类别的较高分辨率RS图像，区域级推理是必要的，同时进行整体场景解释。此外，缺乏RS领域的多模态指令跟随数据以及强大的骨干模型也使得模型难以将行为与用户查询对齐。为了克服这些限制，我们提出了GeoChat - 第一个具有高分辨率RS图像多任务会话功能的遥感VLM。具体来说，GeoChat不仅可以回答图像级别的问题，还可以接受区域输入来保持区域特定的对话。此外，它可以通过参考它们的空间坐标在响应中视觉 grounding 对象。为了克服缺乏RS领域特定数据集的问题，我们通过扩展现有多样RS数据集中的图像-文本对来生成一个新的RS多模态指令跟随数据。我们为RS多任务会话建立了全面的基准，并将其与多个基线方法进行比较。GeoChat在各种RS任务上都表现出出色的零散 shot性能，例如图像和区域捕捉、视觉问题回答、场景分类、视觉 grounded 对话和参考检测。我们的代码可在此处访问：https:// this URL.
URL
https://arxiv.org/abs/2311.15826
