Abstract
Satellite imagery differs fundamentally from natural images: its aerial viewpoint, very high resolution, diverse scale variations, and abundance of small objects demand both region-level spatial reasoning and holistic scene understanding. Current remote-sensing approaches remain fragmented between dual-encoder retrieval models, which excel at large-scale cross-modal search but cannot interleave modalities, and generative assistants, which support region-level interpretation but lack scalable retrieval capabilities. We propose $\textbf{VLM2GeoVec}$, an instruction-following, single-encoder vision-language model trained contrastively to embed interleaved inputs (images, text, bounding boxes, and geographic coordinates) in a unified vector space. Our single encoder interleaves all inputs into one joint embedding trained with a contrastive loss, eliminating multi-stage pipelines and task-specific modules. To evaluate its versatility, we introduce $\textbf{RSMEB}$, a novel benchmark covering key remote-sensing embedding applications: scene classification; cross-modal search; compositional retrieval; visual-question answering; visual grounding and region-level reasoning; and semantic geospatial retrieval. On RSMEB, it achieves $\textbf{26.6%}$ P@1 on region-caption retrieval (+25 pp vs. dual-encoder baselines), $\textbf{32.5%}$ P@1 on referring-expression retrieval (+19 pp), and $\textbf{17.8%}$ P@1 on semantic geo-localization retrieval (over $3\times$ prior best), while matching or exceeding specialized baselines on conventional tasks such as scene classification and cross-modal retrieval. VLM2GeoVec unifies scalable retrieval with region-level spatial reasoning, enabling cohesive multimodal analysis in remote sensing. We will publicly release the code, checkpoints, and data upon acceptance.
Abstract (translated)
卫星图像与自然图像在本质上有所不同：其空中视角、极高分辨率、多样化的尺度变化以及丰富的小物体，这些特性要求进行区域级空间推理和整体场景理解。当前的遥感方法仍处于片段化状态，一方面表现为双编码器检索模型，在大规模跨模态搜索方面表现出色但不能交织不同模式；另一方面则是生成式助手，支持区域级别的解释但是缺乏可扩展的检索能力。我们提出了**VLM2GeoVec**，这是一种训练中的指令跟随型单编码视觉-语言模型，该模型采用对比学习方式将交错输入（包括图像、文本、边界框和地理坐标）嵌入到统一的向量空间中。我们的单一编码器将所有输入交织成一个联合嵌入，并使用对比损失进行训练，从而消除了多阶段管道和特定任务模块的需求。 为了评估其多样性，我们引入了**RSMEB**这一新基准测试，涵盖关键的遥感嵌入应用：场景分类；跨模态搜索；组合检索；视觉问题回答；视觉定位与区域级推理；以及语义地理空间检索。在RSMEB上，VLM2GeoVec实现了以下性能： - 区域描述检索上的P@1为**26.6%**（相比双编码器基线高出25个百分点）； - 引用表达式检索上的P@1为**32.5%**（比基线高19个百分点）； - 语义地理定位检索上的P@1为**17.8%**（是先前最佳结果的三倍以上）。 同时，它在场景分类和跨模态检索等传统任务上也能匹配或超越专门化的基准。VLM2GeoVec统一了可扩展检索与区域级空间推理的能力，使遥感中的多模式分析更加连贯。我们将在接受后公开发布代码、检查点及数据集。
URL
https://arxiv.org/abs/2512.11490
