Abstract
Earth vision has achieved milestones in geospatial object recognition but lacks exploration in object-relational reasoning, limiting comprehensive scene understanding. To address this, a progressive Earth vision-language understanding and generation framework is proposed, including a multi-task dataset (EarthVLSet) and a semantic-guided network (EarthVLNet). Focusing on city planning applications, EarthVLSet includes 10.9k sub-meter resolution remote sensing images, land-cover masks, and 761.5k textual pairs involving both multiple-choice and open-ended visual question answering (VQA) tasks. In an object-centric way, EarthVLNet is proposed to progressively achieve semantic segmentation, relational reasoning, and comprehensive understanding. The first stage involves land-cover segmentation to generate object semantics for VQA guidance. Guided by pixel-wise semantics, the object awareness based large language model (LLM) performs relational reasoning and knowledge summarization to generate the required answers. As for optimization, the numerical difference loss is proposed to dynamically add difference penalties, addressing the various objects' statistics. Three benchmarks, including semantic segmentation, multiple-choice, and open-ended VQA demonstrated the superiorities of EarthVLNet, yielding three future directions: 1) segmentation features consistently enhance VQA performance even in cross-dataset scenarios; 2) multiple-choice tasks show greater sensitivity to the vision encoder than to the language decoder; and 3) open-ended tasks necessitate advanced vision encoders and language decoders for an optimal performance. We believe this dataset and method will provide a beneficial benchmark that connects ''image-mask-text'', advancing geographical applications for Earth vision.
Abstract (translated)
地球视觉在地理空间对象识别方面取得了里程碑式的成就，但在物体间关系推理方面的探索不足，这限制了对场景的全面理解。为了解决这一问题，提出了一种渐进式地球视觉-语言理解和生成框架，包括一个多任务数据集（EarthVLSet）和一个语义引导网络（EarthVLNet）。该框架特别关注城市规划应用领域，EarthVLSet包含了10.9k张亚米级分辨率的遥感图像、土地覆盖掩模以及761.5万个涉及多项选择题和开放式视觉问答(VQA)任务的文本对。以对象为中心的方式，EarthVLNet旨在逐步实现语义分割、关系推理和综合理解。第一阶段是进行土地覆盖分割，生成用于VQA引导的对象语义信息。通过像素级语义指导，基于大语言模型（LLM）的对象意识执行关系推理和知识总结，从而生成所需答案。为了优化性能，提出了数值差异损失函数，动态添加差异惩罚以应对各种对象的统计特性。 在三个基准测试中——包括语义分割、多项选择题以及开放式VQA任务，EarthVLNet表现出了优越性，并指出了未来研究的三大方向：1）段落特征能够持续提升VQA性能，即使是在跨数据集场景下；2）多项选择任务对视觉编码器比语言解码器更为敏感；3）开放式任务需要更先进的视觉和语言模型以实现最佳效果。我们相信这一数据集和方法将为连接“图像-掩模-文本”的研究提供一个有益的基准，从而促进地球视觉在地理应用领域的进一步发展。
URL
https://arxiv.org/abs/2601.02783
