Abstract
As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. Our dataset and code are available in this https URL.
Abstract (translated)
作为人类活动的中心，城市表面包含了大量的语义实体。从卫星图像中分割这些不同的实体对于各种下游应用至关重要。目前先进的分割模型可以可靠地分割由物理属性定义的实体（如建筑物、水域），但对于社会定义的类别（如学校、公园）仍然存在困难。在这项工作中，我们通过视觉-语言模型推理实现了社会语义分割。为此，我们引入了名为SocioSeg的城市社会语义分割数据集，这是一个包含卫星图像、数字地图和按层次结构组织的社会语义实体像素级标签的新资源。 此外，我们提出了一种新颖的视觉-语言推理框架，称为SocioReasoner，该框架通过跨模态识别和多阶段推理来模拟人类识别并标注社会语义实体的过程。我们利用强化学习优化这个非微分过程，并激发了视觉-语言模型的推理能力。实验表明，我们的方法在最先进的模型上表现出色，并具有强大的零样本泛化性能。 我们的数据集和代码可在以下网址获取：[此URL]。
URL
https://arxiv.org/abs/2601.10477
