Abstract
Automatically extracting vectorized building contours from remote sensing imagery is crucial for urban planning, population estimation, and disaster assessment. Current state-of-the-art methods rely on complex multi-stage pipelines involving pixel segmentation, vectorization, and polygon refinement, which limits their scalability and real-world applicability. Inspired by the remarkable reasoning capabilities of Large Language Models (LLMs), we introduce VectorLLM, the first Multi-modal Large Language Model (MLLM) designed for regular building contour extraction from remote sensing images. Unlike existing approaches, VectorLLM performs corner-point by corner-point regression of building contours directly, mimicking human annotators' labeling process. Our architecture consists of a vision foundation backbone, an MLP connector, and an LLM, enhanced with learnable position embeddings to improve spatial understanding capability. Through comprehensive exploration of training strategies including pretraining, supervised fine-tuning, and preference optimization across WHU, WHU-Mix, and CrowdAI datasets, VectorLLM significantly outperformed the previous SOTA methods by 5.6 AP, 7.1 AP, 13.6 AP, respectively in the three datasets. Remarkably, VectorLLM exhibits strong zero-shot performance on unseen objects including aircraft, water bodies, and oil tanks, highlighting its potential for unified modeling of diverse remote sensing object contour extraction tasks. Overall, this work establishes a new paradigm for vector extraction in remote sensing, leveraging the topological reasoning capabilities of LLMs to achieve both high accuracy and exceptional generalization. All the codes and weights will be published for promoting community development.
Abstract (translated)
从遥感影像中自动提取向量化的建筑轮廓对于城市规划、人口估算和灾害评估至关重要。当前最先进的方法依赖于复杂的多阶段流水线，包括像素分割、向量化和多边形优化，这限制了它们的可扩展性和实际应用性。受大型语言模型（LLM）卓越推理能力的启发，我们介绍了VectorLLM，这是首个专为从遥感图像中提取规则建筑轮廓而设计的多模态大型语言模型（MLLM）。与现有方法不同的是，VectorLLM直接进行逐角点回归以生成建筑轮廓，模仿人类标注者的标注过程。我们的架构包括视觉基础骨干网络、MLP连接器和一个增强了可学习位置嵌入的LLM，从而提高了空间理解能力。 通过全面探索包括预训练、监督微调和偏好优化在内的多种训练策略，VectorLLM在WHU、WHU-Mix和CrowdAI数据集上分别比之前的最先进方法（SOTA）高出5.6 AP、7.1 AP和13.6 AP的性能。值得注意的是，VectorLLM在包括飞机、水面和油罐在内的未见过物体上表现出强大的零样本性能，突显了其在统一建模各种遥感目标轮廓提取任务中的潜力。 总体而言，这项工作为遥感领域向量抽取建立了一种新的范式，通过利用大型语言模型的拓扑推理能力，在高精度的同时实现了出色的泛化效果。所有代码和权重将发布以促进社区发展。
URL
https://arxiv.org/abs/2507.04664
