Abstract
Automated building facade inspection is a critical component of urban resilience and smart city maintenance. Traditionally, this field has relied on specialized discriminative models (e.g., YOLO, Mask R-CNN) that excel at pixel-level localization but are constrained to passive perception and worse generization without the visual understandng to interpret structural topology. Large Multimodal Models (LMMs) promise a paradigm shift toward active reasoning, yet their application in such high-stakes engineering domains lacks rigorous evaluation standards. To bridge this gap, we introduce a human-in-the-loop semi-automated annotation framework, leveraging expert-proposal verification to unify 12 fragmented datasets into a standardized, hierarchical ontology. Building on this foundation, we present \textit{DefectBench}, the first multi-dimensional benchmark designed to interrogate LMMs beyond basic semantic recognition. \textit{DefectBench} evaluates 18 state-of-the-art (SOTA) LMMs across three escalating cognitive dimensions: Semantic Perception, Spatial Localization, and Generative Geometry Segmentation. Extensive experiments reveal that while current LMMs demonstrate exceptional topological awareness and semantic understanding (effectively diagnosing "what" and "how"), they exhibit significant deficiencies in metric localization precision ("where"). Crucially, however, we validate the viability of zero-shot generative segmentation, showing that general-purpose foundation models can rival specialized supervised networks without domain-specific training. This work provides both a rigorous benchmarking standard and a high-quality open-source database, establishing a new baseline for the advancement of autonomous AI agents in civil engineering.
Abstract (translated)
自动化建筑外立面检测是城市韧性与智慧城市维护的关键环节。传统方法依赖专用判别模型（如YOLO、Mask R-CNN），虽擅长像素级定位，却受限于被动感知能力，且因缺乏对结构拓扑的视觉理解而泛化能力不足。大型多模态模型（LMMs）有望推动向主动推理的范式转变，但其在高风险工程领域的应用尚缺乏严谨的评估标准。为填补这一空白，我们提出了人机协同的半自动标注框架，通过专家提案验证机制，将12个分散数据集整合为标准化分层本体。基于此基础，我们发布首个多维基准测试\textit{DefectBench}，旨在超越基础语义识别，深入检验LMMs的认知能力。该基准从三个递进认知维度评估18个前沿LMMs：语义感知、空间定位与几何生成分割。大量实验表明，当前LMMs虽展现出卓越的拓扑意识与语义理解能力（能有效诊断"何类缺陷"与"如何形成"），却在度量级定位精度（"何处缺陷"）方面存在显著不足。尤为关键的是，我们验证了零样本几何生成分割的可行性——证明通用基础模型无需领域特定训练即可媲美专业监督网络。本研究既提供了严谨的基准测试标准与高质量开源数据库，也为自主AI智能体在土木工程领域的进步确立了新基线。
URL
https://arxiv.org/abs/2603.20148
