Abstract
Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.
Abstract (translated)
开放词汇3D语义分割旨在对训练集之外的任意类别进行分割。现有方法主要依赖从2D开放词汇模型中蒸馏知识，然而将3D特征对齐到2D表示空间会限制3D几何本质学习，并继承2D预测的误差。为应对这些局限，我们提出GeoGuide新颖框架，利用预训练3D模型整合分层几何-语义一致性以实现开放词汇3D分割。具体而言，我们引入基于不确定性的超点蒸馏模块，融合几何与语义特征以估计逐点不确定性，在超点内自适应加权2D特征，在抑制噪声的同时保留判别信息以增强局部语义一致性。此外，我们的实例级掩码重建模块利用几何先验，通过重建完整实例掩码来强化实例内语义一致性。同时，实例间关系一致性模块对齐几何与语义相似度矩阵，对同类物体的跨实例一致性进行校准，缓解视角引起的语义漂移。在ScanNet v2、Matterport3D和nuScenes上的大量实验验证了GeoGuide的优越性能。
URL
https://arxiv.org/abs/2603.26260
