Abstract
Current methods for 3D semantic segmentation propose training models with limited annotations to address the difficulty of annotating large, irregular, and unordered 3D point cloud data. They usually focus on the 3D domain only, without leveraging the complementary nature of 2D and 3D data. Besides, some methods extend original labels or generate pseudo labels to guide the training, but they often fail to fully use these labels or address the noise within them. Meanwhile, the emergence of comprehensive and adaptable foundation models has offered effective solutions for segmenting 2D data. Leveraging this advancement, we present a novel approach that maximizes the utility of sparsely available 3D annotations by incorporating segmentation masks generated by 2D foundation models. We further propagate the 2D segmentation masks into the 3D space by establishing geometric correspondences between 3D scenes and 2D views. We extend the highly sparse annotations to encompass the areas delineated by 3D masks, thereby substantially augmenting the pool of available labels. Furthermore, we apply confidence- and uncertainty-based consistency regularization on augmentations of the 3D point cloud and select the reliable pseudo labels, which are further spread on the 3D masks to generate more labels. This innovative strategy bridges the gap between limited 3D annotations and the powerful capabilities of 2D foundation models, ultimately improving the performance of 3D weakly supervised segmentation.
Abstract (translated)
当前的三维语义分割方法通过使用有限注释的数据来训练模型，以解决对大量、不规则且无序的三维点云数据进行标注的困难。这些方法通常只专注于三维领域，而没有利用二维和三维数据之间的互补性。此外，一些方法扩展原始标签或生成伪标签以指导训练，但它们往往未能充分利用这些标签或有效处理其中的噪声。与此同时，全面且适应性强的基础模型的出现为分割二维数据提供了有效的解决方案。借助这一进展，我们提出了一种新方法，通过引入由二维基础模型生成的分割掩码来最大化稀疏可用三维注释的效用。进一步地，我们将二维分割掩码传播到三维空间中，通过在三维场景和二维视图之间建立几何对应关系实现这一点。我们将高度稀疏的标注扩展到由三维掩码界定的区域，从而大幅增加了可用标签的数量。此外，我们对三维点云的数据增强应用了基于置信度和不确定性的一致性正则化，并选择可靠的伪标签，这些伪标签进一步被传播在三维掩码上以生成更多标签。这一创新策略弥合了有限的三维注释与二维基础模型的强大能力之间的差距，最终提升了弱监督下三维语义分割的表现。
URL
https://arxiv.org/abs/2508.19909
