Abstract
Advancements in 3D instance segmentation have traditionally been tethered to the availability of annotated datasets, limiting their application to a narrow spectrum of object categories. Recent efforts have sought to harness vision-language models like CLIP for open-set semantic reasoning, yet these methods struggle to distinguish between objects of the same categories and rely on specific prompts that are not universally applicable. In this paper, we introduce SAI3D, a novel zero-shot 3D instance segmentation approach that synergistically leverages geometric priors and semantic cues derived from Segment Anything Model (SAM). Our method partitions a 3D scene into geometric primitives, which are then progressively merged into 3D instance segmentations that are consistent with the multi-view SAM masks. Moreover, we design a hierarchical region-growing algorithm with a dynamic thresholding mechanism, which largely improves the robustness of finegrained 3D scene parsing. Empirical evaluations on Scan-Net and the more challenging ScanNet++ datasets demonstrate the superiority of our approach. Notably, SAI3D outperforms existing open-vocabulary baselines and even surpasses fully-supervised methods in class-agnostic segmentation on ScanNet++.
Abstract (translated)
传统的3D实例分割进展通常与已标注的数据集的可用性相关，限制了其应用范围局限于少数物体类别。最近的努力试图利用像CLIP这样的视觉语言模型进行开放式语义推理，然而这些方法很难区分同一类别的物体，并依赖于不适用于所有任务的特定提示。在本文中，我们介绍了SAI3D，一种新颖的零散3D实例分割方法，它通过协同利用基于Segment Anything Model（SAM）生成的几何先验和语义线索来取得成功。我们的方法将3D场景分割为几何基本单元，然后将这些单元逐步合并为与多视角SAM掩码一致的3D实例分割。此外，我们还设计了一个具有动态阈值机制的分层区域生长算法，大大提高了细粒度3D场景解析的鲁棒性。在ScanNet和更具挑战性的ScanNet+ datasets上的实证评估表明，我们的方法具有优越性。值得注意的是，SAI3D在ScanNet和ScanNet+ datasets上优于现有开发生词基准，甚至超过了完全监督方法。
URL
https://arxiv.org/abs/2312.11557
