Abstract
3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) have advanced novel-view synthesis. Recent methods extend multi-view 2D segmentation to 3D, enabling instance/semantic segmentation for better scene understanding. A key challenge is the inconsistency of 2D instance labels across views, leading to poor 3D predictions. Existing methods use a two-stage approach in which some rely on contrastive learning with hyperparameter-sensitive clustering, while others preprocess labels for consistency. We propose a unified framework that merges these steps, reducing training time and improving performance by introducing a learnable feature embedding for segmentation in Gaussian primitives. This embedding is then efficiently decoded into instance labels through a novel "Embedding-to-Label" process, effectively integrating the optimization. While this unified framework offers substantial benefits, we observed artifacts at the object boundaries. To address the object boundary issues, we propose hard-mining samples along these boundaries. However, directly applying hard mining to the feature embeddings proved unstable. Therefore, we apply a linear layer to the rasterized feature embeddings before calculating the triplet loss, which stabilizes training and significantly improves performance. Our method outperforms baselines qualitatively and quantitatively on the ScanNet, Replica3D, and Messy-Rooms datasets.
Abstract (translated)
3D高斯点置（3D Gaussian Splatting，简称3DGS）和神经辐射场（Neural Radiance Fields，简称NeRF）在新颖视角合成方面取得了进展。最近的方法将多视图2D分割扩展到3D空间中，从而能够进行实例/语义分割，以实现更好的场景理解。一个关键挑战是不同视角之间2D实例标签的一致性问题，这会导致较差的3D预测效果。现有的方法采用两阶段方法，其中一些依赖于敏感于超参数设置的对比学习聚类，而另一些则对标签进行预处理以提高一致性。我们提出了一种统一框架，将这些步骤合并在一起，并通过引入高斯原语中的可学习特征嵌入来实现分割功能，从而减少训练时间并提升性能表现。该嵌入随后通过新颖的“Embedding-to-Label”过程被高效地解码为实例标签，有效集成了优化流程。 尽管这种统一框架带来了显著的好处，但我们观察到在对象边界处存在一些伪影问题。为了应对这些问题，我们提出了一种沿这些边界进行困难样本挖掘的方法。然而，直接将困难采样应用到特征嵌入上显示出不稳定性。因此，在计算三元组损失之前，我们在栅格化后的特征嵌入上添加了一个线性层，这有助于稳定训练过程并显著提升了性能表现。 我们的方法在ScanNet、Replica3D和Messy-Rooms数据集上的定性和定量评估中均优于基准方法。
URL
https://arxiv.org/abs/2512.24763
