Abstract
Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.
Abstract (translated)
最近的研究提出将3DGS（三维高斯模型）扩展为使用语义特征向量，以同时进行语义分割和图像渲染。然而，这些方法通常分别处理语义分支和渲染分支，并且主要依赖于2D监督信息，而忽略了3D高斯几何结构。此外，现有的自适应策略仅根据渲染梯度调整高斯集，在细节较少或无纹理的区域中可能效果不足。 在本工作中，我们提出了一种联合增强框架用于3D语义高斯建模，该框架将语义分支和渲染分支协同起来共同工作。首先，不同于传统的点云形状编码方法，我们引入了一个各向异性三维高斯切比雪夫描述符（使用拉普拉斯-贝尔特拉米算子），以捕捉精细的3D形状细节，并区分具有类似外观的对象，同时减少了对潜在噪声较多的2D引导信息的依赖。此外，在不完全依赖于渲染梯度的情况下，我们根据局部语义和形状信号自适应调整高斯分配以及球谐函数，通过选择性资源分配提升渲染效率。 最后，我们采用跨场景知识转移模块来持续更新已学习到的形状模式，这使得在面对新场景时无需从零开始重新学习形状信息即可实现更快收敛和稳健表示。多项数据集上的实验表明，在保持高渲染帧率的同时，我们的方法可以提高分割准确性和渲染质量。
URL
https://arxiv.org/abs/2601.02339
