Abstract
This paper investigates the potential of enhancing Neural Radiance Fields (NeRF) with semantics to expand their applications. Although NeRF has been proven useful in real-world applications like VR and digital creation, the lack of semantics hinders interaction with objects in complex scenes. We propose to imitate the backbone feature of off-the-shelf perception models to achieve zero-shot semantic segmentation with NeRF. Our framework reformulates the segmentation process by directly rendering semantic features and only applying the decoder from perception models. This eliminates the need for expensive backbones and benefits 3D consistency. Furthermore, we can project the learned semantics onto extracted mesh surfaces for real-time interaction. With the state-of-the-art Segment Anything Model (SAM), our framework accelerates segmentation by 16 times with comparable mask quality. The experimental results demonstrate the efficacy and computational advantages of our approach. Project page: \url{https://me.kiui.moe/san/}.
Abstract (translated)
本论文研究如何将神经网络辐射场(NeRF)与语义增强其应用范围。尽管NeRF在虚拟现实和数字创造等实际应用领域已经被证明有用，但缺乏语义会阻碍复杂场景下与物体的互动。我们提议仿效现有的感知模型的主干特性，以通过直接渲染语义特征来实现NeRF的零次元语义分割。我们的框架重写了分割过程，仅从感知模型中应用解码器，从而消除了昂贵的主干需求并实现了3D一致性。此外，我们可以将学到的语义投影到提取的网格表面，实现实时交互。利用最先进的分割任意模型(SAM)，我们的框架将分割速度提高了16倍，与同等掩模质量相比。实验结果证明了我们方法的有效性和计算优势。项目页面： \url{https://me.kiui.moe/san/}。
URL
https://arxiv.org/abs/2305.16233
