Abstract
In this paper, we aim to learn a semantic radiance field from multiple scenes that is accurate, efficient and generalizable. While most existing NeRFs target at the tasks of neural scene rendering, image synthesis and multi-view reconstruction, there are a few attempts such as Semantic-NeRF that explore to learn high-level semantic understanding with the NeRF structure. However, Semantic-NeRF simultaneously learns color and semantic label from a single ray with multiple heads, where the single ray fails to provide rich semantic information. As a result, Semantic NeRF relies on positional encoding and needs to train one specific model for each scene. To address this, we propose Semantic Ray (S-Ray) to fully exploit semantic information along the ray direction from its multi-view reprojections. As directly performing dense attention over multi-view reprojected rays would suffer from heavy computational cost, we design a Cross-Reprojection Attention module with consecutive intra-view radial and cross-view sparse attentions, which decomposes contextual information along reprojected rays and cross multiple views and then collects dense connections by stacking the modules. Experiments show that our S-Ray is able to learn from multiple scenes, and it presents strong generalization ability to adapt to unseen scenes.
Abstract (translated)
在本文中，我们旨在从多个场景中提取准确的、高效的、可泛化的语义亮度场。虽然大多数现有的NeRF目标是基于神经网络场景渲染、图像合成和多视角重构的任务，但也有一些尝试，例如语义NeRF，探索使用NeRF结构学习高级别的语义理解。然而，语义NeRF同时从具有多个头的光线中提取颜色和语义标签，其中单个光线无法提供丰富的语义信息。因此，NeRF依赖于位置编码，需要为每个场景训练一个特定的模型。为了解决此问题，我们提出了语义射线(S-Ray)，以充分利用沿着射线方向的信息，从多视角重投影中获取。由于直接在重投影射线上执行密集关注将会面临高昂的计算成本，我们设计了一个交叉重投影注意力模块，其中包括连续内视角放射线和交叉视角稀疏关注，该模块分解了重投影射线和交叉多视角中的上下文信息，然后通过堆叠模块收集密集连接。实验结果表明，我们的S-Ray可以从多个场景学习，并表现出强大的适应未知场景的能力。
URL
https://arxiv.org/abs/2303.13014
