Abstract
Sparse view NeRF is challenging because limited input images lead to an under constrained optimization problem for volume rendering. Existing methods address this issue by relying on supplementary information, such as depth maps. However, generating this supplementary information accurately remains problematic and often leads to NeRF producing images with undesired artifacts. To address these artifacts and enhance robustness, we propose SSNeRF, a sparse view semi supervised NeRF method based on a teacher student framework. Our key idea is to challenge the NeRF module with progressively severe sparse view degradation while providing high confidence pseudo labels. This approach helps the NeRF model become aware of noise and incomplete information associated with sparse views, thus improving its robustness. The novelty of SSNeRF lies in its sparse view specific augmentations and semi supervised learning mechanism. In this approach, the teacher NeRF generates novel views along with confidence scores, while the student NeRF, perturbed by the augmented input, learns from the high confidence pseudo labels. Our sparse view degradation augmentation progressively injects noise into volume rendering weights, perturbs feature maps in vulnerable layers, and simulates sparse view blurriness. These augmentation strategies force the student NeRF to recognize degradation and produce clearer rendered views. By transferring the student's parameters to the teacher, the teacher gains increased robustness in subsequent training iterations. Extensive experiments demonstrate the effectiveness of our SSNeRF in generating novel views with less sparse view degradation. We will release code upon acceptance.
Abstract (translated)
稀疏视图NeRF具有挑战性，因为有限的输入图像导致体积渲染中的约束优化问题。现有的方法通过依赖补充信息（如深度图）来解决此问题。然而，生成此补充信息准确仍然具有问题，并通常导致NeRF产生具有不希望伪影的图像。为了解决这些伪影并提高稳健性，我们提出了SSNeRF，一种基于教师-学生框架的稀疏视图半监督NeRF方法。我们的关键想法是，在提供高置信度伪标签的同时，逐步挑战NeRF模块，从而改善其稳健性。SSNeRF的创新之处在于其稀疏视图特定的增强和半监督学习机制。在这种方法中，教师NeRF生成新的视图并伴信心分数，而学生NeRF在增强的输入下扰动，从高置信度伪标签中学习。我们的稀疏视图衰减增强策略逐级注入噪声到体积渲染权重中，扰动易受影响的层特征图并模拟稀疏视图模糊。这些增强策略迫使学生NeRF识别衰减并产生更清晰的渲染视图。通过将学生的参数传递给教师，教师在后续训练迭代中获得了更大的稳健性。大量实验证明了我们SSNeRF在生成新颖视图的同时具有较低的稀疏视图衰减的有效性。我们将在审核通过后发布代码。
URL
https://arxiv.org/abs/2408.09144
