Abstract
Neural Radiance Fields (NeRF) have shown impressive results in 3D reconstruction and generating novel views. A key challenge within NeRF is the editing of reconstructed scenes, such as object removal, which requires maintaining consistency across multiple views and ensuring high-quality synthesised perspectives. Previous studies have incorporated depth priors, typically from LiDAR or sparse depth measurements provided by COLMAP, to improve the performance of object removal in NeRF. However, these methods are either costly or time-consuming. In this paper, we propose a novel approach that integrates monocular depth estimates with NeRF-based object removal models to significantly reduce time consumption and enhance the robustness and quality of scene generation and object removal. We conducted a thorough evaluation of COLMAP's dense depth reconstruction on the KITTI dataset to verify its accuracy in depth map generation. Our findings suggest that COLMAP can serve as an effective alternative to a ground truth depth map where such information is missing or costly to obtain. Additionally, we integrated various monocular depth estimation methods into the removal NeRF model, i.e., SpinNeRF, to assess their capacity to improve object removal performance. Our experimental results highlight the potential of monocular depth estimation to substantially improve NeRF applications.
Abstract (translated)
Neural Radiance Fields (NeRF) 在 3D 重建和生成新视图方面已经取得了令人印象深刻的成果。 NeRF 中的关键挑战之一是编辑重构场景，例如物体移除，这需要在多个视图中保持一致并确保高质合成视角。之前的研究已经利用深度优先项，通常来自 LiDAR 或稀疏深度测量提供的 COLMAP，来提高 NeRF 中物体移除的性能。然而，这些方法要么代价高昂，要么费时。在本文中，我们提出了一种新方法，将单目深度估计与基于 NeRF 的物体移除模型相结合，显著减少了时间消耗，并提高了场景生成和物体移除的稳健性和质量。我们对 COLMAP 在 KITTI 数据集上的密集深度重建进行了详细的评估，以验证其深度图生成的准确性。我们的研究结果表明，COLMAP 可以作为当深度图缺失或昂贵无法获得时的有效地面真值深度图的替代。此外，我们将各种单目深度估计方法（例如 SpinNeRF）集成到移除 NeRF 模型中，以评估它们提高物体移除性能的能力。我们的实验结果突出了单目深度估计在极大地改善 NeRF 应用中的潜力。
URL
https://arxiv.org/abs/2405.00630
