Abstract
Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural radiance fields (NeRFs) may allow better scalability through the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely collinear camera motions and sparser samplings at higher speeds. On the other hand, the application often demands rendering from camera views that deviate from the inputs to accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes. First, our framework learns a geometric scene representation from Lidar, which is fused with the implicit grid-based representation for radiance decoding, thereby supplying stronger geometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views from Lidar points for further improvement. Our insights translate to largely improved novel view synthesis under real driving scenes.
Abstract (translated)
照片现实模拟在自动驾驶等应用中扮演着关键角色，因为神经辐射场（NeRFs）的进步可能允许通过自动创建数字3D资产来实现更好的可扩展性。然而，在街景中，由于主要是平行的相机运动和高速时的采样稀疏，重建质量下降。另一方面，应用程序通常要求从相机视角进行渲染，以准确模拟行为，如变道。在本文中，我们提出了几个见解，使得Lidar数据能够更好地用于改善街景中的NeRF质量。首先，我们的框架从Lidar中学习几何场景表示，并将其与隐式网格表示的辐射解码相结合，从而提供来自明确点云的更强的几何信息。其次，我们提出了一个鲁棒的可视化深度监督方案，允许通过累积使用密集的Lidar点。第三，我们从Lidar点生成增强的训练视图，以进一步改进。我们的见解使得在现实驾驶场景中产生了显著改进的新视图合成。
URL
https://arxiv.org/abs/2405.00900
