Abstract
Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose \textbf{ViewCrafter}, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.
Abstract (translated)
尽管神经3D重建领域最近取得了进展，但依赖密集多视角捕捉的限制使得它们的适用范围受到限制。在这项工作中，我们提出了ViewCrafter，一种新方法，用于从单张或多张图像中合成具有视频扩散模型预测的高保真度的全新场景视角。我们的方法利用了视频扩散模型的强大生成能力以及点基础表示中提供的粗3D线索，通过精确的相机姿态控制生成高质量的视频帧。为了进一步扩展生成范围的新视角，我们共同优化了一个迭代式视图合成策略和一个相机轨迹规划算法，以逐步扩展3D线索和 novel views 的覆盖范围。通过ViewCrafter，我们可以促进各种应用，例如通过有效地优化通过重构3D点和生成的新视角来实现的虚拟现实体验，以及用于更有想象力的内容创作的场景级文本到3D生成。在多样数据集上的广泛实验证明了我们方法在生成高保真度和一致性新视角方面的优异性能和强大的泛化能力。
URL
https://arxiv.org/abs/2409.02048
