Abstract
This paper addresses a challenging question: How can we efficiently create high-quality, wide-scope 3D scenes from a single arbitrary image? Existing methods face several constraints, such as requiring multi-view data, time-consuming per-scene optimization, low visual quality in backgrounds, and distorted reconstructions in unseen areas. We propose a novel pipeline to overcome these limitations. Specifically, we introduce a large-scale reconstruction model that uses latents from a video diffusion model to predict 3D Gaussian Splattings for the scenes in a feed-forward manner. The video diffusion model is designed to create videos precisely following specified camera trajectories, allowing it to generate compressed video latents that contain multi-view information while maintaining 3D consistency. We train the 3D reconstruction model to operate on the video latent space with a progressive training strategy, enabling the efficient generation of high-quality, wide-scope, and generic 3D scenes. Extensive evaluations across various datasets demonstrate that our model significantly outperforms existing methods for single-view 3D scene generation, particularly with out-of-domain images. For the first time, we demonstrate that a 3D reconstruction model can be effectively built upon the latent space of a diffusion model to realize efficient 3D scene generation.
Abstract (translated)
本文探讨了一个具有挑战性的问题：如何从单张任意图像高效地创建高质量、宽范围的3D场景？现有的方法面临诸多限制，如需要多视角数据、耗时的逐个场景优化、背景视觉质量低以及未见区域重建失真。我们提出了一种全新的流水线来克服这些限制。具体而言，我们引入了一个大规模重构模型，该模型利用视频扩散模型中的潜变量以前馈方式预测场景的3D高斯散射点。该视频扩散模型设计为能够根据指定的相机轨迹精确生成视频，使其能够在保持3D一致性的同时生成包含多视角信息的压缩视频潜变量。我们采用渐进训练策略对3D重构模型进行训练，使它能在视频潜变量空间中高效地生成高质量、宽范围和通用的3D场景。通过在各种数据集上的广泛评估表明，我们的模型显著优于现有的单视图3D场景生成方法，尤其是在处理域外图像时表现突出。首次证明了可以在扩散模型的潜在空间上有效地构建3D重构模型以实现高效的3D场景生成。
URL
https://arxiv.org/abs/2412.12091
