Abstract
Reconstructing 3D scenes from a single image is a fundamentally ill-posed task due to the severely under-constrained nature of the problem. Consequently, when the scene is rendered from novel camera views, existing single image to 3D reconstruction methods render incoherent and blurry views. This problem is exacerbated when the unseen regions are far away from the input camera. In this work, we address these inherent limitations in existing single image-to-3D scene feedforward networks. To alleviate the poor performance due to insufficient information beyond the input image's view, we leverage a strong generative prior in the form of a pre-trained latent video diffusion model, for iterative refinement of a coarse scene represented by optimizable Gaussian parameters. To ensure that the style and texture of the generated images align with that of the input image, we incorporate on-the-fly Fourier-style transfer between the generated images and the input image. Additionally, we design a semantic uncertainty quantification module that calculates the per-pixel entropy and yields uncertainty maps used to guide the refinement process from the most confident pixels while discarding the remaining highly uncertain ones. We conduct extensive experiments on real-world scene datasets, including in-domain RealEstate-10K and out-of-domain KITTI-v2, showing that our approach can provide more realistic and high-fidelity novel view synthesis results compared to existing state-of-the-art methods.
Abstract (translated)
从单张图像重建三维场景是一个本质上未定义的问题，因为该问题的约束条件极为不足。因此，当使用现有的一图转三维重构方法从新颖视角渲染场景时，生成的画面通常是不连贯且模糊不清的，尤其是对于远离输入图像视点的未见区域而言，这一问题更为严重。在这项工作中，我们针对现有单张图像到三维场景前馈网络中存在的固有限制进行了改进。 为了缓解由于缺乏超出输入图像视野的信息而导致的表现不佳的问题，我们利用了一种强大的生成先验知识的形式——预训练的潜在视频扩散模型，以迭代的方式优化粗略场景表示中的可调优高斯参数。为了确保生成图像的风格和纹理与输入图像一致，我们在生成的图像与输入图像之间进行了实时傅里叶样式传输。 此外，我们设计了一个语义不确定性量化模块，该模块计算每个像素的熵值，并据此产生用于指导从最自信像素开始优化过程的同时排除高度不确定像素的不确定性地图。 通过在包括领域内的RealEstate-10K和跨领域的KITTI-v2在内的真实场景数据集上进行广泛实验，我们展示了我们的方法能够提供比现有最先进的技术更逼真且高质量的新视角合成结果。
URL
https://arxiv.org/abs/2503.15742
