Abstract
Despite recent advances in leveraging generative prior from pre-trained diffusion models for 3D scene reconstruction, existing methods still face two critical limitations. First, due to the lack of reliable geometric supervision, they struggle to produce high-quality reconstructions even in observed regions, let alone in unobserved areas. Second, they lack effective mechanisms to mitigate multi-view inconsistencies in the generated images, leading to severe shape-appearance ambiguities and degraded scene geometry. In this paper, we identify accurate geometry as the fundamental prerequisite for effectively exploiting generative models to enhance 3D scene reconstruction. We first propose to leverage the prevalence of planar structures to derive accurate metric-scale depth maps, providing reliable supervision in both observed and unobserved regions. Furthermore, we incorporate this geometry guidance throughout the generative pipeline to improve visibility mask estimation, guide novel view selection, and enhance multi-view consistency when inpainting with video diffusion models, resulting in accurate and consistent scene completion. Extensive experiments on Replica, ScanNet++, and DeepBlending show that our method consistently outperforms existing baselines in both geometry and appearance reconstruction, particularly for unobserved regions. Moreover, our method naturally supports single-view inputs and unposed videos, with strong generalizability in both indoor and outdoor scenarios with practical real-world applicability. The project page is available at this https URL.
Abstract (translated)
尽管近期利用预训练扩散模型的生成先验进行3D场景重建取得了进展，现有方法仍面临两个关键限制。首先，由于缺乏可靠的几何监督，它们在已观察区域难以产生高质量的重建效果，更不用说未观察到的区域了。其次，这些方法缺少有效机制来减少多视角图像中的不一致性问题，导致严重的形状-外观模糊性和场景几何结构退化。 在这篇论文中，我们识别出准确的几何信息是利用生成模型增强3D场景重建的关键前提条件。首先，我们提出利用平面结构的普遍性推导出精确的度量级深度图，这为已观察和未观察区域提供了可靠的监督。此外，我们将这种几何指导贯穿于整个生成管道中，以改进可见性掩码估计、引导新颖视角选择，并在使用视频扩散模型进行填补时提高多视图一致性，从而实现准确且一致的场景完成。 我们在Replica、ScanNet++ 和 DeepBlending 数据集上的大量实验表明，我们的方法在几何和外观重建方面始终优于现有的基准方法，特别是在未观察区域。此外，我们的方法自然支持单视角输入及未经定位处理的视频，并具有在室内和室外实际应用场景中强大泛化的特性。 该项目页面可在此 [URL] 访问。
URL
https://arxiv.org/abs/2510.12099
