Abstract
Synthesizing novel views of large-scale scenes from unconstrained in-the-wild images is an important but challenging task in computer vision. Existing methods, which optimize per-image appearance and transient occlusion through implicit neural networks from dense training views (approximately 1000 images), struggle to perform effectively under sparse input conditions, resulting in noticeable artifacts. To this end, we propose SparseGS-W, a novel framework based on 3D Gaussian Splatting that enables the reconstruction of complex outdoor scenes and handles occlusions and appearance changes with as few as five training images. We leverage geometric priors and constrained diffusion priors to compensate for the lack of multi-view information from extremely sparse input. Specifically, we propose a plug-and-play Constrained Novel-View Enhancement module to iteratively improve the quality of rendered novel views during the Gaussian optimization process. Furthermore, we propose an Occlusion Handling module, which flexibly removes occlusions utilizing the inherent high-quality inpainting capability of constrained diffusion priors. Both modules are capable of extracting appearance features from any user-provided reference image, enabling flexible modeling of illumination-consistent scenes. Extensive experiments on the PhotoTourism and Tanks and Temples datasets demonstrate that SparseGS-W achieves state-of-the-art performance not only in full-reference metrics, but also in commonly used non-reference metrics such as FID, ClipIQA, and MUSIQ.
Abstract (translated)
合成大规模场景的新视角是从不受约束的野外图像中一个重要的但具有挑战性的任务，在计算机视觉领域。现有的方法通过隐式神经网络从密集训练视图（约1000张图片）来优化每个图像的外观和瞬时遮挡，但在稀疏输入条件下难以有效运行，导致明显的伪影出现。 为此，我们提出了SparseGS-W框架，这是一个基于3D高斯点阵的新方法，它能够仅使用五张训练图像就重建复杂的室外场景，并处理遮挡和外观变化。我们利用了几何先验知识和受限制的扩散先验来补偿极度稀疏输入中多视角信息的不足。 具体来说，我们提出了一种即插即用的Constrained Novel-View Enhancement模块，在高斯优化过程中迭代提高渲染的新视图质量。此外，我们还提出了一个Occlusion Handling模块，它灵活地利用受限制扩散先验固有的高质量填图能力来去除遮挡。 两个模块都能够从任何用户提供的参考图像中提取外观特征，使光照一致的场景建模更加灵活。在PhotoTourism和Tanks and Temples数据集上的大量实验表明，SparseGS-W不仅在全参照度量标准上取得了最先进的性能，在常用的无参照度量标准（如FID、ClipIQA和MUSIQ）上也表现出色。
URL
https://arxiv.org/abs/2503.19452
