Abstract
Reconstructing semantic-aware 3D scenes from sparse views is a challenging yet essential research direction, driven by the demands of emerging applications such as virtual reality and embodied AI. Existing per-scene optimization methods require dense input views and incur high computational costs, while generalizable approaches often struggle to reconstruct regions outside the input view cone. In this paper, we propose OGGSplat, an open Gaussian growing method that expands the field-of-view in generalizable 3D reconstruction. Our key insight is that the semantic attributes of open Gaussians provide strong priors for image extrapolation, enabling both semantic consistency and visual plausibility. Specifically, once open Gaussians are initialized from sparse views, we introduce an RGB-semantic consistent inpainting module applied to selected rendered views. This module enforces bidirectional control between an image diffusion model and a semantic diffusion model. The inpainted regions are then lifted back into 3D space for efficient and progressive Gaussian parameter optimization. To evaluate our method, we establish a Gaussian Outpainting (GO) benchmark that assesses both semantic and generative quality of reconstructed open-vocabulary scenes. OGGSplat also demonstrates promising semantic-aware scene reconstruction capabilities when provided with two view images captured directly from a smartphone camera.
Abstract (translated)
从稀疏视角重建语义感知的3D场景是一项具有挑战性但又至关重要的研究方向，这一方向受到了虚拟现实和具身AI等新兴应用的需求推动。现有的逐场景优化方法需要密集的输入视图并且计算成本高昂，而通用化的方法则往往难以重构输入视锥之外的区域。在本文中，我们提出了OGGSplat（开放高斯增长法），这是一种用于广义3D重建中的视野扩展的开放方法。我们的关键见解是：开放高斯的语义属性为图像外推提供了强有力的先验信息，这使得在保持语义一致性和视觉合理性的同时能够进行有效重构。具体来说，在从稀疏视图初始化开放高斯后，我们引入了一个RGB-语义一致性修复模块应用于选定的渲染视图中。这个模块强制执行了图像扩散模型和语义扩散模型之间的双向控制机制。然后将修补后的区域重新投射回3D空间，并进行高效且渐进式的高斯参数优化。 为了评估我们的方法，我们建立了一个高斯外推（GO）基准测试，以衡量重建开放词汇场景的语义质量和生成质量。OGGSplat还展示了当提供两张从智能手机摄像头直接捕获的视图图像时，在具有感知意识场景重构方面表现出巨大的潜力。
URL
https://arxiv.org/abs/2506.05204
