Abstract
Reconstructing complete and interactive 3D scenes remains a fundamental challenge in computer vision and robotics, particularly due to persistent object occlusions and limited sensor coverage. Multiview observations from a single scene scan often fail to capture the full structural details. Existing approaches typically rely on multi stage pipelines, such as segmentation, background completion, and inpainting or require per-object dense scanning, both of which are error-prone, and not easily scalable. We propose IGFuse, a novel framework that reconstructs interactive Gaussian scene by fusing observations from multiple scans, where natural object rearrangement between captures reveal previously occluded regions. Our method constructs segmentation aware Gaussian fields and enforces bi-directional photometric and semantic consistency across scans. To handle spatial misalignments, we introduce a pseudo-intermediate scene state for unified alignment, alongside collaborative co-pruning strategies to refine geometry. IGFuse enables high fidelity rendering and object level scene manipulation without dense observations or complex pipelines. Extensive experiments validate the framework's strong generalization to novel scene configurations, demonstrating its effectiveness for real world 3D reconstruction and real-to-simulation transfer. Our project page is available online.
Abstract (translated)
重建完整且可交互的三维场景仍然是计算机视觉和机器人技术中的一个基本挑战，特别是由于持续的对象遮挡和有限的传感器覆盖范围。来自单次场景扫描的多视角观测往往无法捕捉到完整的结构细节。现有方法通常依赖于分阶段的工作流程，如分割、背景补全和像素级修复，或者需要对每个对象进行密集扫描，这些方法都容易出错且不易扩展。我们提出了IGFuse，一种新的框架，通过融合来自多次扫描的观察结果来重建交互式高斯场景，其中自然物体在捕获之间的重新排列会揭示之前被遮挡的区域。我们的方法构建了具有分割感知能力的高斯场，并强制执行跨扫描的双向光度和语义一致性。为了解决空间对齐问题，我们引入了一个伪中间场景状态以实现统一的对齐，并提出了协作共修剪策略来细化几何结构。IGFuse能够在没有密集观测或复杂管道的情况下提供高质量的渲染和对象级别的场景操作。广泛的实验验证了该框架在新的场景配置中的强大泛化能力，证明其在现实世界的三维重建和从真实到模拟转换方面的有效性。我们的项目页面在线可用。
URL
https://arxiv.org/abs/2508.13153
