Abstract
Building an efficient and physically consistent world model from limited observations is a long standing challenge in vision and robotics. Many existing world modeling pipelines are based on implicit generative models, which are hard to train and often lack 3D or physical consistency. On the other hand, explicit 3D methods built from a single state often require multi-stage processing-such as segmentation, background completion, and inpainting-due to occlusions. To address this, we leverage two perturbed observations of the same scene under different object configurations. These dual states offer complementary visibility, alleviating occlusion issues during state transitions and enabling more stable and complete reconstruction. In this paper, we present DSG-World, a novel end-to-end framework that explicitly constructs a 3D Gaussian World model from Dual State observations. Our approach builds dual segmentation-aware Gaussian fields and enforces bidirectional photometric and semantic consistency. We further introduce a pseudo intermediate state for symmetric alignment and design collaborative co-pruning trategies to refine geometric completeness. DSG-World enables efficient real-to-simulation transfer purely in the explicit Gaussian representation space, supporting high-fidelity rendering and object-level scene manipulation without relying on dense observations or multi-stage pipelines. Extensive experiments demonstrate strong generalization to novel views and scene states, highlighting the effectiveness of our approach for real-world 3D reconstruction and simulation.
Abstract (translated)
从有限的观察构建一个高效且物理上一致的世界模型，一直是视觉和机器人学领域长期面临的挑战。许多现有的世界建模流程依赖于隐式生成模型，这些模型难以训练并且通常缺乏三维或物理上的一致性。另一方面，基于单一状态构建的显式3D方法由于遮挡问题往往需要多阶段处理——如分割、背景补全以及修复缺失信息等步骤。为了解决这些问题，我们利用了同一场景在不同对象配置下的两个扰动观察结果。这些双重状态提供了互补的可见性，缓解了状态转换期间的遮挡问题，并促进了更稳定和完整的重建。 在这篇论文中，我们提出了一种名为DSG-World的新颖端到端框架，该框架从双状态观测数据显式构造了一个三维高斯世界模型（3D Gaussian World Model）。我们的方法构建了双重分割感知高斯场，并强制执行双向光度学和语义一致性。此外，我们还引入了伪中间态用于对称校准，并设计了一种协作共修剪策略以优化几何完整性。 DSG-World能够在仅使用显式高斯表示空间的情况下实现高效的真实到模拟转换，支持高质量渲染以及对象级别的场景操作，而无需依赖密集观察数据或多阶段流程。大量的实验展示了其在新视角和场景状态下的强大泛化能力，突显了我们方法对于真实世界3D重建与模拟的有效性。
URL
https://arxiv.org/abs/2506.05217
