Abstract
The ability to generate virtual environments is crucial for applications ranging from gaming to physical AI domains such as robotics, autonomous driving, and industrial AI. Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data, which is not always readily available. Recent advancements in video diffusion models have shown remarkable imagination capabilities, yet their 2D nature limits the applications to simulation where a robot needs to navigate and interact with the environment. In this paper, we propose a self-distillation framework that aims to distill the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for multi-view training data. Specifically, we augment the typical RGB decoder with a 3DGS decoder, which is supervised by the output of the RGB decoder. In this approach, the 3DGS decoder can be purely trained with synthetic data generated by video diffusion models. At inference time, our model can synthesize 3D scenes from either a text prompt or a single image for real-time rendering. Our framework further extends to dynamic 3D scene generation from a monocular input video. Experimental results show that our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.
Abstract (translated)
生成虚拟环境的能力对于从游戏到物理AI领域（如机器人、自动驾驶和工业AI）的应用至关重要。目前基于学习的三维重建方法依赖于捕捉到的真实世界多视角数据，而这种数据并不总是易于获取。最近视频扩散模型的进步展示了令人印象深刻的想象能力，但其二维特性限制了在需要机器人导航与环境互动等模拟场景中的应用。在这篇论文中，我们提出了一种自我蒸馏框架，旨在将视频扩散模型中的隐含三维知识转化为显式的三维高斯点（3DGS）表示形式，从而无需多视角训练数据。具体来说，我们在典型的RGB解码器上添加了一个3DGS解码器，并通过RGB解码器的输出对其进行监督。在此方法中，3DGS解码器可以完全依赖于视频扩散模型生成的合成数据进行训练。在推理时，我们的模型可以从文本提示或单张图像实时渲染三维场景。此外，该框架还可以扩展到从单一视角输入视频生成动态三维场景。实验结果表明，我们提出的框架在静态和动态三维场景生成方面达到了最先进的性能。
URL
https://arxiv.org/abs/2509.19296
