Abstract
This paper studies the problem of generating an unbounded dynamic scene from a single view, which has wide applications in augmented/virtual reality and robotics. Since the scene is changing over time, different generated views need to be consistent with the underlying 3D motions. While previous works learn such consistency by training from multiple views, the generated scene regions are bounded to be close to the training views with limited camera movements. To address this issue, we propose DynamicVoyager that reformulates the dynamic scene generation as a scene outpainting process for new dynamic content. As 2D outpainting models can hardly generate 3D consistent motions from only 2D pixels at a single view, we consider pixels as rays to enrich the pixel input with the ray context, so that the 3D motion consistency can be learned from the ray information. More specifically, we first map the single-view video input to a dynamic point cloud with the estimated video depths. Then we render the partial video at a novel view and outpaint the video with ray contexts from the point cloud to generate 3D consistent motions. We employ the outpainted video to update the point cloud, which is used for scene outpainting from future novel views. Experiments show that our model is able to generate unbounded scenes with consistent motions along fly-through cameras, and the generated contents can be controlled with scene prompts.
Abstract (translated)
本文研究了从单视图生成无边界动态场景的问题，该问题在增强/虚拟现实和机器人技术中具有广泛的应用。由于场景随时间变化，不同生成的视图需要与底层3D运动保持一致。虽然先前的工作通过多视角训练来学习这种一致性，但生成的场景区域被限制在靠近训练视图且相机移动有限的位置上。为了解决这个问题，我们提出了DynamicVoyager，它将动态场景生成重新表述为新动态内容的场景外推过程（scene outpainting）。由于2D外推模型难以仅从单视图的2D像素中生成3D一致的运动，我们将像素视为射线，并通过添加射线上下文来丰富像素输入，从而可以从射线信息中学到3D运动一致性。具体来说，我们首先将单视角视频输入映射为动态点云，使用估计的视频深度信息进行映射。然后，在新视图中渲染部分视频，并利用点云中的射线上下文外推视频以生成一致的3D运动。我们采用外推后的视频来更新点云，并用于从未来的新的视图中进行场景外推。实验表明，我们的模型能够生成沿飞行穿越相机路径具有连续动作的无边界场景，并且可以使用场景提示控制生成的内容。
URL
https://arxiv.org/abs/2507.04183
