Abstract
Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control.
Abstract (translated)
场景一致的视频生成旨在基于相机轨迹探索三维场景并创建相应的视频。以往的方法依赖于使用外部内存来保持一致性或迭代地进行3D重建和图像修复，但这些方法在推理过程中会因中间输出错误、不可微分过程以及模型分离而积累误差。为克服这些限制，我们引入了“几何作为上下文”（geometry-as-context）的概念。该方法利用自回归相机控制的视频生成模型迭代地完成以下步骤：(1) 估计当前视图所需的三维重建几何形状；(2) 模拟并恢复由3D场景渲染的新视角图像。 在此多任务框架下，我们开发了相机门控注意模块（camera gated attention module），以增强模型有效利用相机姿态的能力。在训练阶段，使用文本上下文来确定生成的输出是几何信息还是RGB图像。为了确保模型在推断过程中可以仅生成RGB输出，我们在交错的文本-图像-几何训练序列中随机丢弃几何上下文。 该方法已在单向和往返轨迹的场景视频生成上进行了测试，并且结果表明其在保持场景一致性和相机控制方面优于先前的方法。
URL
https://arxiv.org/abs/2602.21929
