Abstract
We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be "inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.
Abstract (translated)
我们研究了从单目视频生成动态场景的新视角合成方法。先前的方法依赖于昂贵的测试时间4D表示优化，或者在前馈训练方式下不能保持场景几何结构。我们的方法基于三个关键见解：(1) 对于在输入和目标视图中都可见的共视像素（covisible pixels），可以通过首先重建动态3D场景然后从新视角进行渲染来生成；(2) 新视角中的隐藏像素可以使用前馈2D视频扩散模型（即填充缺失信息）来“修复”。特别地，我们的视频修复扩散模型(CogNVS)可以从2D视频中自我监督学习，这使我们能够在其上训练大量野外视频数据集。进而使得CogNVS可以通过测试时间微调的方式零样本应用于新的测试视频。(3) 我们通过实证验证发现，CogNVS在从单目视频生成动态场景的新视角合成方面几乎超越了所有先前的方法。
URL
https://arxiv.org/abs/2507.12646
