Abstract
We present CAT4D, a method for creating 4D (dynamic 3D) scenes from monocular video. CAT4D leverages a multi-view video diffusion model trained on a diverse combination of datasets to enable novel view synthesis at any specified camera poses and timestamps. Combined with a novel sampling approach, this model can transform a single monocular video into a multi-view video, enabling robust 4D reconstruction via optimization of a deformable 3D Gaussian representation. We demonstrate competitive performance on novel view synthesis and dynamic scene reconstruction benchmarks, and highlight the creative capabilities for 4D scene generation from real or generated videos. See our project page for results and interactive demos: this https URL.
Abstract (translated)
我们介绍了CAT4D，这是一种从单目视频中创建4D（动态3D）场景的方法。CAT4D利用了一个多视角视频扩散模型，该模型在多样化的数据集组合上进行了训练，能够在任何指定的相机姿态和时间戳下实现新视角合成。结合一种新颖的采样方法，该模型能够将单一单目视频转换为多视角视频，通过优化可变形3D高斯表示，实现了稳健的4D重建。我们在新视角合成和动态场景重建基准测试中展示了具有竞争力的表现，并突出了从真实或生成视频中创造4D场景的能力。请访问我们的项目页面查看结果和互动演示：此 https URL。
URL
https://arxiv.org/abs/2411.18613
