Abstract
Reconstructing 4D dynamic scenes from casually captured monocular videos is valuable but highly challenging, as each timestamp is observed from a single viewpoint. We introduce Vivid4D, a novel approach that enhances 4D monocular video synthesis by augmenting observation views - synthesizing multi-view videos from a monocular input. Unlike existing methods that either solely leverage geometric priors for supervision or use generative priors while overlooking geometry, we integrate both. This reformulates view augmentation as a video inpainting task, where observed views are warped into new viewpoints based on monocular depth priors. To achieve this, we train a video inpainting model on unposed web videos with synthetically generated masks that mimic warping occlusions, ensuring spatially and temporally consistent completion of missing regions. To further mitigate inaccuracies in monocular depth priors, we introduce an iterative view augmentation strategy and a robust reconstruction loss. Experiments demonstrate that our method effectively improves monocular 4D scene reconstruction and completion.
Abstract (translated)
从随意捕捉的单目视频中重建动态的四维场景具有很高的价值，但同时也极具挑战性，因为每个时间戳只能从单一视角观察。为此，我们提出了Vivid4D这一创新方法，它通过增强观测视角来提升四维单目视频合成的质量——即从单目输入生成多视点视频。与现有的要么单纯依赖几何先验进行监督、要么利用生成式先验而忽视了几何特性的方法不同，我们的方法整合了这两者。 我们将视角增强重新定义为一个视频修复任务，在此过程中，根据单目深度先验将已观测到的视角扭曲至新的视点。为了实现这一点，我们训练了一个视频修复模型，并在未经过预设处理的网络视频上进行训练，同时生成模仿视差遮挡效果的合成掩码，以确保缺失区域的空间和时间一致性得到妥善完成。 为进一步减少单目深度先验中的不准确性，我们引入了一种迭代视角增强策略及一种鲁棒重建损失函数。实验结果表明，我们的方法能够有效提升基于单目的四维场景重建与填补的效果。
URL
https://arxiv.org/abs/2504.11092
