Abstract
While video generation models excel at producing high-quality monocular videos, generating 3D stereoscopic and spatial videos for immersive applications remains an underexplored challenge. We present a pose-free and training-free method that leverages an off-the-shelf monocular video generation model to produce immersive 3D videos. Our approach first warps the generated monocular video into pre-defined camera viewpoints using estimated depth information, then applies a novel \textit{frame matrix} inpainting framework. This framework utilizes the original video generation model to synthesize missing content across different viewpoints and timestamps, ensuring spatial and temporal consistency without requiring additional model fine-tuning. Moreover, we develop a \dualupdate~scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. The resulting multi-view videos are then adapted into stereoscopic pairs or optimized into 4D Gaussians for spatial video synthesis. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, such as Sora, Lumiere, WALT, and Zeroscope. The experiments demonstrate that our method has a significant improvement over previous methods. Project page at: this https URL
Abstract (translated)
虽然视频生成模型在制作高质量的单目视频方面表现出色，但生成用于沉浸式应用的3D立体和空间视频仍是一个尚未充分探索的挑战。我们提出了一种无需姿势信息且无需训练的方法，该方法利用现成的单目视频生成模型来生产沉浸式的3D视频。我们的方法首先使用估计出的深度信息将生成的单目视频变形到预定义的摄像机视角中，然后应用一种新颖的“帧矩阵”插值框架。这一框架借助原始的视频生成模型在不同的视角和时间戳间合成缺失的内容，从而确保空间和时间的一致性，无需额外调整模型参数。此外，我们开发了一种\dualupdate（双更新）方案，通过减轻从遮挡区域传递到潜在空间中的负面影响来进一步提高视频插值的质量。最终的多视图视频随后被转换为立体对或优化成4D高斯分布，用于空间视频合成。我们在各种生成模型（如Sora、Lumiere、WALT和Zeroscope）生成的视频上进行了实验验证，结果表明我们提出的方法在效果上明显优于以往方法。 项目页面在此链接：[请将此URL替换为实际提供的链接]
URL
https://arxiv.org/abs/2508.08048
