Abstract
Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
Abstract (translated)
最近，基础视频扩散模型（VDMs）取得了显著进展。尽管这些模型生成的视频具有出色的视觉质量，但由于相机控制能力有限以及从不同摄像机轨迹观看时内容的一致性较差，重建连贯的三维场景仍然面临挑战。 本文提出了一个新框架WorldStereo，该框架通过两个专门的几何记忆模块将基于相机指导的视频生成与3D重建连接起来。具体而言，全局几何记忆模块实现了精确的相机控制，并且通过增量更新点云来注入粗略的结构先验知识；而空间立体记忆则通过使用三维对应关系约束模型的关注力接收域，使模型能够从内存库中关注细粒度的细节。 这些组件使得WorldStereo能够在精确控制摄像机的情况下生成多视角一致性的视频，并促进了高质量的3D重建。此外，基于灵活控制分支的WorldStereo表现出令人印象深刻的效率，在无需联合训练的前提下受益于分布匹配蒸馏VDM骨干网络。通过针对基于相机指导的视频生成和3D重建基准进行广泛的实验，展示了我们方法的有效性。 值得注意的是，结果显示WorldStereo充当了一个强大的世界模型，能够处理各种场景生成任务（无论从透视图还是全景图像开始），并以高保真度获得三维结果。相关模型将对外发布。
URL
https://arxiv.org/abs/2603.02049
