Abstract
We present Gen3R, a method that bridges the strong priors of foundational reconstruction models and video diffusion models for scene-level 3D generation. We repurpose the VGGT reconstruction model to produce geometric latents by training an adapter on its tokens, which are regularized to align with the appearance latents of pre-trained video diffusion models. By jointly generating these disentangled yet aligned latents, Gen3R produces both RGB videos and corresponding 3D geometry, including camera poses, depth maps, and global point clouds. Experiments demonstrate that our approach achieves state-of-the-art results in single- and multi-image conditioned 3D scene generation. Additionally, our method can enhance the robustness of reconstruction by leveraging generative priors, demonstrating the mutual benefit of tightly coupling reconstruction and generative models.
Abstract (translated)
我们介绍了Gen3R方法，该方法将基础重建模型的强先验与视频扩散模型相结合，以实现场景级别的三维生成。通过在VGGT重建模型的令牌上训练一个适配器，我们将VGGT模型重新用于产生几何潜变量，并且这些令牌被正则化以与预训练的视频扩散模型的外观潜变量对齐。通过联合生成这些分离但又对齐的潜变量，Gen3R可以同时生成RGB视频和对应的三维几何结构，包括相机姿态、深度图和全局点云。 实验表明，在单图像和多图像条件下的三维场景生成中，我们的方法达到了最先进的性能水平。此外，我们的方法可以通过利用生成先验来增强重建的鲁棒性，从而展示出紧密耦合重建模型与生成模型的相互收益。
URL
https://arxiv.org/abs/2601.04090
