Abstract
Multi-view image generation in autonomous driving demands consistent 3D scene understanding across camera views. Most existing methods treat this problem as a 2D image set generation task, lacking explicit 3D modeling. However, we argue that a structured representation is crucial for scene generation, especially for autonomous driving applications. This paper proposes BEV-VAE for consistent and controllable view synthesis. BEV-VAE first trains a multi-view image variational autoencoder for a compact and unified BEV latent space and then generates the scene with a latent diffusion transformer. BEV-VAE supports arbitrary view generation given camera configurations, and optionally 3D layouts. Experiments on nuScenes and Argoverse 2 (AV2) show strong performance in both 3D consistent reconstruction and generation. The code is available at: this https URL.
Abstract (translated)
在自动驾驶中，多视角图像生成要求不同相机视图之间具有连贯的三维场景理解。大多数现有的方法将这个问题视为二维图像集生成任务，并没有明确地进行三维建模。然而，我们认为一个结构化的表示对于场景生成至关重要，特别是在自动驾驶应用中。本文提出了BEV-VAE（Bird's Eye View Variational Autoencoder），以实现一致且可控的视图合成。BEV-VAE首先训练一个多视角图像变分自编码器，以形成紧凑而统一的鸟瞰图隐式空间，然后使用一个隐式扩散变压器来生成场景。BEV-VAE支持给定相机配置以及可选的三维布局的情况下进行任意视图生成。在nuScenes和Argoverse 2（AV2）数据集上的实验表明，在3D一致重建和生成方面具有出色表现。代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2507.00707
