Abstract
Accurate and high-fidelity driving scene reconstruction demands the effective utilization of comprehensive scene information as conditional inputs. Existing methods predominantly rely on 3D bounding boxes and BEV road maps for foreground and background control, which fail to capture the full complexity of driving scenes and adequately integrate multimodal information. In this work, we present DualDiff, a dual-branch conditional diffusion model designed to enhance driving scene generation across multiple views and video sequences. Specifically, we introduce Occupancy Ray-shape Sampling (ORS) as a conditional input, offering rich foreground and background semantics alongside 3D spatial geometry to precisely control the generation of both elements. To improve the synthesis of fine-grained foreground objects, particularly complex and distant ones, we propose a Foreground-Aware Mask (FGM) denoising loss function. Additionally, we develop the Semantic Fusion Attention (SFA) mechanism to dynamically prioritize relevant information and suppress noise, enabling more effective multimodal fusion. Finally, to ensure high-quality image-to-video generation, we introduce the Reward-Guided Diffusion (RGD) framework, which maintains global consistency and semantic coherence in generated videos. Extensive experiments demonstrate that DualDiff achieves state-of-the-art (SOTA) performance across multiple datasets. On the NuScenes dataset, DualDiff reduces the FID score by 4.09% compared to the best baseline. In downstream tasks, such as BEV segmentation, our method improves vehicle mIoU by 4.50% and road mIoU by 1.70%, while in BEV 3D object detection, the foreground mAP increases by 1.46%. Code will be made available at this https URL.
Abstract (translated)
准确且高保真的驾驶场景重建需要有效地利用全面的场景信息作为条件输入。现有的方法主要依赖于三维边界框和鸟瞰图道路地图来控制前景和背景，这无法捕捉驾驶场景的全部复杂性，并且不足以充分融合多模态信息。在本工作中，我们提出了DualDiff，这是一种双分支条件扩散模型，旨在增强从多个视角和视频序列生成驾驶场景的能力。具体来说，我们引入了Occupancy Ray-shape Sampling（ORS）作为条件输入，提供丰富的前景和背景语义以及三维空间几何信息来精确控制两者的生成。 为了改进细粒度前景对象的合成，特别是复杂且距离较远的对象，我们提出了一种基于前景的掩码去噪损失函数（FGM）。此外，我们开发了语义融合注意机制（SFA），以动态优先处理相关信息并抑制噪声，从而实现更有效的多模态融合。 最后，为了确保高质量的图像到视频生成，我们引入了引导式扩散框架（RGD），它在生成的视频中保持全局一致性和语义连贯性。广泛的实验表明，DualDiff在多个数据集上达到了最先进的性能。在NuScenes数据集上，与最佳基线相比，DualDiff将FID得分降低了4.09%。在下游任务（如鸟瞰图分割）中，我们的方法使车辆的mIoU提高了4.50%，道路的mIoU提高了1.70%，而在鸟瞰图3D对象检测中，前景的mAP增加了1.46%。 代码将在[此链接](https://this-is-the-url-to-be-filled-in-later.com)提供。
URL
https://arxiv.org/abs/2503.03689
