Abstract
We introduce Scenario Dreamer, a fully data-driven generative simulator for autonomous vehicle planning that generates both the initial traffic scene - comprising a lane graph and agent bounding boxes - and closed-loop agent behaviours. Existing methods for generating driving simulation environments encode the initial traffic scene as a rasterized image and, as such, require parameter-heavy networks that perform unnecessary computation due to many empty pixels in the rasterized scene. Moreover, we find that existing methods that employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer instead employs a novel vectorized latent diffusion model for initial scene generation that directly operates on the vectorized scene elements and an autoregressive Transformer for data-driven agent behaviour simulation. Scenario Dreamer additionally supports scene extrapolation via diffusion inpainting, enabling the generation of unbounded simulation environments. Extensive experiments show that Scenario Dreamer outperforms existing generative simulators in realism and efficiency: the vectorized scene-generation base model achieves superior generation quality with around 2x fewer parameters, 6x lower generation latency, and 10x fewer GPU training hours compared to the strongest baseline. We confirm its practical utility by showing that reinforcement learning planning agents are more challenged in Scenario Dreamer environments than traditional non-generative simulation environments, especially on long and adversarial driving environments.
Abstract (translated)
我们介绍了Scenario Dreamer，这是一种完全基于数据驱动的自动驾驶车辆规划生成模拟器。它可以生成初始交通场景（包括车道图和代理边框）以及闭环代理行为。现有的用于生成驾驶仿真环境的方法将初始交通场景编码为栅格化图像，并因此需要参数繁重的网络来执行不必要的计算，因为栅格化场景中有许多空白像素。此外，我们发现使用基于规则的代理行为方法缺乏多样性和现实性。 与现有方法不同，Scenario Dreamer采用了一种新颖的向量化潜在扩散模型进行初始场景生成，该模型直接操作向量化的场景元素，并且使用自回归Transformer来进行数据驱动的代理行为模拟。此外，Scenario Dreamer通过扩散填补技术支持场景扩展，从而能够生成不受限的仿真环境。 大量的实验表明，与现有的生成式模拟器相比，Scenario Dreamer在真实性和效率方面表现更佳：向量化场景生成基础模型实现了更高的生成质量，参数数量减少了大约2倍，生成延迟降低了6倍，并且GPU训练时间也减少了10倍。我们通过展示强化学习规划代理在Scenario Dreamer环境中面临更大挑战的事实来确认其实际效用，特别是在长时间和对抗性驾驶环境中的表现优于传统的非生成式仿真环境。
URL
https://arxiv.org/abs/2503.22496
