Abstract
Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model (LLM), enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at this https URL.
Abstract (translated)
驾驶世界模型（DWMs）已成为自动驾驶不可或缺的一部分，通过实现未来场景预测来支持自动驾驶。然而，现有的DWM仅限于生成新场景，并未能整合对驾驶环境的理解，即无法解读和推理驾驶环境中的情况。在本文中，我们提出了一种统一的驾驶世界模型HERMES。我们在驾驶场景下通过一个统一框架无缝地集成了三维场景理解和未来场景演变（生成）。具体而言，HERMES利用鸟瞰图（BEV）表示来整合多视角空间信息的同时保持几何关系和互动性。此外，我们还引入了世界查询机制，该机制通过在大型语言模型中应用因果注意力将世界知识融入到BEV特征中，从而为理解和生成任务提供上下文增强。我们在nuScenes和OmniDrive-nuScenes数据集上进行了全面的研究以验证我们的方法的有效性。HERMES取得了最先进的性能，在减少生成误差方面比现有技术降低了32.4%，在理解指标如CIDEr的评估中提高了8.0%。该模型及其代码将在[此链接](https://this https URL)公开发布。
URL
https://arxiv.org/abs/2501.14729
