Abstract
Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propose a novel unified DWM framework based on 3D Gaussian scene representation, which enables both 3D scene understanding and multi-modal scene generation, while also enabling contextual enrichment for understanding and generation tasks. Our approach directly aligns textual information with the 3D scene by embedding rich linguistic features into each Gaussian primitive, thereby achieving early modality alignment. In addition, we design a novel task-aware language-guided sampling strategy that removes redundant 3D Gaussians and injects accurate and compact 3D tokens into LLM. Furthermore, we design a dual-condition multi-modal generation model, where the information captured by our vision-language model is leveraged as a high-level language condition in combination with a low-level image condition, jointly guiding the multi-modal generation process. We conduct comprehensive studies on the nuScenes, and NuInteract datasets to validate the effectiveness of our framework. Our method achieves state-of-the-art performance. We will release the code publicly on GitHub this https URL.
Abstract (translated)
随着生成模型的进步，驾驶世界模型（DWMs）正在迅速发展。然而，现有的DWM缺乏对3D场景的理解能力，并且只能根据输入数据生成内容，而无法解释或推理关于驾驶环境的信息。此外，当前的方法用点云或BEV特征表示3D空间信息，这些方法不能准确地将文本信息与底层的3D场景相匹配。 为了解决这些问题，我们提出了一种基于3D高斯场景表示的新统一DWM框架，它能够同时实现对3D场景的理解和多模态场景生成，并且还支持理解和生成任务的情境丰富化。我们的方法通过将丰富的语言特征嵌入到每个高斯原语中直接将文本信息与3D场景进行对齐，从而实现了早期的模式对齐。 此外，我们设计了一种新颖的任务感知的语言引导采样策略，该策略可以去除冗余的3D高斯模型，并注入准确且紧凑的3D令牌至LLM（大型语言模型）。另外，我们还设计了一个双条件多模态生成模型，在其中我们的视觉-语言模型捕获的信息被用作高级语言条件与低级图像条件结合使用，共同引导多模态生成过程。 我们在nuScenes和NuInteract数据集上进行了全面的研究以验证框架的有效性。我们的方法达到了最先进的性能。我们将在GitHub上公开发布代码，该链接为[此URL]（请将此URL替换为您实际的发布地址）。
URL
https://arxiv.org/abs/2512.23180
