Abstract
Multimodal Large Language Models (MLLMs) are rapidly becoming the intelligence brain of end-to-end autonomous driving systems. A key challenge is to assess whether MLLMs can truly understand and follow complex real-world traffic rules. However, existing benchmarks mainly focus on single-rule scenarios like traffic sign recognition, neglecting the complexity of multi-rule concurrency and conflicts in real driving. Consequently, models perform well on simple tasks but often fail or violate rules in real world complex situations. To bridge this gap, we propose DriveCombo, a text and vision-based benchmark for compositional traffic rule reasoning. Inspired by human drivers' cognitive development, we propose a systematic Five-Level Cognitive Ladder that evaluates reasoning from single-rule understanding to multi-rule integration and conflict resolution, enabling quantitative assessment across cognitive stages. We further propose a Rule2Scene Agent that maps language-based traffic rules to dynamic driving scenes through rule crafting and scene generation, enabling scene-level traffic rule visual reasoning. Evaluations of 14 mainstream MLLMs reveal performance drops as task complexity grows, particularly during rule conflicts. After splitting the dataset and fine-tuning on the training set, we further observe substantial improvements in both traffic rule reasoning and downstream planning capabilities. These results highlight the effectiveness of DriveCombo in advancing compliant and intelligent autonomous driving systems.
Abstract (translated)
多模态大型语言模型（MLLMs）正在迅速成为端到端自动驾驶系统的智能核心。一个关键挑战在于评估这些模型是否能够真正理解并遵守复杂的现实世界交通规则。然而，现有的基准测试主要集中在单一规则场景上，如交通标志识别，忽视了实际驾驶中多重规则共存和冲突的复杂性。因此，虽然模型在简单任务上的表现良好，但在真实世界的复杂情况下往往会出现失误或违反规则的现象。 为了弥合这一差距，我们提出了DriveCombo，这是一个基于文本和视觉的基准测试平台，旨在评估组合型交通规则推理能力。受人类驾驶员认知发展过程启发，我们提出了一套五级认知阶梯系统评价从单一规则理解到多重规则整合与冲突解决的能力，并能够跨认知阶段进行量化评估。此外，我们还提出了Rule2Scene Agent这一概念，通过制定规则和生成场景将基于语言的交通规则映射为动态驾驶场景，从而实现基于场景级别的视觉推理。 对14种主流MLLMs的评测显示，在任务复杂度增加时性能显著下降，特别是在处理规则冲突的情况下。进一步地，在分割数据集并在训练集上进行微调后，我们观察到了交通规则推理和下游规划能力方面的重要改进。这些结果凸显了DriveCombo在推动合规且智能的自动驾驶系统发展方面的有效性。
URL
https://arxiv.org/abs/2603.01637
