Abstract
Traffic scene understanding is essential for intelligent transportation systems and autonomous driving, ensuring safe and efficient vehicle operation. While recent advancements in VLMs have shown promise for holistic scene understanding, the application of VLMs to traffic scenarios, particularly using BEV maps, remains under explored. Existing methods often suffer from limited task design and narrow data amount, hindering comprehensive scene understanding. To address these challenges, we introduce ChatBEV-QA, a novel BEV VQA benchmark contains over 137k questions, designed to encompass a wide range of scene understanding tasks, including global scene understanding, vehicle-lane interactions, and vehicle-vehicle interactions. This benchmark is constructed using an novel data collection pipeline that generates scalable and informative VQA data for BEV maps. We further fine-tune a specialized vision-language model ChatBEV, enabling it to interpret diverse question prompts and extract relevant context-aware information from BEV maps. Additionally, we propose a language-driven traffic scene generation pipeline, where ChatBEV facilitates map understanding and text-aligned navigation guidance, significantly enhancing the generation of realistic and consistent traffic scenarios. The dataset, code and the fine-tuned model will be released.
Abstract (translated)
交通场景理解对于智能运输系统和自动驾驶至关重要，确保车辆的安全高效运行。虽然最近视觉语言模型（VLMs）在全面场景理解方面取得了显著进展，但将这些技术应用于基于鸟瞰图（BEV）的地图的交通场景中仍处于探索阶段。现有方法通常受限于任务设计狭窄及数据量有限的问题，这阻碍了对复杂交通场景的理解。为了解决这些问题，我们引入了一种新的基准测试ChatBEV-QA，这是一个包含超过137,000个问题的BEV视觉问答（VQA）基准测试，旨在涵盖广泛的场景理解任务，包括全局场景理解、车辆与车道之间的互动以及车辆间的互动。该基准测试通过一个创新的数据收集流程构建而成，能够生成用于BEV地图的大规模和信息丰富的VQA数据。 我们进一步微调了一个专门的视觉语言模型ChatBEV，使其能够解析各种问题提示，并从BEV地图中提取相关且上下文相关的资讯。此外，我们还提出了一种由语言驱动的交通场景生成流程，在此过程中，ChatBEV促进了对地图的理解以及文本与导航指导的一致性，显著提升了现实主义和一致性的交通场景生成。 该数据集、代码及微调后的模型将在未来发布。
URL
https://arxiv.org/abs/2503.13938
