Abstract
Vision-language models (VLMs) have recently emerged as powerful representation learning systems that align visual observations with natural language concepts, offering new opportunities for semantic reasoning in safety-critical autonomous driving. This paper investigates how vision-language representations support driving scene safety assessment and decision-making when integrated into perception, prediction, and planning pipelines. We study three complementary system-level use cases. First, we introduce a lightweight, category-agnostic hazard screening approach leveraging CLIP-based image-text similarity to produce a low-latency semantic hazard signal. This enables robust detection of diverse and out-of-distribution road hazards without explicit object detection or visual question answering. Second, we examine the integration of scene-level vision-language embeddings into a transformer-based trajectory planning framework using the Waymo Open Dataset. Our results show that naively conditioning planners on global embeddings does not improve trajectory accuracy, highlighting the importance of representation-task alignment and motivating the development of task-informed extraction methods for safety-critical planning. Third, we investigate natural language as an explicit behavioral constraint on motion planning using the doScenes dataset. In this setting, passenger-style instructions grounded in visual scene elements suppress rare but severe planning failures and improve safety-aligned behavior in ambiguous scenarios. Taken together, these findings demonstrate that vision-language representations hold significant promise for autonomous driving safety when used to express semantic risk, intent, and behavioral constraints. Realizing this potential is fundamentally an engineering problem requiring careful system design and structured grounding rather than direct feature injection.
Abstract (translated)
最近，视觉-语言模型（VLMs）作为强大的表示学习系统出现，它们将视觉观察与自然语言概念相匹配，为安全关键的自动驾驶中的语义推理提供了新的机会。本文研究了当这些视觉-语言表示被整合到感知、预测和规划管道中时，它们如何支持驾驶场景的安全评估和决策制定。 文中探讨了三个互补的系统级用例： 1. **轻量级无类别风险筛查方法**：我们引入了一种基于CLIP（一种图像-文本相似性模型）的方法，该方法利用图像-文本相似性来生成低延迟的语义风险信号。这种方法能够在没有显式对象检测或视觉问答的情况下，对各种类型和分布外的道路危险进行稳健地识别。 2. **场景级视觉语言嵌入与轨迹规划框架集成**：我们研究了将场景级别的视觉语言嵌入整合到基于Transformer的轨迹规划框架中的方法，并使用Waymo Open Dataset进行实验。结果表明，直接用全局嵌入来调整规划器并不会提高轨迹精度，这强调了表示和任务之间对齐的重要性，并推动了针对安全关键性规划的任务导向提取方法的发展。 3. **自然语言作为运动规划的行为约束**：我们研究了在doScenes数据集上使用自然语言作为明确行为约束的方法。在这种设置中，乘客式的指令（基于视觉场景元素）可以抑制罕见但严重的规划失败，并改善含糊情景中的安全一致行为。 综上所述，这些发现表明，在表达语义风险、意图和行为限制时，利用视觉-语言表示对于自动驾驶的安全性具有巨大的潜力。实现这一潜力本质上是一个工程问题，需要精心设计系统结构并进行有序的锚定，而不仅仅是直接注入特征。
URL
https://arxiv.org/abs/2602.07680
