Abstract
Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time this http URL work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.
Abstract (translated)
现实中的自动驾驶必须遵循超出法律规定的交通规则的复杂人类社会规范。许多这样的语义约束，例如给紧急车辆让路、遵守交通警察的手势或为校车停车等行为，对于人类来说是直观的，但对于机器来说却很难明确编码。尽管大型视觉语言模型（VLM）能够解释这些语义，但其推理成本使其无法在实时环境中使用。这项工作提出了LSRE框架，即潜在语义规则编码框架，它将稀疏采样的VLM判断转化为递归世界模型的隐空间中的决策边界。通过将用语言定义的安全语义编码到轻量级的隐分类器中，LSRE能够在不进行每帧VLM查询的情况下以10 Hz的速度实时评估语义风险。在CARLA环境下的六种语义故障场景实验表明，LSRE达到了与大型VLM基线相当的语义风险检测准确性，并提供了显著更早的安全预警同时保持低计算延迟。此外，LSRE还推广到很少见但语义相似的测试案例中，表明语言指导的潜在分类为自动驾驶中的语义安全监控提供了一种有效且可部署的方法。
URL
https://arxiv.org/abs/2512.24712
