Abstract
The looking-in-looking-out (LILO) framework has enabled intelligent vehicle applications that understand both the outside scene and the driver state to improve safety outcomes, with examples in smart airbag deployment, takeover time prediction in autonomous control transitions, and driver attention monitoring. In this research, we propose an augmentation to this framework, making a case for the audio modality as an additional source of information to understand the driver, and in the evolving autonomy landscape, also the passengers and those outside the vehicle. We expand LILO by incorporating audio signals, forming the looking-and-listening inside-and-outside (L-LIO) framework to enhance driver state assessment and environment understanding through multimodal sensor fusion. We evaluate three example cases where audio enhances vehicle safety: supervised learning on driver speech audio to classify potential impairment states (e.g., intoxication), collection and analysis of passenger natural language instructions (e.g., "turn after that red building") to motivate how spoken language can interface with planning systems through audio-aligned instruction data, and limitations of vision-only systems where audio may disambiguate the guidance and gestures of external agents. Datasets include custom-collected in-vehicle and external audio samples in real-world environments. Pilot findings show that audio yields safety-relevant insights, particularly in nuanced or context-rich scenarios where sound is critical to safe decision-making or visual signals alone are insufficient. Challenges include ambient noise interference, privacy considerations, and robustness across human subjects, motivating further work on reliability in dynamic real-world contexts. L-LIO augments driver and scene understanding through multimodal fusion of audio and visual sensing, offering new paths for safety intervention.
Abstract (translated)
"观察与倾听框架（LILO）使得智能车辆应用能够在了解外部场景和驾驶员状态的基础上提高安全性能，例如智能气囊部署、自动驾驶控制转换时的接管时间预测以及驾驶员注意力监控。在此研究中，我们提议对此框架进行增强，并强调音频模态作为理解驾驶员及其他人员的重要信息来源，在不断发展的自主驾驶环境中尤其如此。我们将LILO扩展为结合声音信号的形式，形成“倾听与观察内部和外部”（L-LIO）框架，通过多模态传感器融合提高对驾驶员状态的评估及环境的理解能力。 我们在三个案例中评估了音频如何增强车辆安全性：使用监督学习分析司机语音音频来分类潜在的身体状况（如酒后驾驶），收集并分析乘客自然语言指令（例如，“在那个红房子之后转弯”）以展示口语如何通过与视觉信号对齐的指令数据与规划系统交互，以及仅依赖视觉系统的局限性，在这种情况下声音可以消除外部代理行为指导和手势中的歧义。这些案例的数据集包括从真实世界环境中定制收集的车内及车外音频样本。 初步研究结果表明，音频提供了重要的安全信息，尤其是在细微或背景丰富的场景中，其中声音对于做出安全决策至关重要或是视觉信号本身是不够的时候。然而，挑战还包括环境噪音干扰、隐私考虑以及在不同人类受试者间实现稳健性的难题，这需要我们在动态的真实世界环境中进一步增强系统的可靠性。 L-LIO框架通过多模态融合的听觉和视觉感应来提高驾驶员状态和场景的理解能力，并为安全干预提供了新的路径。"
URL
https://arxiv.org/abs/2602.07668
