Abstract
Driver activity classification is crucial for ensuring road safety, with applications ranging from driver assistance systems to autonomous vehicle control transitions. In this paper, we present a novel approach leveraging generalizable representations from vision-language models for driver activity classification. Our method employs a Semantic Representation Late Fusion Neural Network (SRLF-Net) to process synchronized video frames from multiple perspectives. Each frame is encoded using a pretrained vision-language encoder, and the resulting embeddings are fused to generate class probability predictions. By leveraging contrastively-learned vision-language representations, our approach achieves robust performance across diverse driver activities. We evaluate our method on the Naturalistic Driving Action Recognition Dataset, demonstrating strong accuracy across many classes. Our results suggest that vision-language representations offer a promising avenue for driver monitoring systems, providing both accuracy and interpretability through natural language descriptors.
Abstract (translated)
驾驶员活动分类对于确保道路安全具有至关重要的作用，应用范围从驾驶员辅助系统到自动驾驶车辆控制转换。在本文中，我们提出了一种利用可扩展性来自视觉-语言模型的驾驶员活动分类新方法。我们的方法采用了一个预训练的视觉-语言编码器来处理来自多个视角的同步视频帧。每个帧都使用预训练的视觉-语言编码器进行编码，然后将得到的嵌入进行融合以生成分类概率预测。通过利用对比学习得到的视觉-语言表示，我们的方法在多样驾驶员活动中取得了稳健的性能。我们在自然驾驶行动识别数据集上评估我们的方法，证明了在许多类别中具有强大的准确性。我们的结果表明，视觉-语言表示为驾驶员监测系统提供了有前途的途径，通过自然语言描述符实现准确性和可解释性。
URL
https://arxiv.org/abs/2404.14906
