Abstract
Traffic video description and analysis have received much attention recently due to the growing demand for efficient and reliable urban surveillance systems. Most existing methods only focus on locating traffic event segments, which severely lack descriptive details related to the behaviour and context of all the subjects of interest in the events. In this paper, we present TrafficVLM, a novel multi-modal dense video captioning model for vehicle ego camera view. TrafficVLM models traffic video events at different levels of analysis, both spatially and temporally, and generates long fine-grained descriptions for the vehicle and pedestrian at different phases of the event. We also propose a conditional component for TrafficVLM to control the generation outputs and a multi-task fine-tuning paradigm to enhance TrafficVLM's learning capability. Experiments show that TrafficVLM performs well on both vehicle and overhead camera views. Our solution achieved outstanding results in Track 2 of the AI City Challenge 2024, ranking us third in the challenge standings. Our code is publicly available at this https URL.
Abstract (translated)
近年来，由于对高效且可靠的城郊监控系统需求的不断增加，交通视频描述和分析得到了广泛关注。目前，大多数现有方法仅关注于定位交通事件段，这严重缺乏与所有感兴趣对象的行为和上下文相关的详细描述。在本文中，我们提出了TrafficVLM，一种用于车辆自相机视场的多模态密集视频标注模型。TrafficVLM在不同的分析和空间水平上对交通视频事件进行建模，并生成不同事件阶段车辆和行人的详细描述。我们还提出了一种条件组件，用于控制TrafficVLM的生成输出，以及一种多任务微调范式，以增强TrafficVLM的学习能力。实验证明，TrafficVLM在车辆和 overhead 相机视图上表现出色。我们的解决方案在2024年AI城市挑战赛的第二部分（Track 2）中取得了突出成绩，排名第三。我们的代码公开可用，位于此链接：https://www.example.com。
URL
https://arxiv.org/abs/2404.09275
