Abstract
The task of spatiotemporal action localization in chaotic scenes is a challenging task toward advanced video understanding. Paving the way with high-quality video feature extraction and enhancing the precision of detector-predicted anchors can effectively improve model performance. To this end, we propose a high-performance dual-stream spatiotemporal feature extraction network SFMViT with an anchor pruning strategy. The backbone of our SFMViT is composed of ViT and SlowFast with prior knowledge of spatiotemporal action localization, which fully utilizes ViT's excellent global feature extraction capabilities and SlowFast's spatiotemporal sequence modeling capabilities. Secondly, we introduce the confidence maximum heap to prune the anchors detected in each frame of the picture to filter out the effective anchors. These designs enable our SFMViT to achieve a mAP of 26.62% in the Chaotic World dataset, far exceeding existing models. Code is available at this https URL.
Abstract (translated)
在复杂场景中进行时空动作局部化的任务是高级视频理解的一个具有挑战性的任务。通过高质量的视频特征提取和增强检测器预测锚点的精度，可以有效地提高模型性能。为此，我们提出了一个高性能的双流时空特征提取网络SFMViT，采用锚点剪枝策略。我们SFMViT的骨干网络由ViT和SlowFast组成，基于先前对时空动作局部化的知识，充分利用ViT的卓越全局特征提取能力和SlowFast的时空序列建模能力。其次，我们引入了最大置信度堆来剪枝检测器在每个帧中检测到的锚点，以过滤出有效的锚点。这些设计使我们的SFMViT在Chaotic World数据集上的mAP达到26.62%，远超过现有模型。代码可以从该链接获得。
URL
https://arxiv.org/abs/2404.16609
