Abstract
Recently, multimodal tasks have strongly advanced the field of action recognition with their rich multimodal information. However, due to the scarcity of tri-modal data, research on tri-modal action recognition tasks faces many challenges. To this end, we have proposed a comprehensive multimodal action recognition solution that effectively utilizes multimodal information. First, the existing data are transformed and expanded by optimizing data enhancement techniques to enlarge the training scale. At the same time, more RGB datasets are used to pre-train the backbone network, which is better adapted to the new task by means of transfer learning. Secondly, multimodal spatial features are extracted with the help of 2D CNNs and combined with the Temporal Shift Module (TSM) to achieve multimodal spatial-temporal feature extraction comparable to 3D CNNs and improve the computational efficiency. In addition, common prediction enhancement methods, such as Stochastic Weight Averaging (SWA), Ensemble and Test-Time augmentation (TTA), are used to integrate the knowledge of models from different training periods of the same architecture and different architectures, so as to predict the actions from different perspectives and fully exploit the target information. Ultimately, we achieved the Top-1 accuracy of 99% and the Top-5 accuracy of 100% on the competition leaderboard, demonstrating the superiority of our solution.
Abstract (translated)
最近，多模态任务通过其丰富的信息促进了动作识别领域的进步。然而，由于三模态数据的稀缺性，对三模态动作识别任务的研究面临着许多挑战。为此，我们提出了一种全面的多模态动作识别解决方案，能够有效利用多模态信息。首先，通过对现有数据进行优化的数据增强技术来转换和扩展这些数据，以扩大训练规模。同时，使用更多的RGB数据集预训练骨干网络，并通过迁移学习将其更好地适应新任务。其次，在2D CNNs的帮助下提取多模态空间特征，并与时间偏移模块（TSM）结合，实现与3D CNNs相媲美的多模态时空特征提取，从而提高计算效率。此外，还采用了诸如随机权重平均(SWA)、集成和测试时增强(TTA)等常见的预测增强方法，以整合相同架构的不同训练时期以及不同架构的模型的知识，从不同的角度进行动作预测，并充分挖掘目标信息。最终，在竞赛排行榜上我们达到了99%的第一准确率和100%的前五准确率，展示了我们的解决方案的优势。
URL
https://arxiv.org/abs/2506.09345
