Abstract
This study introduces a pioneering methodology for human action recognition by harnessing deep neural network techniques and adaptive fusion strategies across multiple modalities, including RGB, optical flows, audio, and depth information. Employing gating mechanisms for multimodal fusion, we aim to surpass limitations inherent in traditional unimodal recognition methods while exploring novel possibilities for diverse applications. Through an exhaustive investigation of gating mechanisms and adaptive weighting-based fusion architectures, our methodology enables the selective integration of relevant information from various modalities, thereby bolstering both accuracy and robustness in action recognition tasks. We meticulously examine various gated fusion strategies to pinpoint the most effective approach for multimodal action recognition, showcasing its superiority over conventional unimodal methods. Gating mechanisms facilitate the extraction of pivotal features, resulting in a more holistic representation of actions and substantial enhancements in recognition performance. Our evaluations across human action recognition, violence action detection, and multiple self-supervised learning tasks on benchmark datasets demonstrate promising advancements in accuracy. The significance of this research lies in its potential to revolutionize action recognition systems across diverse fields. The fusion of multimodal information promises sophisticated applications in surveillance and human-computer interaction, especially in contexts related to active assisted living.
Abstract (translated)
这项研究介绍了一种利用深度神经网络技术和跨多模态（包括RGB、光学流、音频和深度信息）的自适应融合策略进行人体动作识别的开创性方法。通过采用门控机制进行多模态融合，我们旨在超越传统单一模态识别方法固有的局限性，并探索多样化的应用可能。通过对各种门控机制及基于自适应加权的融合架构进行详尽调查，本研究的方法能够选择性地整合来自不同模式的相关信息，从而增强动作识别任务中的准确性和鲁棒性。我们仔细评估了多种门控融合策略，以确定最适合多模态行动识别的最佳方法，并展示了其相对于传统单一模态方法的优势。门控机制有助于提取关键特征，使动作的表示更加全面，显著提高了识别性能。我们在人类行为识别、暴力行为检测及多个自监督学习任务上的基准数据集评估表明，在准确性方面取得了令人鼓舞的进步。这项研究的意义在于它有可能在各个领域革新动作识别系统。多模态信息融合预示着在监控和人机交互领域的复杂应用，特别是在主动辅助生活相关的情境中具有重要意义。
URL
https://arxiv.org/abs/2512.04943
