Abstract
Understanding human actions in videos requires more than raw pixel analysis; it relies on high-level semantic reasoning and effective integration of multimodal features. We propose a deep translational action recognition framework that enhances recognition accuracy by jointly predicting action concepts and auxiliary features from RGB video frames. At test time, hallucination streams infer missing cues, enriching feature representations without increasing computational overhead. To focus on action-relevant regions beyond raw pixels, we introduce two novel domain-specific descriptors. Object Detection Features (ODF) aggregate outputs from multiple object detectors to capture contextual cues, while Saliency Detection Features (SDF) highlight spatial and intensity patterns crucial for action recognition. Our framework seamlessly integrates these descriptors with auxiliary modalities such as optical flow, Improved Dense Trajectories, skeleton data, and audio cues. It remains compatible with state-of-the-art architectures, including I3D, AssembleNet, Video Transformer Network, FASTER, and recent models like VideoMAE V2 and InternVideo2. To handle uncertainty in auxiliary features, we incorporate aleatoric uncertainty modeling in the hallucination step and introduce a robust loss function to mitigate feature noise. Our multimodal self-supervised action recognition framework achieves state-of-the-art performance on multiple benchmarks, including Kinetics-400, Kinetics-600, and Something-Something V2, demonstrating its effectiveness in capturing fine-grained action dynamics.
Abstract (translated)
理解视频中的人类动作不仅仅是对原始像素的分析；它还需要高层次的语义推理和有效整合多模态特征。我们提出了一种深度转换式动作识别框架，通过共同预测RGB视频帧的动作概念和辅助特征来提高识别准确性。在测试时，幻觉流可以推断缺失的线索，丰富特征表示而不增加计算开销。为了超越原始像素关注与动作相关区域，我们引入了两种新颖的领域特定描述符：物体检测特征（ODF）聚合来自多个对象检测器的输出以捕捉上下文线索；而注意力检测特征（SDF）则突出空间和强度模式，这些模式对动作识别至关重要。我们的框架可以无缝地将这些描述符与光流、改进的密集轨迹、骨骼数据以及音频提示等辅助模态整合在一起，并且兼容包括I3D、AssembleNet、视频变换网络（Video Transformer Network）、FASTER以及最近的模型如VideoMAE V2和InternVideo2在内的最先进架构。为了处理辅助特征中的不确定性，我们在幻觉步骤中引入了经验不确定性建模，并提出了一种稳健的损失函数以减少特征噪声。我们的多模态自监督动作识别框架在Kinetics-400、Kinetics-600以及Something-Something V2等多个基准测试上取得了最先进的性能，证明其能够有效捕捉细粒度的动作动态变化。
URL
https://arxiv.org/abs/2506.20342
