Abstract
Fine-grained video classification requires understanding complex spatio-temporal and semantic cues that often exceed the capacity of a single modality. In this paper, we propose a multimodal framework that fuses video, image, and text representations using GRU-based sequence encoders and cross-modal attention mechanisms. The model is trained using a combination of classification or regression loss, depending on the task, and is further regularized through feature-level augmentation and autoencoding techniques. To evaluate the generality of our framework, we conduct experiments on two challenging benchmarks: the DVD dataset for real-world violence detection and the Aff-Wild2 dataset for valence-arousal estimation. Our results demonstrate that the proposed fusion strategy significantly outperforms unimodal baselines, with cross-attention and feature augmentation contributing notably to robustness and performance.
Abstract (translated)
精细粒度的视频分类需要理解复杂的时空和语义线索，这些往往超出了单一模态的能力。在本文中，我们提出了一种多模态框架，该框架融合了视频、图像和文本表示，并使用基于GRU（门控循环单元）的序列编码器和跨模态注意力机制来实现这一目标。模型通过结合分类或回归损失进行训练，具体取决于任务需求，并且通过特征级增强和自编码技术进一步正则化。 为了评估我们框架的通用性，我们在两个具有挑战性的基准数据集上进行了实验：DVD数据集用于现实世界的暴力检测，以及Aff-Wild2数据集用于效价唤醒度估计。我们的结果显示，所提出的融合策略显著优于单模态基线模型，在鲁棒性和性能方面，跨注意力机制和特征增强的贡献尤为突出。
URL
https://arxiv.org/abs/2507.03531
