Abstract
Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.
Abstract (translated)
音频-视觉情感识别（AVER）方法通常融合话语级特征，即使是一些帧级别的注意力模型也较少解决跨模态的帧率不匹配问题。在本文中，我们提出了一种基于Transformer框架的方法，专注于多模态特征的时间对齐。我们的设计采用了一个多模态自注意编码器，在共享的特征空间内同时捕捉了内部和跨模态之间的依赖关系。为了解决异质采样率的问题，我们引入了时间对齐旋转位置嵌入（TaRoPE），它隐式地同步音频和视频标记。此外，我们提出了一种交叉时间匹配（CTM）损失函数，该函数强制相邻时间点的一致性，引导编码器向更好的对齐方向发展。在CREMA-D和RAVDESS数据集上的实验表明，与最近的基准方法相比，我们的方法取得了持续改进的效果，这表明明确解决帧率不匹配问题有助于保留时间线索并增强跨模态融合效果。
URL
https://arxiv.org/abs/2603.11095
