Abstract
Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel \textit{Multi Layer Contrastive Supervision} (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of \textit{Multi Layer Contrastive Supervision} as a general and robust strategy for SER. Official implementation can be found in this https URL.
Abstract (translated)
在真实场景中，语音情感识别（SER）仍面临严重类别不平衡和自发自然语音普遍存在的挑战。尽管近期方法利用自监督学习（SSL）表示以及语音与文本的多模态融合，但现有方法大多仅在最终分类层施加监督，限制了中间表征的判别能力。本研究提出Crab（对比表征与多模态对齐瓶颈），一种双模态跨模态Transformer架构，集成来自WavLM的语音表征和来自RoBERTa的文本表征，并引入新颖的\textit{多层对比监督}（MLCS）策略。MLCS在网络多层注入多正例对比学习信号，促使模型各层生成具有情感判别力的表征，且推理时不引入额外参数。为应对数据不平衡，训练中采用加权交叉熵。我们在涵盖不同情感自然度程度的三个基准数据集IEMOCAP、MELD和MSP-Podcast 2.0上评估所提方法。实验结果表明，Crab在所有数据集上持续优于强单模态与多模态基线，尤其在自然且高度不平衡条件下提升显著。这些发现凸显了\textit{多层对比监督}作为SER通用鲁棒策略的有效性。官方实现代码见此https URL。
URL
https://arxiv.org/abs/2603.23673
