Abstract
Speech emotion recognition (SER) systems can exhibit gender-related performance disparities, but how such bias manifests in multilingual speech LLMs across languages and modalities is unclear. We introduce a novel multilingual, multimodal benchmark built on MELD-ST, spanning English, Japanese, and German, to quantify language-specific SER performance and gender gaps. We find bias is strongly language-dependent, and multimodal fusion does not reliably improve fairness. To address these, we propose ERM-MinMaxGAP, a fairness-informed training objective, which augments empirical risk minimization (ERM) with a proposed adaptive fairness weight mechanism and a novel MinMaxGAP regularizer on the maximum male-female loss gap within each language and modality. Building upon the Qwen2-Audio backbone, our ERM-MinMaxGAP approach improves multilingual SER performance by 5.5% and 5.0% while reducing the overall gender bias gap by 0.1% and 1.4% in the unimodal and multimodal settings, respectively.
Abstract (translated)
语音情感识别（SER）系统可能表现出与性别相关的性能差异，但此类偏见在多语言语音大语言模型中如何跨语言和模态体现尚不明确。我们基于MELD-ST构建了一个新颖的多语言多模态基准，涵盖英语、日语和德语，以量化特定语言的SER性能及性别差距。研究发现偏见高度依赖语言，且多模态融合并不能可靠地提升公平性。针对这些问题，我们提出了ERM-MinMaxGAP——一种考虑公平性的训练目标，它在经验风险最小化（ERM）基础上增加了提出的自适应公平权重机制，以及一种针对每种语言和模态内男女损失最大差距的新型MinMaxGAP正则化器。基于Qwen2-Audio主干网络，我们的ERM-MinMaxGAP方法在单模态和多模态设置中分别将多语言SER性能提升5.5%和5.0%，同时将整体性别偏见差距降低0.1%和1.4%。
URL
https://arxiv.org/abs/2603.21050
