Abstract
Existing speech emotion recognition (SER) methods commonly suffer from the lack of high-quality large-scale corpus, partly due to the complex, psychological nature of emotion which makes accurate labeling difficult and time consuming. Recently, transfer learning based methods that exploit the encoders pretrained on large-scale speech corpus (e.g., Wav2Vec2.0 and HuBERT) have shown strong potential for downstream SER tasks. However, task-specific fine-tuning remains necessary for various conversational scenarios of different topics, speakers and languages to achieve satisfactory performance. It generally requires costly encoder retraining for individual SER tasks. To address this issue, we propose to train an adapter with local attributes for efficient transfer learning. Specifically, a weighted average pooling-Transformer (WAP-Transformer) is proposed as a lightweight backbone to enrich the frame-level representation. An adapter with teacher-student branches is exploited for task-agnostic transfer learning, where the student branch is jointly optimized via mask prediction and self-distillation objectives, and the teacher branch is obtained online from the student via exponential moving average (EMA). Meanwhile, local attributes are learned from the teacher branch via unsupervised clustering, which aims to act as a universal model that provides additional semantic-rich supervisions. A statistical attentive pooling (SAP) module is proposed to obtain utterance representation for fine-tuning. To evaluate the effectiveness of the proposed adapter with local attributes, extensive experiments have been conducted on IEMOCAP. Superior performance has been reported, compared to the previous state-of-the-art methods in similar settings.
Abstract (translated)
现有的语音情感识别（SER）方法通常因为高质量的大规模语料库的缺乏而表现不佳，这在一定程度上是因为情感具有复杂和心理性质的特点，使得准确标记情感变得困难且耗时。最近，基于迁移学习的方法通过利用大规模语音语料库（如Wav2Vec2.0和HuBERT）预先训练的编码器，在下游SER任务中表现出强大的潜力。然而，为了在不同主题、说话者和语言的各种对话场景中实现满意的性能，针对特定任务的微调仍然是必要的，这通常需要对每个SER任务进行昂贵的编码器重新训练。 为了解决这个问题，我们提出了通过具有局部属性的适配器来进行高效的迁移学习的方法。具体来说，我们提出了一种轻量级的骨干网络——加权平均池化-变压器（WAP-Transformer），以丰富帧级别的表示。一个带有教师-学生分支的适配器被用于无任务特定性的迁移学习中，其中学生分支通过掩码预测和自我蒸馏目标联合优化，而教师分支则在线从学生处通过指数移动平均（EMA）获得。同时，局部属性通过无监督聚类从教师分支中学得，并旨在作为一个通用模型提供额外的语义丰富的监督信息。一个统计注意力池化（SAP）模块被提出用于获取进行微调的语音表示。 为了评估具有局部属性的适配器的有效性，在IEMOCAP数据集上进行了广泛的实验，结果表明该方法在类似设置下比之前的最先进的SER方法表现出色。
URL
https://arxiv.org/abs/2509.23795
