Abstract
In this paper, we propose a new Multimodal Representation Learning (MRL) method for Multimodal Sentiment Analysis (MSA), which facilitates the adaptive interaction between modalities through Cooperative Sentiment Agents, named Co-SA. Co-SA comprises two critical components: the Sentiment Agents Establishment (SAE) phase and the Sentiment Agents Cooperation (SAC) phase. During the SAE phase, each sentiment agent deals with an unimodal signal and highlights explicit dynamic sentiment variations within the modality via the Modality-Sentiment Disentanglement (MSD) and Deep Phase Space Reconstruction (DPSR) modules. Subsequently, in the SAC phase, Co-SA meticulously designs task-specific interaction mechanisms for sentiment agents so that coordinating multimodal signals to learn the joint representation. Specifically, Co-SA equips an independent policy model for each sentiment agent that captures significant properties within the modality. These policies are optimized mutually through the unified reward adaptive to downstream tasks. Benefitting from the rewarding mechanism, Co-SA transcends the limitation of pre-defined fusion modes and adaptively captures unimodal properties for MRL in the multimodal interaction setting. To demonstrate the effectiveness of Co-SA, we apply it to address Multimodal Sentiment Analysis (MSA) and Multimodal Emotion Recognition (MER) tasks. Our comprehensive experimental results demonstrate that Co-SA excels at discovering diverse cross-modal features, encompassing both common and complementary aspects. The code can be available at this https URL.
Abstract (translated)
在本文中，我们提出了一个新的多模态表示学习（MRL）方法，名为合作情感代理（Co-SA），用于多模态情感分析（MSA），并通过合作情感代理促进模态之间的自适应交互。Co-SA包括两个关键组件：情感代理建立（SAE）阶段和情感代理合作（SAC）阶段。在SAE阶段，每个情感代理处理一个单模态信号，并通过模态情感解离（MSD）和深度时域重构（DPSR）模块在模态内突出显示动态情感变化。然后，在SAC阶段，Co-SA精心设计了一系列任务特定的情感代理交互机制，以协调多模态信号以学习联合表示。具体来说，Co-SA为每个情感代理配备了一个独立的政策模型，该模型捕捉模态内的显著属性。这些策略通过统一奖励适应下游任务进行优化。得益于奖励机制，Co-SA超越了预定义的融合模式，并适应了多模态交互设置中的情感代理学习（MRL）。为了证明Co-SA的有效性，我们将它应用于情感多模态分析和情感识别任务。我们全面的实验结果表明，Co-SA在发现跨模态特征方面表现出色，涵盖模态共性和互补性的各个方面。代码可以从该链接获取。
URL
https://arxiv.org/abs/2404.12642
