Abstract
Multimodal Sentiment Analysis (MSA) aims to identify speakers' sentiment tendencies in multimodal video content, raising serious concerns about privacy risks associated with multimodal data, such as voiceprints and facial images. Recent distributed collaborative learning has been verified as an effective paradigm for privacy preservation in multimodal tasks. However, they often overlook the privacy distinctions among different modalities, struggling to strike a balance between performance and privacy preservation. Consequently, it poses an intriguing question of maximizing multimodal utilization to improve performance while simultaneously protecting necessary modalities. This paper forms the first attempt at modality-specified (i.e., audio and visual) privacy preservation in MSA tasks. We propose a novel Hybrid Distributed cross-modality cGAN framework (HyDiscGAN), which learns multimodality alignment to generate fake audio and visual features conditioned on shareable de-identified textual data. The objective is to leverage the fake features to approximate real audio and visual content to guarantee privacy preservation while effectively enhancing performance. Extensive experiments show that compared with the state-of-the-art MSA model, HyDiscGAN can achieve superior or competitive performance while preserving privacy.
Abstract (translated)
多模态情感分析（MSA）旨在识别多模态视频内容中发言者的情感倾向，引发对涉及多模态数据隐私风险（如语音和面部图像）的严重关切。最近分布式协同学习被认为是保护多模态任务隐私的有效范式。然而，它们往往忽视不同模态之间的隐私差异，努力在性能和隐私保护之间找到平衡。因此，提出了一个有趣的问题：在提高多模态利用率的同时保护必要的模态。本文是第一个在MSA任务中实现模态指定（即音频和视觉）隐私保护的尝试。我们提出了一种新颖的混合分布式跨模态cGAN框架（HyDiscGAN），通过共享匿名文本数据学习多模态对齐生成假音频和视觉特征。目标是通过假特征利用来近似真实音频和视觉内容，确保隐私保护的同时有效增强性能。大量实验证明，与最先进的MSA模型相比，HyDiscGAN可以在保持隐私的同时实现卓越或竞争力的性能。
URL
https://arxiv.org/abs/2404.11938
