Abstract
Dynamic Facial Expression Recognition (DFER) has received significant interest in the recent years dictated by its pivotal role in enabling empathic and human-compatible technologies. Achieving robustness towards in-the-wild data in DFER is particularly important for real-world applications. One of the directions aimed at improving such models is multimodal emotion recognition based on audio and video data. Multimodal learning in DFER increases the model capabilities by leveraging richer, complementary data representations. Within the field of multimodal DFER, recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. Another line of research has focused on adapting pre-trained static models for DFER. In this work, we propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders. We identify main challenges associated with this task, namely, intra-modality adaptation, cross-modal alignment, and temporal adaptation, and propose solutions to each of them. As a result, we demonstrate improvement over current state-of-the-art on two popular DFER benchmarks, namely DFEW and MFAW.
Abstract (translated)
近年来，动态面部表情识别（DFER）由于其在实现富有同情心和人性化技术方面的关键作用而受到广泛关注。在DFER中实现对野外数据的稳健性对于现实应用尤为重要。旨在改善此类模型的一个方向是基于音频和视频数据的跨模态情感识别。在DFER的跨模态学习领域，最近的方法专注于利用自监督学习（SSL）预训练强大的多模态编码器。另一条研究路线专注于将预训练静态模型适应DFER。在这项工作中，我们提出了一个不同的观点来研究多模态DFER性能通过自监督预训练的分离单模态编码器进行调整。我们指出了这项任务中与该任务相关的主要挑战，即自模态适应、跨模态对齐和时间适应，并提出了针对每个问题的解决方案。结果，我们在两个流行的DFER基准上展示了改进，这两个基准分别是DFEW和MFAW。
URL
https://arxiv.org/abs/2404.09010
