Abstract
Humans often experience not just a single basic emotion at a time, but rather a blend of several emotions with varying salience. Despite the importance of such blended emotions, most video-based emotion recognition approaches are designed to recognize single emotions only. The few approaches that have attempted to recognize blended emotions typically cannot assess the relative salience of the emotions within a blend. This limitation largely stems from the lack of datasets containing a substantial number of blended emotion samples annotated with relative salience. To address this shortcoming, we introduce BLEMORE, a novel dataset for multimodal (video, audio) blended emotion recognition that includes information on the relative salience of each emotion within a blend. BLEMORE comprises over 3,000 clips from 58 actors, performing 6 basic emotions and 10 distinct blends, where each blend has 3 different salience configurations (50/50, 70/30, and 30/70). Using this dataset, we conduct extensive evaluations of state-of-the-art video classification approaches on two blended emotion prediction tasks: (1) predicting the presence of emotions in a given sample, and (2) predicting the relative salience of emotions in a blend. Our results show that unimodal classifiers achieve up to 29% presence accuracy and 13% salience accuracy on the validation set, while multimodal methods yield clear improvements, with ImageBind + WavLM reaching 35% presence accuracy and HiCMAE 18% salience accuracy. On the held-out test set, the best models achieve 33% presence accuracy (VideoMAEv2 + HuBERT) and 18% salience accuracy (HiCMAE). In sum, the BLEMORE dataset provides a valuable resource to advancing research on emotion recognition systems that account for the complexity and significance of blended emotion expressions.
Abstract (translated)
人类在某一时刻通常体验的不是单一的基本情绪，而是一种混合了多种不同强度情感的状态。尽管这种复合情感的重要性不容忽视，但大多数基于视频的情绪识别方法仅设计用于识别单一情绪。少数尝试识别复合情感的方法通常也无法评估这些情感之间的相对显著性。这一局限主要源于缺少包含大量注有相对显著性的复合情感样本的数据集。 为了解决这个问题，我们引入了BLEMORE数据集，这是一个新的多模态（视频、音频）混合情绪识别数据集，包含了每种情绪在混合中的相对显著信息。BLEMORE包括超过3000个片段，来自58名演员的表演，涵盖了6种基本情感和10种不同的复合情感组合，其中每一种复合情感有三种不同的情感强度配置（50/50, 70/30 和 30/70）。利用这个数据集，我们对最先进的视频分类方法进行了广泛的评估，针对两个混合情绪预测任务：(1) 预测给定样本中是否存在特定情绪；(2) 预测情感混合中的相对显著性。我们的结果显示，在验证集中，单模态分类器在存在准确性上可达到最高29%，在显著性准确性上可达13%；而多模态方法则明显表现出改进，其中ImageBind + WavLM达到了35%的存在准确性，HiCMAE的显著性准确率为18%。在未公开的测试集中，最佳模型实现了VideoMAEv2 + HuBERT的33%存在准确性和HiCMAE的18%显著性准确度。 综上所述，BLEMORE数据集为推进考虑情感表达复杂性和重要性的混合情绪识别系统研究提供了宝贵的资源。
URL
https://arxiv.org/abs/2601.13225
