Abstract
Micro-expressions (MEs) are crucial psychological responses with significant potential for affective computing. However, current automatic micro-expression recognition (MER) research primarily focuses on discrete emotion classification, neglecting a convincing analysis of the subtle dynamic movements and inherent emotional cues. The rapid progress in multimodal large language models (MLLMs), known for their strong multimodal comprehension and language generation abilities, offers new possibilities. MLLMs have shown success in various vision-language tasks, indicating their potential to understand MEs comprehensively, including both fine-grained motion patterns and underlying emotional semantics. Nevertheless, challenges remain due to the subtle intensity and short duration of MEs, as existing MLLMs are not designed to capture such delicate frame-level facial dynamics. In this paper, we propose a novel Micro-Expression Large Language Model (MELLM), which incorporates a subtle facial motion perception strategy with the strong inference capabilities of MLLMs, representing the first exploration of MLLMs in the domain of ME analysis. Specifically, to explicitly guide the MLLM toward motion-sensitive regions, we construct an interpretable motion-enhanced color map by fusing onset-apex optical flow dynamics with the corresponding grayscale onset frame as the model input. Additionally, specialized fine-tuning strategies are incorporated to further enhance the model's visual perception of MEs. Furthermore, we construct an instruction-description dataset based on Facial Action Coding System (FACS) annotations and emotion labels to train our MELLM. Comprehensive evaluations across multiple benchmark datasets demonstrate that our model exhibits superior robustness and generalization capabilities in ME understanding (MEU). Code is available at this https URL.
Abstract (translated)
微表情（MEs）是具有重要情感计算潜力的关键心理反应。然而，当前的自动微表情识别（MER）研究主要集中在离散情绪分类上，忽略了对细微动态运动和内在情绪线索进行深入分析的问题。多模态大型语言模型（MLLMs），以其强大的跨模态理解和语言生成能力而闻名，在许多视觉-语言任务中表现出色，显示出全面理解MEs的潜力，包括精细的动作模式和潜在的情感语义。然而，由于微表情的细微强度和短暂持续时间，现有的MLLMs尚未设计为捕捉这种微妙的画面级面部动态变化。因此，本文提出了一种新颖的微表情大型语言模型（MELLM），该模型结合了细致面部运动感知策略与MLLM强大的推理能力，首次探索了MLLM在ME分析领域的应用。 具体而言，为了明确指导MLLM关注敏感的动作区域，我们构建了一个可解释的增强色彩地图，通过融合起始-峰值光学流动态和相应的灰度起始帧作为模型输入。此外，还引入了专门的微调策略，以进一步提升模型对微表情视觉感知的能力。另外，基于面部动作编码系统（FACS）注释和情绪标签构建了一个指令描述数据集来训练我们的MELLM。 在多个基准测试数据集上的全面评估表明，我们的模型在微表情理解（MEU）方面表现出色的鲁棒性和泛化能力。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2505.07007
