Abstract
Human emotional expression emerges through coordinated vocal, facial, and gestural signals. While speech face alignment is well established, the broader dynamics linking emotionally expressive speech to regional facial and hand motion remains critical for gaining a deeper insight into how emotional and behavior cues are communicated in real interactions. Further modulating the coordination is the structure of conversational exchange like sequential turn taking, which creates stable temporal windows for multimodal synchrony, and simultaneous speech, often indicative of high arousal moments, disrupts this alignment and impacts emotional clarity. Understanding these dynamics enhances realtime emotion detection by improving the accuracy of timing and synchrony across modalities in both human interactions and AI systems. This study examines multimodal emotion coupling using region specific motion capture from dyadic interactions in the IEMOCAP corpus. Speech features included low level prosody, MFCCs, and model derived arousal, valence, and categorical emotions (Happy, Sad, Angry, Neutral), aligned with 3D facial and hand marker displacements. Expressive activeness was quantified through framewise displacement magnitudes, and speech to gesture prediction mapped speech features to facial and hand movements. Nonoverlapping speech consistently elicited greater activeness particularly in the lower face and mouth. Sadness showed increased expressivity during nonoverlap, while anger suppressed gestures during overlaps. Predictive mapping revealed highest accuracy for prosody and MFCCs in articulatory regions while arousal and valence had lower and more context sensitive correlations. Notably, hand speech synchrony was enhanced under low arousal and overlapping speech, but not for valence.
Abstract (translated)
人类情感表达通过协调的语音、面部和手势信号实现。虽然言语与面部表情的一致性已经得到了很好的研究，但将情感丰富的言语与特定区域的面部运动及手部动作联系起来的动力学特性对于深入理解情感和行为线索在实际互动中的传达方式至关重要。 进一步影响这种协调的是对话结构，例如轮换发言模式，在这些模式中，多模态同步会形成稳定的时序窗口。而同时说话的情况通常表明高度兴奋的状态，这会破坏这种一致性和影响情绪清晰度。理解这些动力学有助于提高实时情感检测的准确性，特别是在人类互动和人工智能系统中的不同模态之间的时间和同步性方面。 这项研究使用了IEMOCAP语料库中一对多交互的区域特定运动捕捉数据来考察跨模式情感耦合。语音特征包括低层声调、梅尔频率倒谱系数（MFCCs）以及通过模型得到的情绪唤醒度、效价及分类情绪（快乐、悲伤、愤怒和中立），与3D面部和手部标记位移对齐。表达积极性是通过对帧级别的位移幅度进行量化来测量的，同时将语音特征映射到面部和手部运动以预测手势。 研究发现，在非重叠说话期间，下颌和嘴唇区域表现出更高的活性。悲伤情绪在非重叠时间窗口中表现为更强的情感表现力；而愤怒则会抑制重叠说话时的手势。预测映射显示声调和MFCCs在发音区域具有最高的准确性，而唤醒度和效价的相关性较低且更依赖于具体语境。 值得注意的是，在低唤醒状态下以及言语重叠期间，手部运动与语音同步的增强效果明显；而在效价值方面则没有观察到类似效应。
URL
https://arxiv.org/abs/2506.10010
