Abstract
Micro-gestures are unconsciously performed body gestures that can convey the emotion states of humans and start to attract more research attention in the fields of human behavior understanding and affective computing as an emerging topic. However, the modeling of human emotion based on micro-gestures has not been explored sufficiently. In this work, we propose to recognize the emotion states based on the micro-gestures by reconstructing the behavior patterns with a hypergraph-enhanced Transformer in a hybrid-supervised framework. In the framework, hypergraph Transformer based encoder and decoder are separately designed by stacking the hypergraph-enhanced self-attention and multiscale temporal convolution modules. Especially, to better capture the subtle motion of micro-gestures, we construct a decoder with additional upsampling operations for a reconstruction task in a self-supervised learning manner. We further propose a hypergraph-enhanced self-attention module where the hyperedges between skeleton joints are gradually updated to present the relationships of body joints for modeling the subtle local motion. Lastly, for exploiting the relationship between the emotion states and local motion of micro-gestures, an emotion recognition head from the output of encoder is designed with a shallow architecture and learned in a supervised way. The end-to-end framework is jointly trained in a one-stage way by comprehensively utilizing self-reconstruction and supervision information. The proposed method is evaluated on two publicly available datasets, namely iMiGUE and SMG, and achieves the best performance under multiple metrics, which is superior to the existing methods.
Abstract (translated)
微手势是人类无意识中执行的身体动作，能够传达人的情感状态，在人类行为理解和情感计算领域作为一个新兴话题开始吸引更多的研究关注。然而，基于微手势的人类情绪建模尚未被充分探索。在本工作中，我们提出了一种通过使用增强的超图Transformer在一个混合监督框架内重构行为模式来识别基于微手势的情绪状态的方法。在这个框架中，超图Transformer编码器和解码器分别由堆叠的超图增强自注意力模块和多尺度时间卷积模块设计而成。 特别地，为了更好地捕捉微手势中的细微运动，我们在解码器中增加了额外的上采样操作以在自我监督学习模式下进行重构任务。我们还提出了一种增强的超图自注意模块，在该模块中骨架关节之间的超边逐渐更新，以呈现身体关节之间的关系，从而用于建模局部细小动作。 最后，为了利用情绪状态与微手势局部运动之间关系，我们在编码器输出处设计了一个情感识别头，并采用浅层架构并通过监督方式学习。整个端到端框架通过全面利用自我重构和监督信息在一次阶段联合训练中进行训练。所提出的方法在两个公开可用的数据集iMiGUE和SMG上进行了评估，在多个指标下达到了最佳性能，优于现有方法。 这段文字详细介绍了基于微手势的情感状态识别新方法，包括其新颖的超图增强Transformer框架、用于捕获细微运动的解码器设计以及如何通过编码器输出来学习情绪与动作之间的关系。该方法在两个公开数据集上的评估结果表明了它相对于当前技术的优势。
URL
https://arxiv.org/abs/2507.14867
