Abstract
Micro-gesture recognition and behavior-based emotion prediction are both highly challenging tasks that require modeling subtle, fine-grained human behaviors, primarily leveraging video and skeletal pose data. In this work, we present two multimodal frameworks designed to tackle both problems on the iMiGUE dataset. For micro-gesture classification, we explore the complementary strengths of RGB and 3D pose-based representations to capture nuanced spatio-temporal patterns. To comprehensively represent gestures, video, and skeletal embeddings are extracted using MViTv2-S and 2s-AGCN, respectively. Then, they are integrated through a Cross-Modal Token Fusion module to combine spatial and pose information. For emotion recognition, our framework extends to behavior-based emotion prediction, a binary classification task identifying emotional states based on visual cues. We leverage facial and contextual embeddings extracted using SwinFace and MViTv2-S models and fuse them through an InterFusion module designed to capture emotional expressions and body gestures. Experiments conducted on the iMiGUE dataset, within the scope of the MiGA 2025 Challenge, demonstrate the robust performance and accuracy of our method in the behavior-based emotion prediction task, where our approach secured 2nd place.
Abstract (translated)
微手势识别和基于行为的情绪预测都是极具挑战性的任务，需要对细微的人体动作进行建模。这些任务主要依赖于视频和骨骼姿态数据。在这项工作中，我们提出了两个多模式框架，旨在解决iMiGUE数据集上的这两个问题。 对于微手势分类，我们探索了RGB表示与基于3D姿势的表示互补的优势，以捕捉细腻的空间-时间模式。为了全面表示手势、视频和骨骼嵌入，分别使用MViTv2-S模型提取视频特征，并利用2s-AGCN模型提取骨骼姿态数据。然后，通过跨模态标记融合模块将空间信息与姿态信息相结合。 在情绪识别方面，我们的框架扩展到了基于行为的情绪预测任务上，这是一个二分类问题，旨在根据视觉线索识别情感状态。我们使用SwinFace和MViTv2-S模型提取面部和上下文嵌入，并利用一个设计用于捕捉面部表情和肢体动作的InterFusion模块进行融合。 在MiGA 2025挑战赛中针对iMiGUE数据集进行的实验表明，我们的方法在基于行为的情绪预测任务上展现了强大的性能和准确性。我们在该比赛的任务中取得了第二名的成绩。
URL
https://arxiv.org/abs/2512.23291
