Abstract
To tackle the automatic recognition of "concealed emotions" in videos, this paper proposes a multimodal weak-supervision framework and achieves state-of-the-art results on the iMiGUE tennis-interview dataset. First, YOLO 11x detects and crops human portraits frame-by-frame, and DINOv2-Base extracts visual features from the cropped regions. Next, by integrating Chain-of-Thought and Reflection prompting (CoT + Reflection), Gemini 2.5 Pro automatically generates pseudo-labels and reasoning texts that serve as weak supervision for downstream models. Subsequently, OpenPose produces 137-dimensional key-point sequences, augmented with inter-frame offset features; the usual graph neural network backbone is simplified to an MLP to efficiently model the spatiotemporal relationships of the three key-point streams. An ultra-long-sequence Transformer independently encodes both the image and key-point sequences, and their representations are concatenated with BERT-encoded interview transcripts. Each modality is first pre-trained in isolation, then fine-tuned jointly, with pseudo-labeled samples merged into the training set for further gains. Experiments demonstrate that, despite severe class imbalance, the proposed approach lifts accuracy from under 0.6 in prior work to over 0.69, establishing a new public benchmark. The study also validates that an "MLP-ified" key-point backbone can match - or even surpass - GCN-based counterparts in this task.
Abstract (translated)
为了应对视频中“隐蔽情绪”的自动识别问题，本文提出了一种多模态弱监督框架，并在iMiGUE网球访谈数据集上取得了最先进的成果。首先，YOLO 11x每帧检测并裁剪人物肖像，DINOv2-Base从裁剪区域提取视觉特征。接着，通过整合Chain-of-Thought和反思提示（CoT + Reflection），Gemini 2.5 Pro自动生成伪标签和推理文本，这些作为下游模型的弱监督信号。随后，OpenPose生成137维关键点序列，并加入帧间偏移特征；通常用于建模时空关系的关键点流图神经网络骨干被简化为MLP（多层感知机），以更高效地处理这一任务。一个超长序列Transformer独立编码图像和关键点序列，然后将它们的表示与BERT编码后的访谈记录拼接起来。每种模式首先单独预训练，之后共同微调，并将伪标签样本合并到训练集中以进一步提升性能。实验表明，在严重类不平衡的情况下，所提出的方法使准确性从先前研究中的不足0.6提高到了超过0.69的新水平，确立了一个新的公共基准标准。此外，该研究表明，“MLP化”的关键点骨干网络可以与基于GCN（图卷积网络）的同类方法相匹敌甚至超越。
URL
https://arxiv.org/abs/2602.08057
