Abstract
Co-speech gestures play a vital role in non-verbal communication. In this paper, we introduce a new framework for co-speech gesture understanding in the wild. Specifically, we propose three new tasks and benchmarks to evaluate a model's capability to comprehend gesture-text-speech associations: (i) gesture-based retrieval, (ii) gestured word spotting, and (iii) active speaker detection using gestures. We present a new approach that learns a tri-modal speech-text-video-gesture representation to solve these tasks. By leveraging a combination of global phrase contrastive loss and local gesture-word coupling loss, we demonstrate that a strong gesture representation can be learned in a weakly supervised manner from videos in the wild. Our learned representations outperform previous methods, including large vision-language models (VLMs), across all three tasks. Further analysis reveals that speech and text modalities capture distinct gesture-related signals, underscoring the advantages of learning a shared tri-modal embedding space. The dataset, model, and code are available at: this https URL
Abstract (translated)
共同言语手势在非语言交流中扮演着重要角色。本文介绍了一种新的框架，用于理解自然环境中的共言语手势。具体而言，我们提出了三个新任务和基准测试来评估模型对手势-文本-语音关联的理解能力：(i) 基于手势的检索；(ii) 手势词检测；以及 (iii) 使用手势进行活跃说话人识别。我们提出了一种新的方法，通过学习一种三模态（语音、文本和视频）的手势表示来解决这些任务。通过结合全局短语对比损失和局部手势-单词耦合损失的组合，我们展示了如何在弱监督下从野外视频中以有效的方式学习到强大的手势表示。我们的所学表现在所有三项任务上均超越了之前的模型方法，包括大规模视觉语言模型（VLMs）。进一步分析显示，语音和文本模态捕获了不同的与手势相关的信号，这强调了在共享三模态嵌入空间中进行学习的优势。数据集、模型和代码可在以下链接获取：[此网址]。
URL
https://arxiv.org/abs/2503.22668
