Abstract
While the field of co-speech gesture generation has seen significant advances, producing holistic, semantically grounded gestures remains a challenge. Existing approaches rely on external semantic retrieval methods, which limit their generalisation capability due to dependency on predefined linguistic rules. Flow-matching-based methods produce promising results; however, the network is optimised using only semantically congruent samples without exposure to negative examples, leading to learning rhythmic gestures rather than sparse motion, such as iconic and metaphoric gestures. Furthermore, by modelling body parts in isolation, the majority of methods fail to maintain crossmodal consistency. We introduce a Contrastive Flow Matching-based co-speech gesture generation model that uses mismatched audio-text conditions as negatives, training the velocity field to follow the correct motion trajectory while repelling semantically incongruent trajectories. Our model ensures cross-modal coherence by embedding text, audio, and holistic motion into a composite latent space via cosine and contrastive objectives. Extensive experiments and a user study demonstrate that our proposed approach outperforms state-of-the-art methods on two datasets, BEAT2 and SHOW.
Abstract (translated)
尽管共语音手势生成领域已取得显著进展，但生成整体性、语义合理的手势仍是一项挑战。现有方法依赖外部语义检索，因其依赖预定义语言规则而限制了泛化能力。基于流匹配的方法虽产生令人期待的结果，但网络仅使用语义一致的样本进行优化，缺乏对负样本的接触，导致其学习到的是节奏性手势而非稀疏运动（如图示性手势和隐喻性手势）。此外，多数方法因孤立建模身体部位而无法保持跨模态一致性。本文提出一种基于对比流匹配的共语音手势生成模型，利用不匹配的音频-文本条件作为负样本，训练速度场遵循正确的运动轨迹同时排斥语义不一致的轨迹。该模型通过余弦与对比目标将文本、音频和整体运动嵌入复合潜在空间，确保跨模态连贯性。大量实验与用户研究表明，所提方法在BEAT2和SHOW两个数据集上均优于现有最先进方法。
URL
https://arxiv.org/abs/2603.26553
