Abstract
Creating a virtual avatar with semantically coherent gestures that are aligned with speech is a challenging task. Existing gesture generation research mainly focused on generating rhythmic beat gestures, neglecting the semantic context of the gestures. In this paper, we propose a novel approach for semantic grounding in co-speech gesture generation that integrates semantic information at both fine-grained and global levels. Our approach starts with learning the motion prior through a vector-quantized variational autoencoder. Built on this model, a second-stage module is applied to automatically generate gestures from speech, text-based semantics and speaker identity that ensures consistency between the semantic relevance of generated gestures and co-occurring speech semantics through semantic coherence and relevance modules. Experimental results demonstrate that our approach enhances the realism and coherence of semantic gestures. Extensive experiments and user studies show that our method outperforms state-of-the-art approaches across two benchmarks in co-speech gesture generation in both objective and subjective metrics. The qualitative results of our model, code, dataset and pre-trained models can be viewed at this https URL.
Abstract (translated)
创建一个与说话内容语义一致、动作连贯的虚拟化身是一项具有挑战性的任务。现有的手势生成研究主要集中在产生节奏性拍打手势上，而忽视了手势的语义背景。在本文中，我们提出了一种新颖的方法来解决伴随言语的手势生成中的语义关联问题，这种方法在整个层次结构（从细粒度到全局）整合了语义信息。我们的方法首先通过向量量化变分自动编码器学习运动先验知识。在此模型基础上，第二阶段模块会根据语音、文本语义和说话者身份自动生成手势，并确保生成的手势与伴随出现的言语语义在语境一致性和相关性方面保持一致性。实验结果表明，我们的方法提升了语义手势的真实感和连贯性。广泛的实验证明，在两个基准数据集上，我们提出的方法无论是在客观指标还是主观评价中均优于当前最先进的方法。我们的模型、代码、数据集及预训练模型的相关定性分析可以在[这个链接](https://this-url.com)查看。
URL
https://arxiv.org/abs/2507.19359
