Abstract
Co-speech gesture generation enhances human-computer interaction realism through speech-synchronized gesture synthesis. However, generating semantically meaningful gestures remains a challenging problem. We propose SARGes, a novel framework that leverages large language models (LLMs) to parse speech content and generate reliable semantic gesture labels, which subsequently guide the synthesis of meaningful co-speech this http URL, we constructed a comprehensive co-speech gesture ethogram and developed an LLM-based intent chain reasoning mechanism that systematically parses and decomposes gesture semantics into structured inference steps following ethogram criteria, effectively guiding LLMs to generate context-aware gesture labels. Subsequently, we constructed an intent chain-annotated text-to-gesture label dataset and trained a lightweight gesture label generation model, which then guides the generation of credible and semantically coherent co-speech gestures. Experimental results demonstrate that SARGes achieves highly semantically-aligned gesture labeling (50.2% accuracy) with efficient single-pass inference (0.4 seconds). The proposed method provides an interpretable intent reasoning pathway for semantic gesture synthesis.
Abstract (translated)
协同语音手势生成通过同步语音的手势合成增强了人机交互的逼真度，但生成语义有意义的手势仍然是一个具有挑战性的问题。我们提出了SARGes（Speech-Associated Reliable Gesture Synthesis）这一新颖框架，该框架利用大型语言模型（LLMs）解析语音内容，并生成可靠的语义手势标签，随后指导生成意义明确的协同语音手势。 具体而言，我们构建了一个全面的手势图谱（co-speech gesture ethogram），并开发了一种基于LLM的意图链推理机制。这一机制系统地按照手势图谱的标准解析和拆分手势语义为结构化的推理步骤，有效引导LLMs生成情境感知的手势标签。随后，我们建立了一个带有意图链标注的文本到手势标签的数据集，并训练了轻量级的手势标签生成模型，该模型之后指导生成可信且语义连贯的协同语音手势。 实验结果表明，SARGes在手势标记上的语义对齐度达到了50.2%的准确率，并实现了高效的单次推理（0.4秒）。所提出的方法提供了一条可解释的意图推理路径，用于语义手势合成。
URL
https://arxiv.org/abs/2503.20202
