Abstract
Human communication combines speech with expressive nonverbal cues such as hand gestures that serve manifold communicative functions. Yet, current generative gesture generation approaches are restricted to simple, repetitive beat gestures that accompany the rhythm of speaking but do not contribute to communicating semantic meaning. This paper tackles a core challenge in co-speech gesture synthesis: generating iconic or deictic gestures that are semantically coherent with a verbal utterance. Such gestures cannot be derived from language input alone, which inherently lacks the visual meaning that is often carried autonomously by gestures. We therefore introduce a zero-shot system that generates gestures from a given language input and additionally is informed by imagistic input, without manual annotation or human intervention. Our method integrates an image analysis pipeline that extracts key object properties such as shape, symmetry, and alignment, together with a semantic matching module that links these visual details to spoken text. An inverse kinematics engine then synthesizes iconic and deictic gestures and combines them with co-generated natural beat gestures for coherent multimodal communication. A comprehensive user study demonstrates the effectiveness of our approach. In scenarios where speech alone was ambiguous, gestures generated by our system significantly improved participants' ability to identify object properties, confirming their interpretability and communicative value. While challenges remain in representing complex shapes, our results highlight the importance of context-aware semantic gestures for creating expressive and collaborative virtual agents or avatars, marking a substantial step forward towards efficient and robust, embodied human-agent interaction. More information and example videos are available here: this https URL
Abstract (translated)
人类交流结合了言语与表达性非语言线索，如手势，这些手势在沟通中具有多种功能。然而，当前的生成式手势生成方法仅限于伴随说话节奏出现的简单、重复的手势，并且不参与传递语义意义。本文解决了一个核心挑战：如何根据口头表述合成具象化或指示性的手势，这些手势与语言输入本身所缺乏的视觉含义密切相关。 我们因此引入了一种零样本系统，该系统从给定的语言输入中生成手势，并通过图象输入进一步完善其功能，而无需人工注释或人为干预。我们的方法整合了一个图像分析管道，提取形状、对称性和对齐等关键物体属性，以及一个语义匹配模块将这些视觉细节与口语文本连接起来。随后，逆向动力学引擎会合成具象化和指示性的手势，并将其与自动生成的自然节奏手势结合，以实现连贯的多模态沟通。 一项全面的用户研究证明了我们方法的有效性。在仅凭言语无法明确传达信息的情况下，系统生成的手势显著提升了参与者识别物体属性的能力，从而证实了其可解释性和交流价值。虽然代表复杂形状仍有挑战，但我们的成果强调了语境感知手势对于创造具有表现力和协作性的虚拟代理或化身的重要性，这标志着朝向高效且稳健的人机互动迈出了重要一步。 更多信息及示例视频请参见：[提供的链接]
URL
https://arxiv.org/abs/2510.17617
