Abstract
Along with the explosion of large language models, improvements in speech synthesis, advancements in hardware, and the evolution of computer graphics, the current bottleneck in creating digital humans lies in generating character movements that correspond naturally to text or speech inputs. In this work, we present DeepGesture, a diffusion-based gesture synthesis framework for generating expressive co-speech gestures conditioned on multimodal signals-text, speech, emotion, and seed motion. Built upon the DiffuseStyleGesture model, DeepGesture introduces novel architectural enhancements that improve semantic alignment and emotional expressiveness in generated gestures. Specifically, we integrate fast text transcriptions as semantic conditioning and implement emotion-guided classifier-free diffusion to support controllable gesture generation across affective states. A lightweight Transformer backbone combines full self-attention and cross-local attention for effective feature fusion of heterogeneous modalities. To visualize results, we implement a full rendering pipeline in Unity based on BVH output from the model. Evaluation on the ZeroEGGS dataset shows that DeepGesture produces gestures with improved human-likeness and contextual appropriateness, outperforming baselines on Mean Opinion Score and Frechet Gesture Distance metrics. Our system supports interpolation between emotional states and demonstrates generalization to out-of-distribution speech, including synthetic voices-marking a step forward toward fully multimodal, emotionally aware digital humans.
Abstract (translated)
随着大型语言模型的爆炸式发展、语音合成技术的进步、硬件设备的提升以及计算机图形学的发展，目前在创建数字人类时遇到的一个瓶颈是如何生成与文本或语音输入自然对应的肢体动作。在此研究中，我们介绍了DeepGesture，这是一种基于扩散模型的手势合成框架，能够根据多模态信号（包括文本、语音、情感和初始运动）生成具有表现力的伴随言语手势。 DeepGesture 是在 DiffuseStyleGesture 模型的基础上构建的，并引入了新颖的架构改进以提高生成的手势在语义对齐和情绪表达方面的性能。具体来说，我们整合了快速文字转录作为语义条件输入，并实现了情感引导的无分类器扩散方法来支持跨情绪状态的情感可控手势生成。一个轻量级的Transformer骨干网络结合了完整的自注意力机制与局部交叉注意力，以有效地融合异构模态特征。 为了展示结果，我们在 Unity 中实现了一个基于模型输出 BVH 文件的完整渲染管线。在 ZeroEGGS 数据集上的评估显示，DeepGesture 能够生成更加逼真且符合语境的手势，在平均意见评分和Frechet手势距离等指标上超越了基准方法。我们的系统支持在情绪状态之间进行插值，并展示了对分布外语音（包括合成声音）的泛化能力——这标志着向完全多模态、情感感知数字人类迈进了一步。
URL
https://arxiv.org/abs/2507.03147
