Abstract
Diffusion models have demonstrated remarkable synthesis quality and diversity in generating co-speech gestures. However, the computationally intensive sampling steps associated with diffusion models hinder their practicality in real-world applications. Hence, we present DIDiffGes, for a Decoupled Semi-Implicit Diffusion model-based framework, that can synthesize high-quality, expressive gestures from speech using only a few sampling steps. Our approach leverages Generative Adversarial Networks (GANs) to enable large-step sampling for diffusion model. We decouple gesture data into body and hands distributions and further decompose them into marginal and conditional distributions. GANs model the marginal distribution implicitly, while L2 reconstruction loss learns the conditional distributions exciplictly. This strategy enhances GAN training stability and ensures expressiveness of generated full-body gestures. Our framework also learns to denoise root noise conditioned on local body representation, guaranteeing stability and realism. DIDiffGes can generate gestures from speech with just 10 sampling steps, without compromising quality and expressiveness, reducing the number of sampling steps by a factor of 100 compared to existing methods. Our user study reveals that our method outperforms state-of-the-art approaches in human likeness, appropriateness, and style correctness. Project is this https URL.
Abstract (translated)
扩散模型在生成与语音同步的手势时展示了卓越的合成质量和多样性。然而，与扩散模型相关的计算密集型采样步骤阻碍了它们在现实世界应用中的实用性。为此，我们提出了 DIDiffGes——一个基于解耦半隐式扩散模型框架的方法，能够仅通过少量采样步骤就从语音中生成高质量且富有表现力的手势。我们的方法利用生成对抗网络（GANs）来实现大步长采样的扩散模型。我们将手势数据分解为主体和手部分布，并进一步将其拆分为边缘分布和条件分布。GAN 模型隐式地模拟边缘分布，而 L2 重构损失则显式学习条件分布。这一策略增强了 GAN 训练的稳定性并确保生成的整体身体手势具有表现力。我们的框架还学会了根据局部身体表示对根噪声进行去噪，以保证稳定性和现实性。DIDiffGes 能够仅用10个采样步骤就从语音中生成手势，并且不会牺牲质量和表现力，相比现有方法将采样步骤减少了100倍。我们的用户研究表明，我们提出的方法在人类相似度、适当性和风格正确性方面超过了现有的先进方法。 项目详情请参见：[此链接](https://this.is.the.url.of.project)（这里需要替换为实际的项目网址）。
URL
https://arxiv.org/abs/2503.17059
