Abstract
Gestures are essential for enhancing co-speech communication, offering visual emphasis and complementing verbal interactions. While prior work has concentrated on point-level motion or fully supervised data-driven methods, we focus on co-speech gestures, advocating for weakly supervised learning and pixel-level motion deviations. We introduce a weakly supervised framework that learns latent representation deviations, tailored for co-speech gesture video generation. Our approach employs a diffusion model to integrate latent motion features, enabling more precise and nuanced gesture representation. By leveraging weakly supervised deviations in latent space, we effectively generate hand gestures and mouth movements, crucial for realistic video production. Experiments show our method significantly improves video quality, surpassing current state-of-the-art techniques.
Abstract (translated)
手势对于增强伴随言语的交流至关重要，它提供了视觉上的强调，并补充了口头互动。尽管之前的工作主要集中在点级别的运动或完全监督的数据驱动方法上，但我们专注于伴随言语的手势，并倡导使用弱监督学习和像素级别运动偏差的方法。我们介绍了一种弱监督框架，该框架能够学习潜在表示偏差，专门用于生成伴随言语手势的视频。 我们的方法采用扩散模型来整合潜在的运动特征，从而实现更精确、细腻的手势表现。通过在潜在空间中利用弱监督偏差，我们有效生成了手部动作和嘴部运动，这对于真实的视频制作至关重要。 实验结果表明，与当前最先进的技术相比，我们的方法显著提升了视频的质量。
URL
https://arxiv.org/abs/2503.21616
