Abstract
Co-speech gesture generation is a critical area of research aimed at synthesizing speech-synchronized human-like gestures. Existing methods often suffer from issues such as rhythmic inconsistency, motion jitter, foot sliding and limited multi-sampling diversity. In this paper, we present SmoothSync, a novel framework that leverages quantized audio tokens in a novel dual-stream Diffusion Transformer (DiT) architecture to synthesis holistic gestures and enhance sampling variation. Specifically, we (1) fuse audio-motion features via complementary transformer streams to achieve superior synchronization, (2) introduce a jitter-suppression loss to improve temporal smoothness, (3) implement probabilistic audio quantization to generate distinct gesture sequences from identical inputs. To reliably evaluate beat synchronization under jitter, we introduce Smooth-BC, a robust variant of the beat consistency metric less sensitive to motion noise. Comprehensive experiments on the BEAT2 and SHOW datasets demonstrate SmoothSync's superiority, outperforming state-of-the-art methods by -30.6% FGD, 10.3% Smooth-BC, and 8.4% Diversity on BEAT2, while reducing jitter and foot sliding by -62.9% and -17.1% respectively. The code will be released to facilitate future research.
Abstract (translated)
本文介绍了一种名为SmoothSync的新型框架，该框架旨在解决同步生成人类样貌手势过程中存在的诸如节奏不一致、动作抖动和脚步滑移等问题。SmoothSync利用新颖的双流Diffusion Transformer（DiT）架构中的量化音频令牌来合成整体手势，并增强采样的多样性。 具体而言，我们采用了以下方法： 1. 通过互补的Transformer流融合音频-运动特征以实现更优同步。 2. 引入抖动抑制损失函数，提升时间上的平滑度。 3. 实现概率性音频量化技术，从相同输入生成不同的手势序列。 为了可靠地评估在抖动情况下的节拍同步，我们引入了Smooth-BC，这是一种对运动噪声不敏感的节拍一致性指标的稳健变体。在BEAT2和SHOW数据集上的全面实验表明，与当前最佳方法相比，SmoothSync的表现更优，在BEAT2数据集中，它分别减少了30.6% FGD（固定手势差异）、提高了10.3% Smooth-BC以及增加了8.4%的多样性，并且在减少抖动和脚步滑移方面分别改善了62.9%和17.1%。未来的研究人员可以获取该代码以进一步开展相关研究工作。
URL
https://arxiv.org/abs/2601.04236
