Abstract
Co-speech gesture video generation aims to synthesize realistic, audio-aligned videos of speakers, complete with synchronized facial expressions and body gestures. This task presents challenges due to the significant one-to-many mapping between audio and visual content, further complicated by the scarcity of large-scale public datasets and high computational demands. We propose a lightweight framework that utilizes 2D full-body skeletons as an efficient auxiliary condition to bridge audio signals with visual outputs. Our approach introduces a diffusion model conditioned on fine-grained audio segments and a skeleton extracted from the speaker's reference image, predicting skeletal motions through skeleton-audio feature fusion to ensure strict audio coordination and body shape consistency. The generated skeletons are then fed into an off-the-shelf human video generation model with the speaker's reference image to synthesize high-fidelity videos. To democratize research, we present CSG-405-the first public dataset with 405 hours of high-resolution videos across 71 speech types, annotated with 2D skeletons and diverse speaker demographics. Experiments show that our method exceeds state-of-the-art approaches in visual quality and synchronization while generalizing across speakers and contexts.
Abstract (translated)
演讲伴随的手势视频生成旨在合成与音频同步的、包含面部表情和身体动作的真实人物视频。这项任务面临着挑战，因为语音与视觉内容之间存在显著的一对多映射关系，并且由于大规模公开数据集的缺乏以及高昂的计算需求而变得更加复杂。我们提出了一种轻量级框架，该框架利用2D全身骨架作为高效的辅助条件来连接音频信号和视觉输出。我们的方法引入了一个基于细粒度音频片段并结合从演讲者参考图像中提取的骨骼信息进行条件化的扩散模型，通过融合骨骼-音频特征预测骨骼运动以确保严格的音视频同步和身体形状的一致性。生成的骨架随后被输入到现有的人脸视频生成模型中，并与演讲者的参考图像一起使用来合成高保真度的视频。 为了促进研究民主化，我们提供了CSG-405——这是第一个公开的数据集，包含71种不同类型的高质量语音，总时长为405小时，并且标注有2D骨架和多样化的说话人人口统计信息。实验表明，我们的方法在视觉质量和同步性方面超过了现有最佳的方法，并且能够跨说话者和上下文进行泛化。
URL
https://arxiv.org/abs/2507.06812
