Abstract
Co-Speech Gesture Video Generation aims to generate vivid speech videos from audio-driven still images, which is challenging due to the diversity of different parts of the body in terms of amplitude of motion, audio relevance, and detailed features. Relying solely on audio as the control signal often fails to capture large gesture movements in video, leading to more pronounced artifacts and distortions. Existing approaches typically address this issue by introducing additional a priori information, but this can limit the practical application of the task. Specifically, we propose a Motion Mask-Guided Two-Stage Network (MMGT) that uses audio, as well as motion masks and motion features generated from the audio signal to jointly drive the generation of synchronized speech gesture videos. In the first stage, the Spatial Mask-Guided Audio Pose Generation (SMGA) Network generates high-quality pose videos and motion masks from audio, effectively capturing large movements in key regions such as the face and gestures. In the second stage, we integrate the Motion Masked Hierarchical Audio Attention (MM-HAA) into the Stabilized Diffusion Video Generation model, overcoming limitations in fine-grained motion generation and region-specific detail control found in traditional methods. This guarantees high-quality, detailed upper-body video generation with accurate texture and motion details. Evaluations show improved video quality, lip-sync, and gesture. The model and code are available at this https URL.
Abstract (translated)
Co-Speech Gesture Video Generation（伴随言语的手势视频生成）旨在从音频驱动的静止图像中生成生动的讲话视频。由于身体各部位在运动幅度、与音频的相关性以及细节特征上的多样性，这一任务颇具挑战性。单纯依靠音频作为控制信号往往难以捕捉到视频中的大幅度手势动作，导致更为明显的伪影和失真现象。现有的方法通常通过引入额外的先验信息来解决此问题，但这可能会限制该任务的实际应用。 为了解决这些问题，我们提出了一种名为Motion Mask-Guided Two-Stage Network（MMGT）的方法。该网络不仅使用音频信号，还利用从音频信号生成的运动掩码和运动特征共同驱动同步言语手势视频的生成。在第一阶段中，Spatial Mask-Guided Audio Pose Generation（SMGA）网络从音频输入中生成高质量的姿态视频及运动掩码，有效捕捉面部与手势等关键区域的大范围动作变化。在第二阶段中，我们将Motion Masked Hierarchical Audio Attention（MM-HAA）整合进Stabilized Diffusion Video Generation模型之中，克服了传统方法在细粒度运动生成和局部细节控制方面的局限性。这确保了高质量的上半身视频生成，并能准确捕捉纹理及动作细节。 实验评估显示，该方法显著提升了视频质量、唇同步效果以及手势表现。相关模型和代码可在提供的链接中获取：[此URL]。
URL
https://arxiv.org/abs/2505.23120
