Abstract
Despite the recent progress of audio-driven video generation, existing methods mostly focus on driving facial movements, leading to non-coherent head and body dynamics. Moving forward, it is desirable yet challenging to generate holistic human videos with both accurate lip-sync and delicate co-speech gestures w.r.t. given audio. In this work, we propose AudCast, a generalized audio-driven human video generation framework adopting a cascade Diffusion-Transformers (DiTs) paradigm, which synthesizes holistic human videos based on a reference image and a given audio. 1) Firstly, an audio-conditioned Holistic Human DiT architecture is proposed to directly drive the movements of any human body with vivid gesture dynamics. 2) Then to enhance hand and face details that are well-knownly difficult to handle, a Regional Refinement DiT leverages regional 3D fitting as the bridge to reform the signals, producing the final results. Extensive experiments demonstrate that our framework generates high-fidelity audio-driven holistic human videos with temporal coherence and fine facial and hand details. Resources can be found at this https URL.
Abstract (translated)
尽管最近在音频驱动的视频生成方面取得了进展，现有的方法大多集中在驱动面部动作上，导致头部和身体动态不一致。向前发展，根据给定的音频同时生成准确的唇部同步和细腻的伴随手势的完整人体视频是令人向往但具有挑战性的目标。在这项工作中，我们提出了AudCast，这是一个通用的音频驱动的人体视频生成框架，采用级联扩散-变压器（DiTs）范式，在参考图像和给定音频的基础上合成完整的身体动作。 1) 首先，提出了一种基于音频条件的整体人体DiT架构，可以直接控制任何人类的身体动作，并产生生动的手势动态。 2) 为了增强手部和面部的细节处理，众所周知这些是难以处理的部分，区域细化DiT利用局部3D拟合作为信号重制的桥梁，生成最终的结果。 广泛的实验表明，我们的框架能够生成高保真度的音频驱动的整体人体视频，并具有时间一致性以及细致的手部和面部特征。更多资源可以在这里找到：[此URL]（请将“this https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2503.19824
