Abstract
Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans and enhancing the capabilities of interactive virtual agents, with wide-ranging applications in virtual reality, digital entertainment, and remote communication. Existing approaches often generate audio-driven facial expressions and gestures independently, which introduces a significant limitation: the lack of seamless coordination between facial and gestural elements, resulting in less natural and cohesive animations. To address this limitation, we propose AsynFusion, a novel framework that leverages diffusion transformers to achieve harmonious expression and gesture synthesis. The proposed method is built upon a dual-branch DiT architecture, which enables the parallel generation of facial expressions and gestures. Within the model, we introduce a Cooperative Synchronization Module to facilitate bidirectional feature interaction between the two modalities, and an Asynchronous LCM Sampling strategy to reduce computational overhead while maintaining high-quality outputs. Extensive experiments demonstrate that AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations, consistently outperforming existing methods in both quantitative and qualitative evaluations.
Abstract (translated)
全身音频驱动的虚拟化身姿态和表情生成是创建逼真的数字人类并增强交互式虚拟代理能力的关键任务，其在虚拟现实、数字娱乐以及远程通信等领域有着广泛的应用。现有方法通常独立生成音频驱动的表情和手势，这带来了显著限制：面部与体态元素之间缺乏无缝协调，导致动画显得不自然且连贯性差。为解决这一问题，我们提出了AsynFusion框架，它利用扩散变换器实现和谐的表达和动作合成。该方法基于双分支DiT架构，能够在并行生成面部表情和手势的同时，通过引入合作同步模块促进两者之间双向特征交互，并采用异步LCM采样策略减少计算负担同时保持高质量输出。 实验结果表明，在实时同步全身动画生成方面，AsynFusion达到了目前的最佳性能，在定性和定量评估中均显著优于现有方法。
URL
https://arxiv.org/abs/2505.15058
