Abstract
Audio-driven talking head generation is critical for applications such as virtual assistants, video games, and films, where natural lip movements are essential. Despite progress in this field, challenges remain in producing both consistent and realistic facial animations. Existing methods, often based on GANs or UNet-based diffusion models, face three major limitations: (i) temporal jittering caused by weak temporal constraints, resulting in frame inconsistencies; (ii) identity drift due to insufficient 3D information extraction, leading to poor preservation of facial identity; and (iii) unnatural blinking behavior due to inadequate modeling of realistic blink dynamics. To address these issues, we propose MoDiT, a novel framework that combines the 3D Morphable Model (3DMM) with a Diffusion-based Transformer. Our contributions include: (i) A hierarchical denoising strategy with revised temporal attention and biased self/cross-attention mechanisms, enabling the model to refine lip synchronization and progressively enhance full-face coherence, effectively mitigating temporal jittering. (ii) The integration of 3DMM coefficients to provide explicit spatial constraints, ensuring accurate 3D-informed optical flow prediction and improved lip synchronization using Wav2Lip results, thereby preserving identity consistency. (iii) A refined blinking strategy to model natural eye movements, with smoother and more realistic blinking behaviors.
Abstract (translated)
音频驱动的面部动画生成对于虚拟助手、视频游戏和电影等应用至关重要，特别是在这些场景中自然的嘴唇动作是必不可少的。尽管该领域已经取得了一些进展，但在产生既连贯又逼真的面部动画方面仍然面临着挑战。现有的方法通常基于GAN或UNet扩散模型，面临三大限制：(i) 由于时间约束较弱导致的时间抖动问题，这会导致帧间不一致性；(ii) 因为提取的3D信息不足而导致的身份漂移，从而影响面部身份的一致性保持；以及(iii) 因为对自然眨眼动态建模不足而造成的不自然眨眼行为。为了克服这些挑战，我们提出了MoDiT这一新颖框架，它结合了三维可变形模型（3DMM）和基于扩散的Transformer。 我们的贡献包括： - 一种分层去噪策略，采用修正的时间注意力机制和偏置自/交叉注意力机制，使得模型能够优化唇部同步，并逐步增强全脸的一致性，有效缓解时间抖动问题。 - 将3DMM系数整合进来以提供明确的空间约束条件，确保准确的基于3D信息的光流预测，并利用Wav2Lip的结果改进唇部同步，从而保持身份一致性。 - 一种优化的眨眼策略用于模拟自然的眼部运动，使得眨眼行为更加流畅和逼真。 通过这些创新方法，MoDiT能够更有效地解决现有技术面临的问题，生成既连贯又自然的面部动画。
URL
https://arxiv.org/abs/2507.05092
