Abstract
Generating consecutive images of lip movements that align with a given speech in audio-driven lip synthesis is a challenging task. While previous studies have made strides in synchronization and visual quality, lip intelligibility and video fluency remain persistent challenges. This work proposes FluentLip, a two-stage approach for audio-driven lip synthesis, incorporating three featured strategies. To improve lip synchronization and intelligibility, we integrate a phoneme extractor and encoder to generate a fusion of audio and phoneme information for multimodal learning. Additionally, we employ optical flow consistency loss to ensure natural transitions between image frames. Furthermore, we incorporate a diffusion chain during the training of Generative Adversarial Networks (GANs) to improve both stability and efficiency. We evaluate our proposed FluentLip through extensive experiments, comparing it with five state-of-the-art (SOTA) approaches across five metrics, including a proposed metric called Phoneme Error Rate (PER) that evaluates lip pose intelligibility and video fluency. The experimental results demonstrate that our FluentLip approach is highly competitive, achieving significant improvements in smoothness and naturalness. In particular, it outperforms these SOTA approaches by approximately $\textbf{16.3%}$ in Fréchet Inception Distance (FID) and $\textbf{35.2%}$ in PER.
Abstract (translated)
生成与给定音频中的演讲同步的连续唇部动作图像，以进行基于音频驱动的唇形合成是一项具有挑战性的任务。尽管先前的研究在同步和视觉质量方面取得了进展，但唇形清晰度和视频流畅性仍然是持续存在的难题。本文提出了一种名为FluentLip的两阶段方法用于音频驱动的唇形合成，并整合了三个特色策略。为了提高唇部动作的同步性和可理解性，我们集成了一个音素提取器和编码器来生成音频与音素信息的融合，以进行多模态学习。此外，我们采用了光学流一致性损失，确保图像帧之间的自然过渡。另外，在训练生成对抗网络（GAN）时，我们引入了扩散链，旨在提高稳定性和效率。 我们通过广泛的实验评估了所提出的FluentLip方法，并将其与五个最先进的方法在包括音素错误率（PER）在内的五项指标上进行了比较，该指标用于评估唇部姿态的可理解性及视频流畅度。实验结果表明，我们的FluentLip方法具有很强的竞争性，在平滑性和自然度方面取得了显著改进。特别地，在Fréchet inception距离（FID）和音素错误率（PER）这两项关键指标上，与最先进的方法相比，分别提升了约16.3% 和 35.2%。
URL
https://arxiv.org/abs/2504.04427
