Abstract
Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data.
Abstract (translated)
对话面容生成因其广泛的应用而被广泛研究。用于对话面容生成的两个主要框架包括一个基于文本的框架,该框架从文本中生成同步的口语和对话面容,以及一个基于语音的框架,该框架从语音中生成对话面容。为了整合这些框架,本文提出了一个统一的面部地标生成器(UniFLG)。该提出的系统利用端到端文本到语音技术,不仅合成语音,还提取文本和语音中共同的隐态表示,并将其喂给地标解码器生成面部地标。我们证明,我们系统的语音合成和面部地标生成相比现有基于文本的方法更加自然。我们还证明,我们系统可以从没有面部视频数据或甚至没有语音数据的发言中生成面部地标。
URL
https://arxiv.org/abs/2302.14337
