Abstract
In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.
Abstract (translated)
在这项工作中，我们通过关注音频线索和面部动作之间动态而微妙的關係，来应对在谈话视频生成中提高真实感和表现力的挑战。我们指出了传统技术通常无法捕捉到人类表情的完整范围，以及个体面部风格的独特性。为解决这些问题，我们提出了EMO，一种新框架，利用直接音频到视频合成方法，无需中间3D模型或面部关键点。我们的方法确保了视频中的平滑过渡和身份保持一致，从而产生了高度表现力和逼真的动画。实验结果表明，EMO能够产生不仅是令人信服的讲话视频，还有各种风格的音乐视频，在表现力和真实感方面显著超过了现有最先进的方法。
URL
https://arxiv.org/abs/2402.17485
