Abstract
Real-time interactive video-chat portraits have been increasingly recognized as the future trend, particularly due to the remarkable progress made in text and voice chat technologies. However, existing methods primarily focus on real-time generation of head movements, but struggle to produce synchronized body motions that match these head actions. Additionally, achieving fine-grained control over the speaking style and nuances of facial expressions remains a challenge. To address these limitations, we introduce a novel framework for stylized real-time portrait video generation, enabling expressive and flexible video chat that extends from talking head to upper-body interaction. Our approach consists of the following two stages. The first stage involves efficient hierarchical motion diffusion models, that take both explicit and implicit motion representations into account based on audio inputs, which can generate a diverse range of facial expressions with stylistic control and synchronization between head and body movements. The second stage aims to generate portrait video featuring upper-body movements, including hand gestures. We inject explicit hand control signals into the generator to produce more detailed hand movements, and further perform face refinement to enhance the overall realism and expressiveness of the portrait video. Additionally, our approach supports efficient and continuous generation of upper-body portrait video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting interactive video-chat in real-time. Experimental results demonstrate the capability of our approach to produce portrait videos with rich expressiveness and natural upper-body movements.
Abstract (translated)
实时互动视频聊天肖像已经被视为未来的发展趋势，这主要是由于文字和语音聊天技术取得了显著的进步。然而，现有的方法主要集中在头部动作的实时生成上，但难以产生与这些头部动作相匹配的身体运动同步。此外，在说话风格和面部表情细微之处实现精细控制仍然是一项挑战。为了克服这些限制，我们提出了一种新的框架，用于生成具有风格化的实时肖像视频，该框架扩展了从脸部互动到上半身互动的表达性和灵活性。 我们的方法包括以下两个阶段： **第一阶段：** 这个阶段涉及高效的分层运动扩散模型，考虑到基于音频输入的显式和隐式的运动表示。这些模型可以产生一系列具有风格化控制并且头部与身体动作同步的面部表情变化。 **第二阶段：** 该阶段的目标是生成包含上半身动作（包括手势）的肖像视频。我们通过向生成器注入明确的手部控制信号来产生更详细的手部动作，并进一步进行脸部优化以增强整个视频的真实感和表现力。此外，我们的方法支持在最大512 * 768分辨率下以每秒30帧的速度实时生成上半身肖像视频，在4090 GPU上的性能表现出色。 实验结果表明，该方法能够生成具有丰富表情和自然上半身动作的肖像视频。
URL
https://arxiv.org/abs/2503.21144
