Abstract
Recent breakthroughs in video AIGC have ushered in a transformative era for audio-driven human animation. However, conventional video dubbing techniques remain constrained to mouth region editing, resulting in discordant facial expressions and body gestures that compromise viewer immersion. To overcome this limitation, we introduce sparse-frame video dubbing, a novel paradigm that strategically preserves reference keyframes to maintain identity, iconic gestures, and camera trajectories while enabling holistic, audio-synchronized full-body motion editing. Through critical analysis, we identify why naive image-to-video models fail in this task, particularly their inability to achieve adaptive conditioning. Addressing this, we propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing. This architecture leverages temporal context frames for seamless inter-chunk transitions and incorporates a simple yet effective sampling strategy that optimizes control strength via fine-grained reference frame positioning. Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance. Quantitative metrics confirm superior visual realism, emotional coherence, and full-body motion synchronization.
Abstract (translated)
近期在视频AIGC（人工智能生成内容）领域的突破，标志着音频驱动的人体动画进入了一个变革性的时代。然而，传统的视频配音技术仍然局限于对嘴部区域的编辑，这导致面部表情和身体动作与声音不协调，影响了观众的沉浸感。为了解决这一局限性，我们引入了一种新的范式——稀疏帧视频配音，该方法策略性地保留参考关键帧以维持身份、标志性手势以及摄像机轨迹，同时允许进行全面的身体运动编辑，使整个身体的动作与音频同步。 通过批判性的分析，我们确定了为什么简单的图像到视频模型在这个任务中失败，尤其是它们无法实现自适应条件设置。为了解决这个问题，我们提出了InfiniteTalk，这是一种专为无限长度长序列配音设计的流式音频驱动生成器架构。该架构利用时间上下文帧以实现无缝的跨片段过渡，并结合了一种简单而有效的采样策略，通过细粒度参考帧定位来优化控制强度。 在HDTF、CelebV-HQ和EMTD数据集上的全面评估表明，InfiniteTalk达到了最先进的性能水平。定量指标证实了其在视觉真实感、情感连贯性和全身运动同步方面的优越性。
URL
https://arxiv.org/abs/2508.14033
