Abstract
High-quality AI-powered video dubbing demands precise audio-lip synchronization, high-fidelity visual generation, and faithful preservation of identity and background. Most existing methods rely on a mask-based training strategy, where the mouth region is masked in talking-head videos, and the model learns to synthesize lip movements from corrupted inputs and target audios. While this facilitates lip-sync accuracy, it disrupts spatiotemporal context, impairing performance on dynamic facial motions and causing instability in facial structure and background consistency. To overcome this limitation, we propose SyncAnyone, a novel two-stage learning framework that achieves accurate motion modeling and high visual fidelity simultaneously. In Stage 1, we train a diffusion-based video transformer for masked mouth inpainting, leveraging its strong spatiotemporal modeling to generate accurate, audio-driven lip movements. However, due to input corruption, minor artifacts may arise in the surrounding facial regions and the background. In Stage 2, we develop a mask-free tuning pipeline to address mask-induced artifacts. Specifically, on the basis of the Stage 1 model, we develop a data generation pipeline that creates pseudo-paired training samples by synthesizing lip-synced videos from the source video and random sampled audio. We further tune the stage 2 model on this synthetic data, achieving precise lip editing and better background consistency. Extensive experiments show that our method achieves state-of-the-art results in visual quality, temporal coherence, and identity preservation under in-the wild lip-syncing scenarios.
Abstract (translated)
高质量的人工智能驱动视频配音需要精准的音唇同步、高保真的视觉生成以及对身份和背景的真实保留。现有的大多数方法依赖于基于掩码的训练策略，即在说话头部视频中遮盖嘴巴区域，让模型从受损输入和目标音频中学习合成嘴唇动作。虽然这种方法有助于提高音唇同步精度，但它会破坏时空上下文，影响动态面部运动的表现，并导致面部结构和背景的一致性问题。 为了克服这一局限，我们提出了SyncAnyone，这是一种新颖的两阶段学习框架，它能够同时实现精确的动作建模和高视觉保真度。在第一阶段，我们训练了一个基于扩散模型的视频变压器来进行掩码口部修复，利用其强大的时空建模能力来生成准确且由音频驱动的唇部动作。然而，由于输入受到损坏，在周围面部区域和背景中可能会产生一些小瑕疵。 在第二阶段，我们开发了一条无掩码调整流水线来解决由掩码引起的瑕疵问题。具体而言，基于第一阶段模型，我们开发了一个数据生成流程，通过从原始视频和随机采样音频合成音唇同步的视频片段，创建伪配对训练样本。我们在这种合成数据上进一步微调第二阶段模型，实现了精确的嘴唇编辑，并提高了背景的一致性。 广泛的实验表明，在实际环境下的音唇同步场景中，我们的方法在视觉质量、时间一致性以及身份保留方面均取得了最先进的成果。
URL
https://arxiv.org/abs/2512.21736
