Abstract
Audio-driven visual dubbing aims to synchronize a video's lip movements with new speech, but is fundamentally challenged by the lack of ideal training data: paired videos where only a subject's lip movements differ while all other visual conditions are identical. Existing methods circumvent this with a mask-based inpainting paradigm, where an incomplete visual conditioning forces models to simultaneously hallucinate missing content and sync lips, leading to visual artifacts, identity drift, and poor synchronization. In this work, we propose a novel self-bootstrapping framework that reframes visual dubbing from an ill-posed inpainting task into a well-conditioned video-to-video editing problem. Our approach employs a Diffusion Transformer, first as a data generator, to synthesize ideal training data: a lip-altered companion video for each real sample, forming visually aligned video pairs. A DiT-based audio-driven editor is then trained on these pairs end-to-end, leveraging the complete and aligned input video frames to focus solely on precise, audio-driven lip modifications. This complete, frame-aligned input conditioning forms a rich visual context for the editor, providing it with complete identity cues, scene interactions, and continuous spatiotemporal dynamics. Leveraging this rich context fundamentally enables our method to achieve highly accurate lip sync, faithful identity preservation, and exceptional robustness against challenging in-the-wild scenarios. We further introduce a timestep-adaptive multi-phase learning strategy as a necessary component to disentangle conflicting editing objectives across diffusion timesteps, thereby facilitating stable training and yielding enhanced lip synchronization and visual fidelity. Additionally, we propose ContextDubBench, a comprehensive benchmark dataset for robust evaluation in diverse and challenging practical application scenarios.
Abstract (translated)
音频驱动的视觉配音旨在将视频中的嘴唇动作与新的语音同步，但这一目标在很大程度上受到理想训练数据缺乏的限制：即只有主体的唇部运动不同而所有其他视觉条件都相同的配对视频。现有方法通过一种基于掩码的修补（inpainting）范式来规避这个问题，在这种范式中，不完整的视觉条件迫使模型同时填补缺失内容并同步嘴唇动作，从而导致视觉伪影、身份漂移和较差的时间同步。 在本工作中，我们提出了一种新的自引导框架，将视觉配音从一个病态的修补任务转变为一个良好的视频到视频编辑问题。我们的方法采用扩散变压器（Diffusion Transformer），最初作为数据生成器，来合成理想的训练数据：为每个真实样本创建一个唇部调整后的伴生视频，形成可视对齐的视频配对。然后，基于DiT（Diffusion Image-to-Image）的音频驱动编辑器在这些配对上进行端到端训练，利用完整的、帧对齐的输入视频帧来专注于精确的音频驱动嘴唇修改。这种完整且帧对齐的输入条件为编辑器提供了丰富的视觉上下文，包括完整的身份线索、场景互动和持续的空间时间动态。 通过这一丰富背景的基本要素使我们的方法能够实现高度准确的唇部同步、忠实的身份保持以及在具有挑战性的现实场景中表现出色的能力。此外，我们还引入了一种时序自适应多阶段学习策略作为必要组件，以分离扩散步骤中的冲突编辑目标，从而促进稳定的训练并提高唇部同步和视觉保真度。最后，为了对稳健性进行全面评估，在各种实际应用的挑战性场景下，我们提出了ContextDubBench这一全面基准数据集。
URL
https://arxiv.org/abs/2512.25066
