Abstract
Lip synchronization is the task of aligning a speaker's lip movements in video with corresponding speech audio, and it is essential for creating realistic, expressive video content. However, existing methods often rely on reference frames and masked-frame inpainting, which limit their robustness to identity consistency, pose variations, facial occlusions, and stylized content. In addition, since audio signals provide weaker conditioning than visual cues, lip shape leakage from the original video will affect lip sync quality. In this paper, we present OmniSync, a universal lip synchronization framework for diverse visual scenarios. Our approach introduces a mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks, enabling unlimited-duration inference while maintaining natural facial dynamics and preserving character identity. During inference, we propose a flow-matching-based progressive noise initialization to ensure pose and identity consistency, while allowing precise mouth-region editing. To address the weak conditioning signal of audio, we develop a Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism that adaptively adjusts guidance strength over time and space. We also establish the AIGC-LipSync Benchmark, the first evaluation suite for lip synchronization in diverse AI-generated videos. Extensive experiments demonstrate that OmniSync significantly outperforms prior methods in both visual quality and lip sync accuracy, achieving superior results in both real-world and AI-generated videos.
Abstract (translated)
嘴唇同步是指将视频中说话者的唇部动作与相应的语音音频对齐的任务，对于创建真实且具有表现力的视频内容至关重要。然而，现有的方法往往依赖于参考帧和掩码帧修复技术（如填补缺失或损坏的信息），这限制了它们在身份一致性、姿态变化、面部遮挡以及风格化内容方面的鲁棒性。此外，由于音频信号提供的引导信息弱于视觉线索，来自原始视频的唇形泄漏将影响嘴唇同步的质量。 本文介绍了OmniSync——一个用于多种视觉场景的通用嘴唇同步框架。我们的方法引入了一种无需掩码的训练范式，利用扩散变换模型直接进行帧编辑，而不需要明确的掩码指示，从而支持不限时长的推理，并保持自然的面部动态并保留人物身份。在推理阶段，我们提出了一种基于流匹配的渐进噪声初始化方法，以确保姿态和身份的一致性，同时允许精确调整嘴部区域。 为了应对音频信号较弱引导信息的问题，我们开发了动态时空无分类器指引（Dynamic Spatiotemporal Classifier-Free Guidance, DS-CFG）机制，该机制能够随时间和空间自适应地调节指导强度。此外，我们建立了AIGC-LipSync基准测试集，这是第一个用于评估多样化AI生成视频中嘴唇同步效果的评测套件。 大量的实验表明，OmniSync在视觉质量和嘴唇同步准确度方面显著优于先前的方法，在现实世界和AI生成视频中都表现出色。
URL
https://arxiv.org/abs/2505.21448
