Abstract
The visual dubbing task aims to generate mouth movements synchronized with the driving audio, which has seen significant progress in recent years. However, two critical deficiencies hinder their wide application: (1) Audio-only driving paradigms inadequately capture speaker-specific lip habits, which fail to generate lip movements similar to the target avatar; (2) Conventional blind-inpainting approaches frequently produce visual artifacts when handling obstructions (e.g., microphones, hands), limiting practical deployment. In this paper, we propose StableDub, a novel and concise framework integrating lip-habit-aware modeling with occlusion-robust synthesis. Specifically, building upon the Stable-Diffusion backbone, we develop a lip-habit-modulated mechanism that jointly models phonemic audio-visual synchronization and speaker-specific orofacial dynamics. To achieve plausible lip geometries and object appearances under occlusion, we introduce the occlusion-aware training strategy by explicitly exposing the occlusion objects to the inpainting process. By incorporating the proposed designs, the model eliminates the necessity for cost-intensive priors in previous methods, thereby exhibiting superior training efficiency on the computationally intensive diffusion-based backbone. To further optimize training efficiency from the perspective of model architecture, we introduce a hybrid Mamba-Transformer architecture, which demonstrates the enhanced applicability in low-resource research scenarios. Extensive experimental results demonstrate that StableDub achieves superior performance in lip habit resemblance and occlusion robustness. Our method also surpasses other methods in audio-lip sync, video quality, and resolution consistency. We expand the applicability of visual dubbing methods from comprehensive aspects, and demo videos can be found at this https URL.
Abstract (translated)
视觉配音任务的目标是生成与驱动音频同步的口型动画，近年来取得了显著进展。然而，两个关键缺陷阻碍了其广泛应用：(1) 仅基于音频驱动的方法不足以捕捉特定说话人的唇部习惯，因而无法产生类似于目标角色的口型动作；(2) 常规的盲填充方法在处理遮挡物（如麦克风、手）时常常会产生视觉伪影，限制了实际应用。为此，在本文中我们提出了StableDub，这是一种新颖且简洁的框架，集成了唇部习惯感知建模与遮挡鲁棒合成技术。具体而言，在Stable-Diffusion骨干网络的基础上，我们开发了一种受唇部习惯调制的机制，该机制联合模型音素级音频-视觉同步和特定说话人的口面部动态。为了在存在遮挡的情况下实现逼真的唇形几何结构和对象外观，我们引入了感知遮挡训练策略，并通过显式地将遮挡物暴露给填充过程来提高鲁棒性。通过采用所提出的这些设计，模型消除了对以往方法中成本高昂的先验知识的需求，在计算密集型扩散基础架构上表现出色的培训效率。为从模型架构角度进一步优化培训效率，我们引入了混合Mamba-Transformer架构，这在资源有限的研究场景中显示出增强的应用性。 大量的实验结果表明，StableDub 在唇部习惯相似度和遮挡鲁棒性方面表现卓越，并且在音频与嘴唇同步、视频质量和分辨率一致性等方面也超越了其他方法。我们的方法从多个方面扩展了视觉配音技术的适用性，演示视频可在以下网址找到：[https URL]。
URL
https://arxiv.org/abs/2509.21887
