Abstract
Visual dubbing, the synchronization of facial movements with new speech, is crucial for making content accessible across different languages, enabling broader global reach. However, current methods face significant limitations. Existing approaches often generate talking faces, hindering seamless integration into original scenes, or employ inpainting techniques that discard vital visual information like partial occlusions and lighting variations. This work introduces EdiDub, a novel framework that reformulates visual dubbing as a content-aware editing task. EdiDub preserves the original video context by utilizing a specialized conditioning scheme to ensure faithful and accurate modifications rather than mere copying. On multiple benchmarks, including a challenging occluded-lip dataset, EdiDub significantly improves identity preservation and synchronization. Human evaluations further confirm its superiority, achieving higher synchronization and visual naturalness scores compared to the leading methods. These results demonstrate that our content-aware editing approach outperforms traditional generation or inpainting, particularly in maintaining complex visual elements while ensuring accurate lip synchronization.
Abstract (translated)
视觉配音，即同步新的语音与面部动作，对于跨语言传播内容至关重要，使其能够覆盖更广泛的全球受众。然而，目前的方法存在显著局限性。现有方法通常生成讲话人脸，难以无缝集成到原始场景中，或使用填补技术（inpainting techniques）丢弃部分遮挡和光照变化等重要视觉信息。这项工作介绍了一种名为EdiDub的新框架，它将视觉配音重新定义为一种内容感知编辑任务。通过采用专门的条件设置方案，EdiDub能够在保留原始视频背景的情况下进行忠实而准确的修改，而非简单的复制操作。 在多个基准测试中，包括一个具有挑战性的唇部被遮挡的数据集上，EdiDub显著提升了身份保存和同步性能。人类评估进一步确认了其优越性，在同步性和视觉自然度评分方面超越了现有领先方法。这些结果表明，我们的内容感知编辑方法优于传统的生成或填补技术，特别是在保持复杂视觉元素的同时确保精确的唇部同步。
URL
https://arxiv.org/abs/2505.23406
