Abstract
Video and audio inpainting for mixed audio-visual content has become a crucial task in multimedia editing recently. However, precisely removing an object and its corresponding audio from a video without affecting the rest of the scene remains a significant challenge. To address this, we propose VAInpaint, a novel pipeline that first utilizes a segmentation model to generate masks and guide a video inpainting model in removing objects. At the same time, an LLM then analyzes the scene globally, while a region-specific model provides localized descriptions. Both the overall and regional descriptions will be inputted into an LLM, which will refine the content and turn it into text queries for our text-driven audio separation model. Our audio separation model is fine-tuned on a customized dataset comprising segmented MUSIC instrument images and VGGSound backgrounds to enhance its generalization performance. Experiments show that our method achieves performance comparable to current benchmarks in both audio and video inpainting.
Abstract (translated)
最近，对于混合音频-视频内容的视频和音频修复（inpainting）已成为多媒体编辑中的一个重要任务。然而，精确地从视频中移除一个对象及其对应的音频而不影响其余场景仍然是一个重大挑战。为了解决这个问题，我们提出了VAInpaint，这是一个新颖的工作流程，首先利用分割模型生成掩码，并指导视频修复模型去除对象。同时，一个大规模语言模型（LLM）会进行全局场景分析，而特定区域的模型则提供局部描述。总体和区域描述将被输入到LLM中，该模型将进一步优化内容并将其转化为文本查询，用于驱动我们基于文本的音频分离模型。 我们的音频分离模型在一个定制的数据集上进行了微调，该数据集包含分割后的MUSIC乐器图像及VGGSound背景，以提高其泛化性能。实验结果显示，我们的方法在视频和音频修复方面都达到了与当前基准相媲美的性能。
URL
https://arxiv.org/abs/2509.17022
