Abstract
In this paper, we introduce Object-WIPER, a training-free framework for removing dynamic objects and their associated visual effects from videos, and inpainting them with semantically consistent and temporally coherent content. Our approach leverages a pre-trained text-to-video diffusion transformer (DiT). Given an input video, a user-provided object mask, and query tokens describing the target object and its effects, we localize relevant visual tokens via visual-text cross-attention and visual self-attention. This produces an intermediate effect mask that we fuse with the user mask to obtain a final foreground token mask to replace. We first invert the video through the DiT to obtain structured noise, then reinitialize the masked tokens with Gaussian noise while preserving background tokens. During denoising, we copy values for the background tokens saved during inversion to maintain scene fidelity. To address the lack of suitable evaluation, we introduce a new object removal metric that rewards temporal consistency among foreground tokens across consecutive frames, coherence between foreground and background tokens within each frame, and dissimilarity between the input and output foreground tokens. Experiments on DAVIS and a newly curated real-world associated effect benchmark (WIPER-Bench) show that Object-WIPER surpasses both training-based and training-free baselines in terms of the metric, achieving clean removal and temporally stable reconstruction without any retraining. Our new benchmark, source code, and pre-trained models will be publicly available.
Abstract (translated)
在这篇论文中，我们介绍了Object-WIPER，这是一种无需训练的框架，用于从视频中移除动态对象及其相关视觉效果，并用语义一致且时间连贯的内容进行修补。我们的方法利用了一个预训练的文本到视频扩散变换器（DiT）。给定输入视频、用户提供的物体掩码以及描述目标物体及其影响的查询令牌后，我们通过视觉-文本交叉注意力和视觉自我注意来定位相关的视觉令牌。这会产生一个中间效果掩码，我们将此与用户的掩码融合以获得最终的前景令牌掩码进行替换。首先，我们通过DiT对视频进行逆向处理，得到结构化的噪声，然后在保留背景令牌的同时用高斯噪声重新初始化被屏蔽的令牌。在降噪过程中，我们会复制保存的背景令牌值以保持场景的真实性。为了应对合适的评估方法不足的问题，我们引入了一种新的对象移除度量标准，该标准奖励连续帧之间前景令牌的时间一致性、每个帧内前景和背景令牌之间的连贯性以及输入与输出前景令牌间的差异。 在DAVIS数据集和一个新的现实世界关联效果基准（WIPER-Bench）上的实验表明，Object-WIPER在所提出的度量标准上优于训练基线和无训练基线，在不进行重新训练的情况下实现了干净的移除和时间稳定重建。我们的新基准、源代码以及预训练模型将公开提供。
URL
https://arxiv.org/abs/2601.06391
