Abstract
The task of video inpainting detection is to expose the pixel-level inpainted regions within a video sequence. Existing methods usually focus on leveraging spatial and temporal inconsistencies. However, these methods typically employ fixed operations to combine spatial and temporal clues, limiting their applicability in different scenarios. In this paper, we introduce a novel Multilateral Temporal-view Pyramid Transformer ({\em MumPy}) that collaborates spatial-temporal clues flexibly. Our method utilizes a newly designed multilateral temporal-view encoder to extract various collaborations of spatial-temporal clues and introduces a deformable window-based temporal-view interaction module to enhance the diversity of these collaborations. Subsequently, we develop a multi-pyramid decoder to aggregate the various types of features and generate detection maps. By adjusting the contribution strength of spatial and temporal clues, our method can effectively identify inpainted regions. We validate our method on existing datasets and also introduce a new challenging and large-scale Video Inpainting dataset based on the YouTube-VOS dataset, which employs several more recent inpainting methods. The results demonstrate the superiority of our method in both in-domain and cross-domain evaluation scenarios.
Abstract (translated)
视频修复检测的任务是揭示视频中每个视频帧的像素级修复区域。现有的方法通常利用空间和时间不一致性来结合空间和时间提示。然而，这些方法通常采用固定的操作来结合空间和时间提示，限制了它们在不同场景中的应用。在本文中，我们引入了一种名为Multilateral Temporal-view Pyramid Transformer（MUMPy）的新颖方法，它灵活地合作空间和时间提示。我们的方法利用一个新的多边形时间视图编码器来提取各种空间-时间提示的合作，并引入了一个可变的窗口基于时间视图的交互模块，以增强这些合作的变化。接下来，我们开发了一个多层金字塔解码器来聚合各种特征并生成检测图。通过调整空间和时间提示的贡献强度，我们的方法可以有效地检测修复区域。我们在现有数据集上评估了我们的方法，并还基于YouTube-VOS数据集引入了一个新的具有挑战性和大规模的视频修复检测数据集，该数据集采用了一些更先进的修复方法。结果表明，在我们的方法和跨域评估场景中，我们的方法具有优越性。
URL
https://arxiv.org/abs/2404.11054
