Abstract
In this work, we focus on a novel and practical task, i.e., Time-vAriant iMage inPainting (TAMP). The aim of TAMP is to restore a damaged target image by leveraging the complementary information from a reference image, where both images captured the same scene but with a significant time gap in between, i.e., time-variant images. Different from conventional reference-guided image inpainting, the reference image under TAMP setup presents significant content distinction to the target image and potentially also suffers from damages. Such an application frequently happens in our daily lives to restore a damaged image by referring to another reference image, where there is no guarantee of the reference image's source and quality. In particular, our study finds that even state-of-the-art (SOTA) reference-guided image inpainting methods fail to achieve plausible results due to the chaotic image complementation. To address such an ill-posed problem, we propose a novel Interactive Distribution Transition Estimation (InDiTE) module which interactively complements the time-variant images with adaptive semantics thus facilitate the restoration of damaged regions. To further boost the performance, we propose our TAMP solution, namely Interactive Distribution Transition Estimation-driven Diffusion (InDiTE-Diff), which integrates InDiTE with SOTA diffusion model and conducts latent cross-reference during sampling. Moreover, considering the lack of benchmarks for TAMP task, we newly assembled a dataset, i.e., TAMP-Street, based on existing image and mask datasets. We conduct experiments on the TAMP-Street datasets under two different time-variant image inpainting settings, which show our method consistently outperform SOTA reference-guided image inpainting methods for solving TAMP.
Abstract (translated)
在这项工作中，我们专注于一个新颖且实用的任务，即时变图像修补（Time-vAriant iMage inPainting，TAMP）。TAMP的目标是利用参考图像中的互补信息来修复受损的目标图像，其中两幅图像是在不同时间点拍摄的同一场景，具有显著的时间差。与传统的基于参考引导的图像修补任务不同，在TAMP设定中，参考图像与目标图像之间存在显著的内容差异，并且可能也遭受了损坏。这种应用经常发生在我们的日常生活中，通过引用另一张参考图像来修复受损图像时会出现这种情况，而此时无法保证参考图的质量和来源。 特别地，我们发现即使是当前最先进的（SOTA）基于参考引导的图像修补方法，在面对混沌的图像补全时也无法达到令人信服的结果。为了解决这一病态问题，我们提出了一种新的交互式分布转换估计(InDiTE)模块，该模块能够适应性地补充具有时间变化性的图像，并利用其语义信息来促进受损区域的修复。为了进一步提高性能，我们提出了我们的TAMP解决方案——基于InDiTE驱动扩散模型的方法（InDiTE-Diff），它将InDiTE与最先进的扩散模型相结合，并在采样过程中进行潜在的跨参考操作。 此外，鉴于缺少针对TAMP任务的基准数据集，我们基于现有的图像和蒙版数据集新构建了一个名为TAMP-Street的数据集。我们在两个不同的时变图像修补设置下对TAMP-Street数据集进行了实验，结果表明我们的方法在解决TAMP问题方面始终优于当前最先进的基于参考引导的图像修补方法。
URL
https://arxiv.org/abs/2506.23461
