Abstract
Image inpainting aims to fill the missing region of an image. Recently, there has been a surge of interest in foreground-conditioned background inpainting, a sub-task that fills the background of an image while the foreground subject and associated text prompt are provided. Existing background inpainting methods typically strictly preserve the subject's original position from the source image, resulting in inconsistencies between the subject and the generated background. To address this challenge, we propose a new task, the "Text-Guided Subject-Position Variable Background Inpainting", which aims to dynamically adjust the subject position to achieve a harmonious relationship between the subject and the inpainted background, and propose the Adaptive Transformation Agent (A$^\text{T}$A) for this task. Firstly, we design a PosAgent Block that adaptively predicts an appropriate displacement based on given features to achieve variable subject-position. Secondly, we design the Reverse Displacement Transform (RDT) module, which arranges multiple PosAgent blocks in a reverse structure, to transform hierarchical feature maps from deep to shallow based on semantic information. Thirdly, we equip A$^\text{T}$A with a Position Switch Embedding to control whether the subject's position in the generated image is adaptively predicted or fixed. Extensive comparative experiments validate the effectiveness of our A$^\text{T}$A approach, which not only demonstrates superior inpainting capabilities in subject-position variable inpainting, but also ensures good performance on subject-position fixed inpainting.
Abstract (translated)
图像修复技术旨在填补图像中的缺失区域。近期，人们对前景条件下的背景修复（即在给定前景对象和相关文本提示的情况下填充图像的背景）产生了浓厚的兴趣。现有的背景修复方法通常严格保持主体在源图中的原始位置不变，这会导致修复后的背景与主体之间产生不协调的问题。为解决这一挑战，我们提出了一项新任务：“基于文本指导的位置可变背景修复”，旨在动态调整对象位置，以实现生成的背景与对象之间的和谐关系，并为此任务提出了自适应变换代理（Adaptive Transformation Agent, A$^\text{T}$A）。具体来说： 1. 我们设计了一个PosAgent模块，该模块根据给定的功能特性自适应地预测一个合适的位移量来实现位置可变性。 2. 我们还设计了逆向位移变换（Reverse Displacement Transform, RDT）模块。这个模块采用了多个PosAgent块的反向结构，能够基于语义信息将深层次特征图转化为浅层特征图。 3. 为了控制生成图像中对象的位置是自适应预测还是固定不变，我们在A$^\text{T}$A模型中添加了位置切换嵌入(Position Switch Embedding)。 通过广泛的对比实验验证了我们提出的A$^\text{T}$A方法的有效性。我们的方法不仅在处理位置可变修复时表现出卓越的填充能力，而且还在确保对象位置固定不变的情况下保持良好的性能。
URL
https://arxiv.org/abs/2504.01603
