Inpainting
-
FixingGS: Enhancing 3D Gaussian Splatting via Training-Free Score Distillation
2025-09-23 07:53:46Zhaorui Wang, Yi Gu, Deming Zhou, Renjing XuAbstract
Recently, 3D Gaussian Splatting (3DGS) has demonstrated remarkable success in 3D reconstruction and novel view synthesis. However, reconstructing 3D scenes from sparse viewpoints remains highly challenging due to insufficient visual information, which results in noticeable artifacts persisting across the 3D representation. To address this limitation, recent methods have resorted to generative priors to remove artifacts and complete missing content in under-constrained areas. Despite their effectiveness, these approaches struggle to ensure multi-view consistency, resulting in blurred structures and implausible details. In this work, we propose FixingGS, a training-free method that fully exploits the capabilities of the existing diffusion model for sparse-view 3DGS reconstruction enhancement. At the core of FixingGS is our distillation approach, which delivers more accurate and cross-view coherent diffusion priors, thereby enabling effective artifact removal and inpainting. In addition, we propose an adaptive progressive enhancement scheme that further refines reconstructions in under-constrained regions. Extensive experiments demonstrate that FixingGS surpasses existing state-of-the-art methods with superior visual quality and reconstruction performance. Our code will be released publicly.
Abstract (translated)
最近，3D高斯点阵（3D Gaussian Splatting, 3DGS）在三维重建和新视角合成方面取得了显著的成功。然而，由于视觉信息不足，从稀疏视点重建三维场景仍然极具挑战性，这导致明显的伪影持续存在于三维表示中。为了解决这一限制，最近的方法转向生成先验来移除这些伪影并在约束条件不足的区域补充缺失内容。尽管这些方法效果显著，但它们难以确保多视角一致性，从而导致结构模糊和细节不合理的问题。 在本工作中，我们提出了FixingGS，这是一种无需训练的方法，充分利用现有的扩散模型来进行稀疏视点3DGS重建增强。FixingGS的核心是我们的知识蒸馏方法，这种方法提供了更准确且跨视角一致的扩散先验，从而使有效的伪影移除和修复成为可能。此外，我们还提出了一种自适应渐进式增强方案，进一步细化在约束条件不足区域中的重建结果。 大量的实验表明，FixingGS在视觉质量和重建性能方面都超越了现有的最先进方法。我们的代码将公开发布。
URL
https://arxiv.org/abs/2509.18759
PDF
-
Prompt-Guided Dual Latent Steering for Inversion Problems
2025-09-23 04:11:06Yichen Wu, Xu Liu, Chenxuan Zhao, Xinyu WuAbstract
Inverting corrupted images into the latent space of diffusion models is challenging. Current methods, which encode an image into a single latent vector, struggle to balance structural fidelity with semantic accuracy, leading to reconstructions with semantic drift, such as blurred details or incorrect attributes. To overcome this, we introduce Prompt-Guided Dual Latent Steering (PDLS), a novel, training-free framework built upon Rectified Flow models for their stable inversion paths. PDLS decomposes the inversion process into two complementary streams: a structural path to preserve source integrity and a semantic path guided by a prompt. We formulate this dual guidance as an optimal control problem and derive a closed-form solution via a Linear Quadratic Regulator (LQR). This controller dynamically steers the generative trajectory at each step, preventing semantic drift while ensuring the preservation of fine detail without costly, per-image optimization. Extensive experiments on FFHQ-1K and ImageNet-1K under various inversion tasks, including Gaussian deblurring, motion deblurring, super-resolution and freeform inpainting, demonstrate that PDLS produces reconstructions that are both more faithful to the original image and better aligned with the semantic information than single-latent baselines.
Abstract (translated)
将受损图像转化为扩散模型的潜在空间是一项挑战。当前的方法，通过将图像编码为单一潜变量向量，难以在结构保真度和语义准确性之间取得平衡，导致重构时出现语义漂移问题，如细节模糊或属性不正确。为了克服这一难题，我们引入了基于校正流模型的新型无训练框架——提示引导双潜在控制（PDLS）。该方法利用这些模型稳定的反演路径，并将反演过程分解为两条互补路径：一条用于保持源图像完整性的结构路径和一条由提示指导的语义路径。我们将这种双重指引形式化为最优控制问题，并通过线性二次调节器（LQR）得出闭式解来实现控制器，该控制器在每一步动态调整生成轨迹，防止语义漂移的同时确保细节点的保真度，而无需进行昂贵且针对每个图像的优化。 广泛的实验在FFHQ-1K和ImageNet-1K数据集上进行了各种反演任务（包括高斯去模糊、运动去模糊、超分辨率和自由形式修复），结果表明PDLS生成的重构不仅更忠于原始图像，而且与单一潜变量基线相比，在语义信息方面也更为一致。
URL
https://arxiv.org/abs/2509.18619
PDF
-
Multi-Agent Amodal Completion: Direct Synthesis with Fine-Grained Semantic Guidance
2025-09-22 13:20:06Hongxing Fan, Lipeng Wang, Haohua Chen, Zehuan Huang, Jiangtao Wu, Lu ShengAbstract
Amodal completion, generating invisible parts of occluded objects, is vital for applications like image editing and AR. Prior methods face challenges with data needs, generalization, or error accumulation in progressive pipelines. We propose a Collaborative Multi-Agent Reasoning Framework based on upfront collaborative reasoning to overcome these issues. Our framework uses multiple agents to collaboratively analyze occlusion relationships and determine necessary boundary expansion, yielding a precise mask for inpainting. Concurrently, an agent generates fine-grained textual descriptions, enabling Fine-Grained Semantic Guidance. This ensures accurate object synthesis and prevents the regeneration of occluders or other unwanted elements, especially within large inpainting areas. Furthermore, our method directly produces layered RGBA outputs guided by visible masks and attention maps from a Diffusion Transformer, eliminating extra segmentation. Extensive evaluations demonstrate our framework achieves state-of-the-art visual quality.
Abstract (translated)
非模态完成（生成被遮挡物体的不可见部分）对于图像编辑和AR等应用至关重要。然而，之前的方法在数据需求、泛化能力或逐步处理流水线中的错误累积方面面临挑战。我们提出了一种基于前期协作推理的多代理协同推理框架，以克服这些难题。我们的框架利用多个代理来共同分析遮挡关系并确定必要的边界扩展，从而生成用于图像修复的精确掩码。同时，一个代理会生成细粒度的文字描述，这提供了精细语义指导，确保准确的对象合成，并防止重新生成遮挡物或其他不必要的元素，特别是在大范围的修复区域中更为关键。 此外，我们的方法直接根据可见掩码和来自扩散变换器的注意力图生成分层RGBA输出，从而省去了额外的分割步骤。广泛的评估表明，我们的框架在视觉质量上达到了最先进的水平。
URL
https://arxiv.org/abs/2509.17757
PDF
-
Clothing agnostic Pre-inpainting Virtual Try-ON
2025-09-22 11:58:20Sehyun Kim, Hye Jun Lee, Jiwoo Lee, Taemin LeeAbstract
With the development of deep learning technology, virtual try-on technology has become an important application value in the fields of e-commerce, fashion, and entertainment. The recently proposed Leffa has improved the texture distortion problem of diffu-sion-based models, but there are limitations in that the bottom detection inaccuracy and the existing clothing silhouette remain in the synthesis results. To solve this problem, this study proposes CaP-VTON (Clothing agnostic Pre-inpainting Virtual Try-ON). CaP-VTON has improved the naturalness and consistency of whole-body clothing syn-thesis by integrating multi-category masking based on Dress Code and skin inpainting based on Stable Diffusion. In particular, a generate skin module was introduced to solve the skin restoration problem that occurs when long-sleeved images are converted into short-sleeved or sleeveless ones, and high-quality restoration was implemented consider-ing the human body posture and color. As a result, CaP-VTON recorded 92.5\%, which is 15.4\% better than Leffa in short-sleeved synthesis accuracy, and showed the performance of consistently reproducing the style and shape of reference clothing in visual evaluation. These structures maintain model-agnostic properties and are applicable to various diffu-sion-based virtual inspection systems, and can contribute to applications that require high-precision virtual wearing, such as e-commerce, custom styling, and avatar creation.
Abstract (translated)
随着深度学习技术的发展，虚拟试穿技术在电子商务、时尚和娱乐领域变得越来越重要。最近提出的Leffa模型虽然改善了基于扩散的模型中的纹理失真问题，但仍存在底边检测不准确以及现有服装轮廓残留的问题。为解决这些问题，本研究提出了一种名为CaP-VTON（Clothing agnostic Pre-inpainting Virtual Try-ON）的新方法。CaP-VTON通过整合基于Dress Code的多类别遮罩和基于Stable Diffusion的人体皮肤修复技术，提高了全身服装合成的自然度与一致性。 特别是，该方法引入了一种生成皮肤模块来解决将长袖图像转换为短袖或无袖时出现的皮肤恢复问题，并考虑到人体姿态和颜色实现了高质量的恢复。实验结果显示，CaP-VTON在短袖合成准确率上达到了92.5%，比Leffa高出15.4%；视觉评估显示其能够一致地再现参考服装的风格和形状。这些结构保持了模型无关性属性，适用于各种基于扩散的虚拟试穿系统，并能为需要高精度虚拟穿衣的应用（如电子商务、定制造型以及头像创建）做出贡献。
URL
https://arxiv.org/abs/2509.17654
PDF
-
VAInpaint: Zero-Shot Video-Audio inpainting framework with LLMs-driven Module
2025-09-21 10:31:56Kam Man Wu, Zeyue Tian, Liya Ji, Qifeng ChenAbstract
Video and audio inpainting for mixed audio-visual content has become a crucial task in multimedia editing recently. However, precisely removing an object and its corresponding audio from a video without affecting the rest of the scene remains a significant challenge. To address this, we propose VAInpaint, a novel pipeline that first utilizes a segmentation model to generate masks and guide a video inpainting model in removing objects. At the same time, an LLM then analyzes the scene globally, while a region-specific model provides localized descriptions. Both the overall and regional descriptions will be inputted into an LLM, which will refine the content and turn it into text queries for our text-driven audio separation model. Our audio separation model is fine-tuned on a customized dataset comprising segmented MUSIC instrument images and VGGSound backgrounds to enhance its generalization performance. Experiments show that our method achieves performance comparable to current benchmarks in both audio and video inpainting.
Abstract (translated)
最近，对于混合音频-视频内容的视频和音频修复（inpainting）已成为多媒体编辑中的一个重要任务。然而，精确地从视频中移除一个对象及其对应的音频而不影响其余场景仍然是一个重大挑战。为了解决这个问题，我们提出了VAInpaint，这是一个新颖的工作流程，首先利用分割模型生成掩码，并指导视频修复模型去除对象。同时，一个大规模语言模型（LLM）会进行全局场景分析，而特定区域的模型则提供局部描述。总体和区域描述将被输入到LLM中，该模型将进一步优化内容并将其转化为文本查询，用于驱动我们基于文本的音频分离模型。 我们的音频分离模型在一个定制的数据集上进行了微调，该数据集包含分割后的MUSIC乐器图像及VGGSound背景，以提高其泛化性能。实验结果显示，我们的方法在视频和音频修复方面都达到了与当前基准相媲美的性能。
URL
https://arxiv.org/abs/2509.17022
PDF
-
Octree Latent Diffusion for Semantic 3D Scene Generation and Completion
2025-09-20 00:53:13Xujia Zhang, Brendan Crowe, Christoffer HeckmanAbstract
The completion, extension, and generation of 3D semantic scenes are an interrelated set of capabilities that are useful for robotic navigation and exploration. Existing approaches seek to decouple these problems and solve them oneoff. Additionally, these approaches are often domain-specific, requiring separate models for different data distributions, e.g. indoor vs. outdoor scenes. To unify these techniques and provide cross-domain compatibility, we develop a single framework that can perform scene completion, extension, and generation in both indoor and outdoor scenes, which we term Octree Latent Semantic Diffusion. Our approach operates directly on an efficient dual octree graph latent representation: a hierarchical, sparse, and memory-efficient occupancy structure. This technique disentangles synthesis into two stages: (i) structure diffusion, which predicts binary split signals to construct a coarse occupancy octree, and (ii) latent semantic diffusion, which generates semantic embeddings decoded by a graph VAE into voxellevel semantic labels. To perform semantic scene completion or extension, our model leverages inference-time latent inpainting, or outpainting respectively. These inference-time methods use partial LiDAR scans or maps to condition generation, without the need for retraining or finetuning. We demonstrate highquality structure, coherent semantics, and robust completion from single LiDAR scans, as well as zero-shot generalization to out-of-distribution LiDAR data. These results indicate that completion-through-generation in a dual octree graph latent space is a practical and scalable alternative to regression-based pipelines for real-world robotic perception tasks.
Abstract (translated)
三维语义场景的完成、扩展和生成是一系列相互关联的能力，这些能力对于机器人导航与探索非常有用。现有方法试图将这些问题解耦，并单独解决每一个问题。此外，这些方法往往是特定领域的，需要为不同的数据分布（如室内场景与室外场景）分别构建模型。为了统一这些技术并提供跨域兼容性，我们开发了一个单一的框架，能够在室内和室外场景中执行场景完成、扩展和生成任务，我们将该框架命名为Octree Latent Semantic Diffusion。 我们的方法直接在一种高效且稀疏的记忆节约型占用结构——双八叉树图隐式表示上操作。这种技术将合成过程分为两个阶段：(i) 结构扩散，预测二进制分割信号以构建一个粗略的占用八叉树；(ii) 隐含语义扩散，生成由图VAE解码成体素级别语义标签的语义嵌入。 为了执行语义场景完成或扩展任务，我们的模型利用了推断时的隐式插值（完成）和外推（扩展）。这些推断方法使用部分LiDAR扫描或地图作为条件进行生成，而无需重新训练或微调。我们展示了从单次LiDAR扫描中获得高质量结构、连贯语义以及稳健完成能力的能力，并且还能够零样本泛化到分布外的LiDAR数据。 这些结果表明，在双八叉树图隐式空间中的通过生成进行场景完成是一种实用而可扩展的方法，可以作为基于回归的工作流管道在现实世界的机器人感知任务中的一种替代方案。
URL
https://arxiv.org/abs/2509.16483
PDF
-
Similarity-Guided Diffusion for Long-Gap Music Inpainting
2025-09-19 18:33:51Sean Turland, Eloi Moliner, Vesa V\"alim\"akiAbstract
Music inpainting aims to reconstruct missing segments of a corrupted recording. While diffusion-based generative models improve reconstruction for medium-length gaps, they often struggle to preserve musical plausibility over multi-second gaps. We introduce Similarity-Guided Diffusion Posterior Sampling (SimDPS), a hybrid method that combines diffusion-based inference with similarity search. Candidate segments are first retrieved from a corpus based on contextual similarity, then incorporated into a modified likelihood that guides the diffusion process toward contextually consistent reconstructions. Subjective evaluation on piano music inpainting with 2-s gaps shows that the proposed SimDPS method enhances perceptual plausibility compared to unguided diffusion and frequently outperforms similarity search alone when moderately similar candidates are available. These results demonstrate the potential of a hybrid similarity approach for diffusion-based audio enhancement with long gaps.
Abstract (translated)
音乐补全（Music inpainting）的目标是重建受损录音中缺失的部分。虽然基于扩散的生成模型可以改善中等长度缺口的重构效果，但在处理多秒级缺口时往往难以保持音乐的真实性。我们引入了一种名为“相似性引导的扩散后验采样”（Similarity-Guided Diffusion Posterior Sampling, SimDPS）的混合方法，该方法结合了基于扩散的推理和相似度搜索技术。首先从语料库中根据上下文相似度检索候选片段，然后将这些候选片段整合到一个修改过的似然函数中，以指导扩散过程朝向符合上下文一致性的重建方向。 在钢琴音乐补全实验中，使用2秒长缺口进行主观评估表明：与未引导的扩散方法相比，所提出的SimDPS方法提高了感知真实性；同时，在有适度相似候选片段可用的情况下，该方法通常优于单独使用的相似性搜索。这些结果证明了混合相似度方法在处理长时间间隙时对基于扩散音频增强技术的巨大潜力。
URL
https://arxiv.org/abs/2509.16342
PDF
-
Controllable Localized Face Anonymization Via Diffusion Inpainting
2025-09-18 11:33:47Ali Salar, Qing Liu, Guoying ZhaoAbstract
The growing use of portrait images in computer vision highlights the need to protect personal identities. At the same time, anonymized images must remain useful for downstream computer vision tasks. In this work, we propose a unified framework that leverages the inpainting ability of latent diffusion models to generate realistic anonymized images. Unlike prior approaches, we have complete control over the anonymization process by designing an adaptive attribute-guidance module that applies gradient correction during the reverse denoising process, aligning the facial attributes of the generated image with those of the synthesized target image. Our framework also supports localized anonymization, allowing users to specify which facial regions are left unchanged. Extensive experiments conducted on the public CelebA-HQ and FFHQ datasets show that our method outperforms state-of-the-art approaches while requiring no additional model training. The source code is available on our page.
Abstract (translated)
随着计算机视觉中人像图片使用量的增加，保护个人身份的需求日益凸显。同时，匿名化后的图像必须仍能用于下游的计算机视觉任务。为此，我们提出了一种统一框架，利用潜在扩散模型（latent diffusion models）的修复能力来生成逼真的匿名化图像。与以往的方法不同，通过设计一种自适应属性指导模块，在逆向去噪过程中进行梯度校正，从而完全控制匿名化的全过程，并使生成的面部属性与合成的目标图像中的属性对齐。 此外，我们的框架支持局部匿名化功能，允许用户指定哪些面部区域保持不变。在公开的数据集CelebA-HQ和FFHQ上进行了广泛的实验，结果显示我们提出的方法优于当前最先进的方法，并且无需额外训练模型。源代码可在我们的页面上获取。
URL
https://arxiv.org/abs/2509.14866
PDF
-
End4: End-to-end Denoising Diffusion for Diffusion-Based Inpainting Detection
2025-09-16 16:19:53Fei Wang, Xuecheng Wu, Zheng Zhang, Danlei Huang, Yuheng Huang, BoWangAbstract
The powerful generative capabilities of diffusion models have significantly advanced the field of image synthesis, enhancing both full image generation and inpainting-based image editing. Despite their remarkable advancements, diffusion models also raise concerns about potential misuse for malicious purposes. However, existing approaches struggle to identify images generated by diffusion-based inpainting models, even when similar inpainted images are included in their training data. To address this challenge, we propose a novel detection method based on End-to-end denoising diffusion (End4). Specifically, End4 designs a denoising reconstruction model to improve the alignment degree between the latent spaces of the reconstruction and detection processes, thus reconstructing features that are more conducive to detection. Meanwhile, it leverages a Scale-aware Pyramid-like Fusion Module (SPFM) that refines local image features under the guidance of attention pyramid layers at different scales, enhancing feature discriminability. Additionally, to evaluate detection performance on inpainted images, we establish a comprehensive benchmark comprising images generated from five distinct masked regions. Extensive experiments demonstrate that our End4 effectively generalizes to unseen masking patterns and remains robust under various perturbations. Our code and dataset will be released soon.
Abstract (translated)
扩散模型的强大生成能力在图像合成领域取得了显著进展，不仅提高了完整图像的生成质量，还增强了基于修复（inpainting）的图像编辑技术。尽管这些模型取得了非凡的进步，但也引发了对可能被用于恶意目的的担忧。然而，现有的方法难以识别由基于扩散的修复模型生成的图像，即使训练数据中包含类似的修复后的图像也是如此。为了解决这一挑战，我们提出了一种新的检测方法，即端到端去噪扩散（End4）。具体来说，End4 设计了一个去噪重建模型，以提高重建和检测过程中潜在空间之间的对齐度，从而重构出更有利于检测的特征。同时，它利用了尺度感知金字塔状融合模块（SPFM），在不同尺度下的注意力金字塔层指导下细化局部图像特征，增强了特征的区分性。 为了评估修复图像上的检测性能，我们建立了一个全面的基准测试集，其中包括从五个不同的遮罩区域生成的图像。广泛的实验表明，我们的 End4 方法可以很好地推广到未见过的遮罩模式，并在各种扰动下保持稳健性。我们将很快发布代码和数据集。
URL
https://arxiv.org/abs/2509.13214
PDF
-
Human + AI for Accelerating Ad Localization Evaluation
2025-09-16 00:52:41Harshit Rajgarhia, Shivali Dalmia, Mengyang Zhao, Mukherji Abhishek, Kiran GaneshAbstract
Adapting advertisements for multilingual audiences requires more than simple text translation; it demands preservation of visual consistency, spatial alignment, and stylistic integrity across diverse languages and formats. We introduce a structured framework that combines automated components with human oversight to address the complexities of advertisement localization. To the best of our knowledge, this is the first work to integrate scene text detection, inpainting, machine translation (MT), and text reimposition specifically for accelerating ad localization evaluation workflows. Qualitative results across six locales demonstrate that our approach produces semantically accurate and visually coherent localized advertisements, suitable for deployment in real-world workflows.
Abstract (translated)
将广告适应为多语言受众的需求不仅仅局限于简单的文本翻译；这还需要在多种语言和格式下保持视觉一致性、空间对齐以及风格的完整性。我们提出了一种结构化的框架，该框架结合了自动化组件与人工监督，以解决广告本地化过程中遇到的各种复杂问题。据我们所知，这是首次将场景文本检测、图像修复（inpainting）、机器翻译（MT）和文本重新置入技术结合起来，专门用于加速广告本地化评估工作流程的研究。通过六个不同地区的定性结果证明，我们的方法能够生成语义准确且视觉上连贯的本地化广告，适合应用于实际的工作环境中。
URL
https://arxiv.org/abs/2509.12543
PDF
-
Artist-Created Mesh Generation from Raw Observation
2025-09-15 23:09:27Yao He, Youngjoong Kwon, Wenxiao Cai, Ehsan AdeliAbstract
We present an end-to-end framework for generating artist-style meshes from noisy or incomplete point clouds, such as those captured by real-world sensors like LiDAR or mobile RGB-D cameras. Artist-created meshes are crucial for commercial graphics pipelines due to their compatibility with animation and texturing tools and their efficiency in rendering. However, existing approaches often assume clean, complete inputs or rely on complex multi-stage pipelines, limiting their applicability in real-world scenarios. To address this, we propose an end-to-end method that refines the input point cloud and directly produces high-quality, artist-style meshes. At the core of our approach is a novel reformulation of 3D point cloud refinement as a 2D inpainting task, enabling the use of powerful generative models. Preliminary results on the ShapeNet dataset demonstrate the promise of our framework in producing clean, complete meshes.
Abstract (translated)
我们提出了一种端到端框架，用于从嘈杂或不完整的点云数据中生成符合艺术家风格的网格模型。这些点云数据通常由现实世界中的传感器（如激光雷达或移动RGB-D相机）捕获。艺术家创作的网格模型对于商业图形流水线至关重要，因为它们与动画和纹理工具兼容，并且在渲染效率方面表现出色。然而，现有的方法往往假设输入是干净而完整的，或者依赖于复杂的多阶段管道流程，这限制了它们在现实世界场景中的应用范围。 为了解决这些问题，我们提出了一种端到端的方法，该方法可以优化输入点云数据并直接生成高质量的艺术家风格网格模型。我们的方法核心在于将三维点云细化问题重新表述为二维图像修复任务（即2D inpainting），这样就可以利用强大的生成式模型进行处理。 在ShapeNet数据集上的初步结果显示了我们框架在产生干净且完整网格模型方面的潜力。
URL
https://arxiv.org/abs/2509.12501
PDF
-
LazyDrag: Enabling Stable Drag-Based Editing on Multi-Modal Diffusion Transformers via Explicit Correspondence
2025-09-15 17:59:47Zixin Yin, Xili Dai, Duomin Wang, Xianfang Zeng, Lionel M. Ni, Gang Yu, Heung-Yeung ShumAbstract
The reliance on implicit point matching via attention has become a core bottleneck in drag-based editing, resulting in a fundamental compromise on weakened inversion strength and costly test-time optimization (TTO). This compromise severely limits the generative capabilities of diffusion models, suppressing high-fidelity inpainting and text-guided creation. In this paper, we introduce LazyDrag, the first drag-based image editing method for Multi-Modal Diffusion Transformers, which directly eliminates the reliance on implicit point matching. In concrete terms, our method generates an explicit correspondence map from user drag inputs as a reliable reference to boost the attention control. This reliable reference opens the potential for a stable full-strength inversion process, which is the first in the drag-based editing task. It obviates the necessity for TTO and unlocks the generative capability of models. Therefore, LazyDrag naturally unifies precise geometric control with text guidance, enabling complex edits that were previously out of reach: opening the mouth of a dog and inpainting its interior, generating new objects like a ``tennis ball'', or for ambiguous drags, making context-aware changes like moving a hand into a pocket. Additionally, LazyDrag supports multi-round workflows with simultaneous move and scale operations. Evaluated on the DragBench, our method outperforms baselines in drag accuracy and perceptual quality, as validated by VIEScore and human evaluation. LazyDrag not only establishes new state-of-the-art performance, but also paves a new way to editing paradigms.
Abstract (translated)
依赖于注意力机制的隐式点匹配已成为基于拖拽编辑的核心瓶颈，导致在反转强度减弱和测试时间优化（TTO）成本高昂之间做出了根本性的妥协。这种妥协严重限制了扩散模型的生成能力，抑制了高保真度的修复和文本引导创作。在这篇论文中，我们引入了LazyDrag，这是首个用于多模态扩散变压器的基于拖拽图像编辑方法，它直接消除了对隐式点匹配的依赖。具体来说，我们的方法从用户拖拽输入生成一个显式的对应关系图作为可靠的参考以增强注意力控制。这个可靠参考开启了稳定全强度反转过程的可能性，这在基于拖拽的编辑任务中尚属首次。它消除了TTO的需求，并释放了模型的生成能力。因此，LazyDrag自然地统一了精确几何控制与文本引导，使得之前难以实现的复杂编辑成为可能：例如为一只狗打开嘴巴并修复其内部细节、生成新的物体如“网球”或对于模糊拖拽进行上下文感知变化，比如将手移入口袋中。此外，LazyDrag支持多轮工作流程，并且可以同时执行移动和缩放操作。在DragBench上评估时，我们的方法在拖拽准确性和感知质量方面超越了基准模型，这通过VIEScore和人工评价得到了验证。LazyDrag不仅建立了新的最先进的性能水平，而且还开辟了一条通往新编辑范式的道路。
URL
https://arxiv.org/abs/2509.12203
PDF
-
IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed
2025-09-15 07:16:03Yongzhe Lyu, Yu Wu, Yutian Lin, Bo DuAbstract
Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.
Abstract (translated)
扩散模型在自由形式的图像修复（inpainting）中展现出了很有前景的结果。近期基于改进的扩散采样器或新颖架构设计的研究，产生了逼真的结果和高度的数据一致性。然而，在标准的扩散过程中采用随机初始化种子（噪声），可能会导致掩码区域中的语义信息不匹配，从而产生有偏差的修复结果，例如与未掩码区域的一致性和连贯性较差。 为了解决这个问题，我们提出了初始种子优化扩散模型（Initial Seed refined Diffusion Model, IS-Diff）。这是一种完全无需训练的方法，通过采用分布和谐化的种子来生成和谐的结果。具体来说，IS-Diff 使用从未掩码区域采样的初始种子，以模仿被遮挡数据的分布，并为扩散过程设定一个有前景的方向。此外，我们提出了一种动态选择性细化机制，用于检测中间潜在层中的严重不和谐修复结果，并根据实际情况动态调整初始化先验的强度。 我们在CelebA-HQ、ImageNet和Places2数据集上对标准和大面积掩码图像修复任务进行了验证，结果显示了IS-Diff在所有评估指标上的优越性能，优于现有的最先进的图像修复方法。
URL
https://arxiv.org/abs/2509.11638
PDF
-
Mask Consistency Regularization in Object Removal
2025-09-12 14:02:52Hua Yuan, Jin Yuan, Yicheng Jiang, Yao Zhang, Xin Geng, Yong RuiAbstract
Object removal, a challenging task within image inpainting, involves seamlessly filling the removed region with content that matches the surrounding context. Despite advancements in diffusion models, current methods still face two critical challenges. The first is mask hallucination, where the model generates irrelevant or spurious content inside the masked region, and the second is mask-shape bias, where the model fills the masked area with an object that mimics the mask's shape rather than surrounding content. To address these issues, we propose Mask Consistency Regularization (MCR), a novel training strategy designed specifically for object removal tasks. During training, our approach introduces two mask perturbations: dilation and reshape, enforcing consistency between the outputs of these perturbed branches and the original mask. The dilated masks help align the model's output with the surrounding content, while reshaped masks encourage the model to break the mask-shape bias. This combination of strategies enables MCR to produce more robust and contextually coherent inpainting results. Our experiments demonstrate that MCR significantly reduces hallucinations and mask-shape bias, leading to improved performance in object removal.
Abstract (translated)
对象移除是图像修复中的一项具有挑战性的任务，它涉及到无缝地填充被移除的区域，并使其内容与周围环境相匹配。尽管在扩散模型方面已经取得了进展，但目前的方法仍然面临两个关键挑战：一是“掩码幻觉”，即模型会在遮罩区域内生成不相关或虚假的内容；二是“掩码形状偏见”，即模型会用与其掩码形状相似的对象填充被遮盖的区域，而不是根据周围环境进行填充。为了解决这些问题，我们提出了一种新的训练策略——Mask Consistency Regularization (MCR)，专门针对对象移除任务设计。 在训练过程中，我们的方法引入了两种掩码扰动：膨胀和重塑。前者通过使模型输出与周围内容对齐来强制这些扰动分支的输出与原始掩码保持一致性；后者则促使模型打破掩模形状偏见。这种策略组合使得MCR能够生成更加稳健且上下文一致的图像修复结果。 实验表明，MCR显著减少了幻觉和掩模形状偏差的发生，从而提高了对象移除任务中的性能。
URL
https://arxiv.org/abs/2509.10259
PDF
-
GAMMA: Generalizable Alignment via Multi-task and Manipulation-Augmented Training for AI-Generated Image Detection
2025-09-12 13:46:54Haozhen Yan, Yan Hong, Suning Lang, Jiahui Zhan, Yikun Ji, Yujie Gao, Jun Lan, Huijia Zhu, Weiqiang Wang, Jianfu ZhangAbstract
With generative models becoming increasingly sophisticated and diverse, detecting AI-generated images has become increasingly challenging. While existing AI-genereted Image detectors achieve promising performance on in-distribution generated images, their generalization to unseen generative models remains limited. This limitation is largely attributed to their reliance on generation-specific artifacts, such as stylistic priors and compression patterns. To address these limitations, we propose GAMMA, a novel training framework designed to reduce domain bias and enhance semantic alignment. GAMMA introduces diverse manipulation strategies, such as inpainting-based manipulation and semantics-preserving perturbations, to ensure consistency between manipulated and authentic content. We employ multi-task supervision with dual segmentation heads and a classification head, enabling pixel-level source attribution across diverse generative domains. In addition, a reverse cross-attention mechanism is introduced to allow the segmentation heads to guide and correct biased representations in the classification branch. Our method achieves state-of-the-art generalization performance on the GenImage benchmark, imporving accuracy by 5.8%, but also maintains strong robustness on newly released generative model such as GPT-4o.
Abstract (translated)
随着生成模型变得越来越复杂和多样化，检测AI生成的图像也变得更加具有挑战性。现有的针对AI生成图像的检测器在处理同分布（in-distribution）生成图像时表现出色，但它们对于未见过的新生成模型泛化能力有限。这一局限主要归因于这些检测器依赖于特定生成方式产生的特征，如风格先验和压缩模式。 为了克服这些限制，我们提出了GAMMA框架，这是一个新型的训练方法，旨在减少领域偏差并增强语义一致性。GAMMA引入了多样化的操作策略，包括基于修复（inpainting）的操作和保持语义不变的扰动，以确保经过操作的内容与真实内容具有一致性。我们采用了多任务监督机制，并设置了两个分割头和一个分类头，这使得我们的模型能够在不同生成领域内实现像素级来源归属。此外，引入了一个反向交叉注意机制（reverse cross-attention mechanism），允许分割头指导并纠正分类分支中的偏差表示。 在GenImage基准测试中，我们的方法实现了最先进的泛化性能，准确率提高了5.8%，同时对于新发布的生成模型如GPT-4o也保持了强大的鲁棒性。
URL
https://arxiv.org/abs/2509.10250
PDF
-
Prompt-Driven Image Analysis with Multimodal Generative AI: Detection, Segmentation, Inpainting, and Interpretation
2025-09-10 11:00:12Kaleem AhmadAbstract
Prompt-driven image analysis converts a single natural-language instruction into multiple steps: locate, segment, edit, and describe. We present a practical case study of a unified pipeline that combines open-vocabulary detection, promptable segmentation, text-conditioned inpainting, and vision-language description into a single workflow. The system works end to end from a single prompt, retains intermediate artifacts for transparent debugging (such as detections, masks, overlays, edited images, and before and after composites), and provides the same functionality through an interactive UI and a scriptable CLI for consistent, repeatable runs. We highlight integration choices that reduce brittleness, including threshold adjustments, mask inspection with light morphology, and resource-aware defaults. In a small, single-word prompt segment, detection and segmentation produced usable masks in over 90% of cases with an accuracy above 85% based on our criteria. On a high-end GPU, inpainting makes up 60 to 75% of total runtime under typical guidance and sampling settings, which highlights the need for careful tuning. The study offers implementation-guided advice on thresholds, mask tightness, and diffusion parameters, and details version pinning, artifact logging, and seed control to support replay. Our contribution is a transparent, reliable pattern for assembling modern vision and multimodal models behind a single prompt, with clear guardrails and operational practices that improve reliability in object replacement, scene augmentation, and removal.
Abstract (translated)
提示驱动的图像分析将自然语言指令转换为多个步骤：定位、分割、编辑和描述。我们提出了一种实用案例研究，展示了一个统一的工作流程管道，该管道结合了开放词汇检测、可提示分割、基于文本条件的修复（即在已有内容基础上生成缺失或损坏部分的新内容）、以及视觉-语言描述等功能。系统从单个提示开始，并且在整个过程中保留中间产物以实现透明调试（例如：检测结果、掩码、叠加层、编辑后的图像和前后对比合成）。该系统还通过交互式UI和脚本化CLI提供了相同的功能，确保一致性和可重复性运行。 我们强调了减少脆弱性的集成选择，如调整阈值、使用轻量级形态学进行掩模检查以及资源感知的默认设置。在小规模的一词提示分割实验中，检测与分割产生的可用掩码超过90%的情况达到了85%以上的准确率。在高端GPU上，在典型的指导和采样设定下，修复操作占据了总运行时间的60至75%，这强调了仔细调优的重要性。 这项研究提供了关于阈值、掩模紧致度以及扩散参数等方面的实施建议，并详细说明了版本固定、日志记录和种子控制以支持回放。我们的贡献在于提供了一种透明可靠的模式，用于在单个提示背后组装现代视觉及多模态模型，包括明确的保护措施和操作实践，这些有助于提高对象替换、场景增强以及移除等任务的可靠性。
URL
https://arxiv.org/abs/2509.08489
PDF
-
Sparse Transformer for Ultra-sparse Sampled Video Compressive Sensing
2025-09-10 02:03:06Miao Cao, Siming Zheng, Lishun Wang, Ziyang Chen, David Brady, Xin YuanAbstract
Digital cameras consume ~0.1 microjoule per pixel to capture and encode video, resulting in a power usage of ~20W for a 4K sensor operating at 30 fps. Imagining gigapixel cameras operating at 100-1000 fps, the current processing model is unsustainable. To address this, physical layer compressive measurement has been proposed to reduce power consumption per pixel by 10-100X. Video Snapshot Compressive Imaging (SCI) introduces high frequency modulation in the optical sensor layer to increase effective frame rate. A commonly used sampling strategy of video SCI is Random Sampling (RS) where each mask element value is randomly set to be 0 or 1. Similarly, image inpainting (I2P) has demonstrated that images can be recovered from a fraction of the image pixels. Inspired by I2P, we propose Ultra-Sparse Sampling (USS) regime, where at each spatial location, only one sub-frame is set to 1 and all others are set to 0. We then build a Digital Micro-mirror Device (DMD) encoding system to verify the effectiveness of our USS strategy. Ideally, we can decompose the USS measurement into sub-measurements for which we can utilize I2P algorithms to recover high-speed frames. However, due to the mismatch between the DMD and CCD, the USS measurement cannot be perfectly decomposed. To this end, we propose BSTFormer, a sparse TransFormer that utilizes local Block attention, global Sparse attention, and global Temporal attention to exploit the sparsity of the USS measurement. Extensive results on both simulated and real-world data show that our method significantly outperforms all previous state-of-the-art algorithms. Additionally, an essential advantage of the USS strategy is its higher dynamic range than that of the RS strategy. Finally, from the application perspective, the USS strategy is a good choice to implement a complete video SCI system on chip due to its fixed exposure time.
Abstract (translated)
数字相机每捕获和编码一个视频像素大约消耗0.1微焦耳的能量，这意味着对于30帧/秒运行的4K传感器来说，其功耗约为20瓦。设想一下千兆像素级别的摄像机以100到1000帧/秒的速度工作时，现有的处理模型显然不可持续。为了解决这个问题，人们提出了在物理层面上采用压缩测量来将每个像素的能量消耗减少10至100倍的方案。 视频快照压缩成像（Video Snapshot Compressive Imaging, SCI）通过在光学传感器层面引入高频调制技术，提高了有效帧率。一种常见的视频SCI采样策略是随机采样（Random Sampling, RS），其中每个掩模元素值被随机设置为0或1。同样地，图像修复（Image Inpainting, I2P）展示了仅从一部分像素中就可以恢复出完整图像的可能性。 受到I2P的启发，我们提出了一种超稀疏采样（Ultra-Sparse Sampling, USS）策略，在这一策略下，每个空间位置只有一个子帧被设置为1，其余所有都设为0。随后，我们构建了一个基于数字微镜设备（Digital Micromirror Device, DMD）的编码系统来验证USS策略的有效性。理想情况下，我们可以将USS测量分解成子测量，并利用I2P算法恢复高速图像帧。然而，由于DMD和CCD之间的不匹配问题，USS测量无法完美地被分解。 为此，我们提出了BSTFormer（Block、Sparse、Temporal Transformer），一种稀疏Transformer，它使用局部块注意机制、全局稀疏注意机制以及全局时间注意机制来利用USS测量的稀疏性。在模拟和真实世界数据集上的广泛实验结果表明，我们的方法显著优于所有先前的最佳算法。 此外，USS策略的一个关键优势是其动态范围比RS策略更大。从应用角度来看，由于固定曝光时间的特点，USS策略是一个实施完整视频SCI系统芯片的良好选择。
URL
https://arxiv.org/abs/2509.08228
PDF
-
SplatFill: 3D Scene Inpainting via Depth-Guided Gaussian Splatting
2025-09-09 14:47:47Mahtab Dahaghin, Milind G. Padalkar, Matteo Toso, Alessio Del BueAbstract
3D Gaussian Splatting (3DGS) has enabled the creation of highly realistic 3D scene representations from sets of multi-view images. However, inpainting missing regions, whether due to occlusion or scene editing, remains a challenging task, often leading to blurry details, artifacts, and inconsistent geometry. In this work, we introduce SplatFill, a novel depth-guided approach for 3DGS scene inpainting that achieves state-of-the-art perceptual quality and improved efficiency. Our method combines two key ideas: (1) joint depth-based and object-based supervision to ensure inpainted Gaussians are accurately placed in 3D space and aligned with surrounding geometry, and (2) we propose a consistency-aware refinement scheme that selectively identifies and corrects inconsistent regions without disrupting the rest of the scene. Evaluations on the SPIn-NeRF dataset demonstrate that SplatFill not only surpasses existing NeRF-based and 3DGS-based inpainting methods in visual fidelity but also reduces training time by 24.5%. Qualitative results show our method delivers sharper details, fewer artifacts, and greater coherence across challenging viewpoints.
Abstract (translated)
3D高斯点阵（3DGS）技术使得从多视角图像集合中创建高度逼真的3D场景表示成为可能。然而，填补由于遮挡或场景编辑导致的缺失区域仍然是一个艰巨的任务，这通常会导致模糊细节、伪影以及不一致的几何结构问题。在这项工作中，我们推出了SplatFill，这是一种新颖的基于深度引导的方法，用于解决3DGS场景中的缺口填充问题，它在感知质量和效率提升方面均达到了行业领先水平。我们的方法结合了两个关键理念：(1) 基于联合深度和对象监督的策略，确保填补的高斯点阵能够准确地放置在三维空间中，并且与周围几何结构对齐；(2) 我们提出了一种一致性感知的细化方案，能够有选择性地识别并修正不一致区域，而不会干扰场景的其他部分。在SPIn-NeRF数据集上的评估表明，SplatFill不仅在视觉保真度上超越了现有的基于NeRF和3DGS的缺口填充方法，而且还减少了24.5%的训练时间。定性结果显示我们的方法能够提供更清晰的细节、减少伪影，并且在具有挑战性的视角下表现出更好的一致性。
URL
https://arxiv.org/abs/2509.07809
PDF
-
MAIA: An Inpainting-Based Approach for Music Adversarial Attacks
2025-09-05 10:05:26Yuxuan Liu, Peihong Zhang, Rui Sang, Zhixin Li, Shengchen LiAbstract
Music adversarial attacks have garnered significant interest in the field of Music Information Retrieval (MIR). In this paper, we present Music Adversarial Inpainting Attack (MAIA), a novel adversarial attack framework that supports both white-box and black-box attack scenarios. MAIA begins with an importance analysis to identify critical audio segments, which are then targeted for modification. Utilizing generative inpainting models, these segments are reconstructed with guidance from the output of the attacked model, ensuring subtle and effective adversarial perturbations. We evaluate MAIA on multiple MIR tasks, demonstrating high attack success rates in both white-box and black-box settings while maintaining minimal perceptual distortion. Additionally, subjective listening tests confirm the high audio fidelity of the adversarial samples. Our findings highlight vulnerabilities in current MIR systems and emphasize the need for more robust and secure models.
Abstract (translated)
音乐对抗攻击在音乐信息检索（MIR）领域引起了极大的关注。本文介绍了Music Adversarial Inpainting Attack (MAIA)，这是一种新颖的对抗性攻击框架，支持白盒和黑盒攻击场景。MAIA首先进行重要性分析以识别关键音频片段，然后对这些片段进行修改。利用生成式修复模型，在被攻击模型输出的指导下重构这些片段，确保细微且有效的对抗性扰动。我们在多个MIR任务中评估了MAIA，展示了在白盒和黑盒设置下极高的攻击成功率，并保持了最小的感知失真。此外，主观听觉测试确认了对抗样本具有高音频保真度。我们的研究结果揭示了当前MIR系统中存在的漏洞，并强调需要开发更强大、更安全的模型。
URL
https://arxiv.org/abs/2509.04980
PDF
-
Inpaint4Drag: Repurposing Inpainting Models for Drag-Based Image Editing via Bidirectional Warping
Abstract
Drag-based image editing has emerged as a powerful paradigm for intuitive image manipulation. However, existing approaches predominantly rely on manipulating the latent space of generative models, leading to limited precision, delayed feedback, and model-specific constraints. Accordingly, we present Inpaint4Drag, a novel framework that decomposes drag-based editing into pixel-space bidirectional warping and image inpainting. Inspired by elastic object deformation in the physical world, we treat image regions as deformable materials that maintain natural shape under user manipulation. Our method achieves real-time warping previews (0.01s) and efficient inpainting (0.3s) at 512x512 resolution, significantly improving the interaction experience compared to existing methods that require minutes per edit. By transforming drag inputs directly into standard inpainting formats, our approach serves as a universal adapter for any inpainting model without architecture modification, automatically inheriting all future improvements in inpainting technology. Extensive experiments demonstrate that our method achieves superior visual quality and precise control while maintaining real-time performance. Project page: this https URL
Abstract (translated)
基于拖拽的图像编辑已成为直观操作图像的强大模式。然而，现有的方法主要依赖于生成模型潜在空间的操作，这导致了精度有限、反馈延迟以及特定模型的约束条件等问题。为此，我们提出了Inpaint4Drag这一新颖框架，它将基于拖拽的编辑分解为像素空间中的双向扭曲和图像修复（inpainting）。受到物理世界中弹性物体变形的启发，我们将图像区域视为在用户操作下可以保持自然形态的可变形材料。我们的方法实现了实时预览扭曲效果（0.01秒）以及高效的修补处理（0.3秒），在512x512分辨率下的表现相比现有需要几分钟才能完成一次编辑的方法有了显著改进。 通过将拖拽输入直接转换为标准的图像修复格式，我们这种方法可以作为任意图像修复模型的通用适配器，在不改变架构的情况下继承所有未来的技术改进。大量的实验表明，我们的方法在保持实时性能的同时达到了卓越的视觉质量和精确控制效果。项目页面：[此处请参考原始文本中的链接]
URL
https://arxiv.org/abs/2509.04582
PDF
