Inpainting
-
Common Inpainted Objects In-N-Out of Context
2025-05-31 21:42:12Tianze Yang, Tyson Jordan, Ninghao Liu, Jin SunAbstract
We present Common Inpainted Objects In-N-Out of Context (COinCO), a novel dataset addressing the scarcity of out-of-context examples in existing vision datasets. By systematically replacing objects in COCO images through diffusion-based inpainting, we create 97,722 unique images featuring both contextually coherent and inconsistent scenes, enabling effective context learning. Each inpainted object is meticulously verified and categorized as in- or out-of-context through a multimodal large language model assessment. Our analysis reveals significant patterns in semantic priors that influence inpainting success across object categories. We demonstrate three key tasks enabled by COinCO: (1) training context classifiers that effectively determine whether existing objects belong in their context; (2) a novel Objects-from-Context prediction task that determines which new objects naturally belong in given scenes at both instance and clique levels, and (3) context-enhanced fake detection on state-of-the-art methods without fine-tuning. COinCO provides a controlled testbed with contextual variations, establishing a foundation for advancing context-aware visual understanding in computer vision and image forensics. Our code and data are at: this https URL.
Abstract (translated)
我们介绍了“Common Inpainted Objects In-N-Out of Context”(简称COinCO)，这是一个新颖的数据集，旨在解决现有视觉数据集中缺少上下文外示例的问题。通过利用基于扩散的修复技术系统性地替换COCO图像中的对象，我们创建了97,722张独特的图像，这些图像是经过精心设计的，并且包含在上下文中一致和不一致的场景，从而能够有效地进行上下文学习。每个被修复的对象都经过多模态大型语言模型的评估，以确保其分类为上下文内或上下文外对象。我们的分析揭示了影响不同类别物体修复成功的语义先验模式。 COinCO支持三个关键任务：(1) 训练能够有效判断现有物体是否属于它们所处环境中的上下文分类器；(2) 一种新的“从上下文中预测物体”任务，该任务确定哪些新物体在给定场景中自然地存在，并且以实例和团簇级别进行评估；以及 (3) 在无需微调的情况下使用最先进的方法进行增强的假图像检测。 COinCO提供了一个具有上下文变化的受控测试平台，为计算机视觉和图像取证中的上下文感知视觉理解研究奠定了基础。我们的代码和数据可在以下网址获取：this https URL.
URL
https://arxiv.org/abs/2506.00721
PDF
-
DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds
2025-05-30 15:54:34Jiaxu Zhang, Xianfang Zeng, Xin Chen, Wei Zuo, Gang Yu, Guosheng Lin, Zhigang TuAbstract
This paper presents DreamDance, a novel character art animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories. To achieve this, we re-formulate the animation task as two inpainting-based steps: Camera-aware Scene Inpainting and Pose-aware Video Inpainting. The first step leverages a pre-trained image inpainting model to generate multi-view scene images from the reference art and optimizes a stable large-scale Gaussian field, which enables coarse background video rendering with camera trajectories. However, the rendered video is rough and only conveys scene motion. To resolve this, the second step trains a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality. Specifically, this model is a DiT-based video generation model with a gating strategy that adaptively integrates the character's appearance and pose information into the base background video. Through extensive experiments, we demonstrate the effectiveness and generalizability of DreamDance, producing high-quality and consistent character animations with remarkable camera dynamics.
Abstract (translated)
本文介绍了DreamDance，这是一种新颖的角色艺术动画框架，能够根据精确的相机轨迹生成稳定且一致的角色和场景运动。为了实现这一目标，我们将动画任务重新表述为两个基于修复的方法步骤：感知相机的场景修复（Camera-aware Scene Inpainting）和感知姿态的视频修复（Pose-aware Video Inpainting）。 第一个步骤利用预训练的图像修复模型从参考艺术作品生成多视角场景图像，并优化一个稳定的大型高斯场，从而启用带有相机轨迹的粗略背景视频渲染。然而，渲染出的视频粗糙且仅传达场景运动信息。为了改善这一点，第二个步骤通过训练一种感知姿态的视频修复模型来解决这一问题，该模型将动态角色注入到场景视频中，并提升背景质量。具体而言，这是一种基于DiT（Diffusion Transformer）的视频生成模型，采用了一种门控策略，能够根据自适应地整合角色的外观和姿态信息进基础背景视频。 通过广泛的实验，我们证明了DreamDance的有效性和泛化能力，它能生成高质量且一致的角色动画，并具有显著的相机动态特性。
URL
https://arxiv.org/abs/2505.24733
PDF
-
Video Editing for Audio-Visual Dubbing
2025-05-29 12:56:09Binyamin Manela, Sharon Gannot, Ethan FetyayaAbstract
Visual dubbing, the synchronization of facial movements with new speech, is crucial for making content accessible across different languages, enabling broader global reach. However, current methods face significant limitations. Existing approaches often generate talking faces, hindering seamless integration into original scenes, or employ inpainting techniques that discard vital visual information like partial occlusions and lighting variations. This work introduces EdiDub, a novel framework that reformulates visual dubbing as a content-aware editing task. EdiDub preserves the original video context by utilizing a specialized conditioning scheme to ensure faithful and accurate modifications rather than mere copying. On multiple benchmarks, including a challenging occluded-lip dataset, EdiDub significantly improves identity preservation and synchronization. Human evaluations further confirm its superiority, achieving higher synchronization and visual naturalness scores compared to the leading methods. These results demonstrate that our content-aware editing approach outperforms traditional generation or inpainting, particularly in maintaining complex visual elements while ensuring accurate lip synchronization.
Abstract (translated)
视觉配音，即同步新的语音与面部动作，对于跨语言传播内容至关重要，使其能够覆盖更广泛的全球受众。然而，目前的方法存在显著局限性。现有方法通常生成讲话人脸，难以无缝集成到原始场景中，或使用填补技术（inpainting techniques）丢弃部分遮挡和光照变化等重要视觉信息。这项工作介绍了一种名为EdiDub的新框架，它将视觉配音重新定义为一种内容感知编辑任务。通过采用专门的条件设置方案，EdiDub能够在保留原始视频背景的情况下进行忠实而准确的修改，而非简单的复制操作。 在多个基准测试中，包括一个具有挑战性的唇部被遮挡的数据集上，EdiDub显著提升了身份保存和同步性能。人类评估进一步确认了其优越性，在同步性和视觉自然度评分方面超越了现有领先方法。这些结果表明，我们的内容感知编辑方法优于传统的生成或填补技术，特别是在保持复杂视觉元素的同时确保精确的唇部同步。
URL
https://arxiv.org/abs/2505.23406
PDF
-
MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
Abstract
We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page: this https URL.
Abstract (translated)
我们介绍了MGE-LDM，这是一种统一的潜在扩散框架，用于同时生成音乐、填补缺失音源和基于查询驱动的声源分离。与以前仅限于固定乐器类别的方法不同，MGE-LDM在一个单一紧凑的潜在扩散模型中学习完整混音、子混音以及单独轨道（stem）之间的联合分布。在推理阶段，MGE-LDM能够实现： 1. 完整混音生成， 2. 部分生成（即填补缺失音源），和 3. 条件于文本的任意声源提取。 通过将分离与填补任务都作为潜在空间中的条件填充任务来表述，我们的方法支持灵活、不依赖类别属性地对任何乐器来源进行操作。值得注意的是，MGE-LDM可以在不同类型的多轨数据集（如Slakh2100、MUSDB18和MoisesDB）上联合训练，而无需依赖预定义的乐器类别。音频样本可在我们的项目页面找到：此链接。
URL
https://arxiv.org/abs/2505.23305
PDF
-
RSFAKE-1M: A Large-Scale Dataset for Detecting Diffusion-Generated Remote Sensing Forgeries
2025-05-29 09:30:46Zhihong Tan, Jiayi Wang, Huiying Shi, Binyuan Huang, Hongchen Wei, Zhenzhong ChenAbstract
Detecting forged remote sensing images is becoming increasingly critical, as such imagery plays a vital role in environmental monitoring, urban planning, and national security. While diffusion models have emerged as the dominant paradigm for image generation, their impact on remote sensing forgery detection remains underexplored. Existing benchmarks primarily target GAN-based forgeries or focus on natural images, limiting progress in this critical domain. To address this gap, we introduce RSFAKE-1M, a large-scale dataset of 500K forged and 500K real remote sensing images. The fake images are generated by ten diffusion models fine-tuned on remote sensing data, covering six generation conditions such as text prompts, structural guidance, and inpainting. This paper presents the construction of RSFAKE-1M along with a comprehensive experimental evaluation using both existing detectors and unified baselines. The results reveal that diffusion-based remote sensing forgeries remain challenging for current methods, and that models trained on RSFAKE-1M exhibit notably improved generalization and robustness. Our findings underscore the importance of RSFAKE-1M as a foundation for developing and evaluating next-generation forgery detection approaches in the remote sensing domain. The dataset and other supplementary materials are available at this https URL.
Abstract (translated)
检测伪造的遥感图像变得越来越重要，因为这类影像在环境监测、城市规划和国家安全中发挥着关键作用。虽然扩散模型已经成为图像生成的主要范式，但它们对遥感伪造检测的影响仍然未被充分探索。现有的基准测试主要针对基于GAN的伪造品或关注自然图像，这限制了这一关键领域的进展。为了解决这个缺口，我们引入了RSFAKE-1M，这是一个包含50万个伪造和50万个真实遥感图像的大规模数据集。这些假图像是通过十个在遥感数据上进行微调的扩散模型生成的，并涵盖了六种生成条件，如文本提示、结构指导和修补。本文介绍了RSFAKE-1M的数据构建方法及使用现有检测器与统一基线进行全面实验评估的结果。结果显示，基于扩散模型的遥感伪造品目前的方法仍然难以应对，而基于RSFAKE-1M训练的模型表现出显著改进的一般化能力和鲁棒性。我们的发现强调了RSFAKE-1M作为开发和评估下一代伪造检测方法在遥感领域中的重要基础的作用。该数据集和其他补充材料可以在 [此链接](https://example.com) 获取。
URL
https://arxiv.org/abs/2505.23283
PDF
-
UniTEX: Universal High Fidelity Generative Texturing for 3D Shapes
2025-05-29 08:58:41Yixun Liang, Kunming Luo, Xiao Chen, Rui Chen, Hongyu Yan, Weiyu Li, Jiarui Liu, Ping TanAbstract
We present UniTEX, a novel two-stage 3D texture generation framework to create high-quality, consistent textures for 3D assets. Existing approaches predominantly rely on UV-based inpainting to refine textures after reprojecting the generated multi-view images onto the 3D shapes, which introduces challenges related to topological ambiguity. To address this, we propose to bypass the limitations of UV mapping by operating directly in a unified 3D functional space. Specifically, we first propose that lifts texture generation into 3D space via Texture Functions (TFs)--a continuous, volumetric representation that maps any 3D point to a texture value based solely on surface proximity, independent of mesh topology. Then, we propose to predict these TFs directly from images and geometry inputs using a transformer-based Large Texturing Model (LTM). To further enhance texture quality and leverage powerful 2D priors, we develop an advanced LoRA-based strategy for efficiently adapting large-scale Diffusion Transformers (DiTs) for high-quality multi-view texture synthesis as our first stage. Extensive experiments demonstrate that UniTEX achieves superior visual quality and texture integrity compared to existing approaches, offering a generalizable and scalable solution for automated 3D texture generation. Code will available in: this https URL.
Abstract (translated)
我们介绍了UniTEX，这是一种新颖的两阶段3D纹理生成框架，用于为3D资产创建高质量且一致的纹理。现有方法主要依赖于基于UV的修复技术，在将生成的多视角图像重新投影到3D形状后进行纹理细化，这引入了与拓扑歧义相关的挑战。为了克服这一限制，我们提出直接在统一的3D功能空间中操作，绕过UV映射的局限性。具体而言，我们首先提出了通过Texture Functions (TFs) 将纹理生成提升到三维空间的方法——这是一种连续的体素表示形式，它根据表面距离将任何3D点映射到一个纹理值，与网格拓扑无关。 然后，我们提出使用基于变压器的大规模纹理模型（LTM）直接从图像和几何输入预测这些TFs。为了进一步提高纹理质量并利用强大的2D先验知识，我们开发了一种先进的LoRA策略，可以有效地调整大规模Diffusion Transformers (DiTs) 以进行高质量的多视角纹理合成作为我们的第一阶段。 广泛的实验表明，UniTEX在视觉质量和纹理完整性方面优于现有的方法，为自动化的3D纹理生成提供了通用且可扩展的解决方案。代码将在此 https URL 上提供。
URL
https://arxiv.org/abs/2505.23253
PDF
-
ObjectClear: Complete Object Removal via Object-Effect Attention
2025-05-28 17:51:17Jixin Zhao, Shangchen Zhou, Zhouxia Wang, Peiqing Yang, Chen Change LoyAbstract
Object removal requires eliminating not only the target object but also its effects, such as shadows and reflections. However, diffusion-based inpainting methods often produce artifacts, hallucinate content, alter background, and struggle to remove object effects accurately. To address this challenge, we introduce a new dataset for OBject-Effect Removal, named OBER, which provides paired images with and without object effects, along with precise masks for both objects and their associated visual artifacts. The dataset comprises high-quality captured and simulated data, covering diverse object categories and complex multi-object scenes. Building on OBER, we propose a novel framework, ObjectClear, which incorporates an object-effect attention mechanism to guide the model toward the foreground removal regions by learning attention masks, effectively decoupling foreground removal from background reconstruction. Furthermore, the predicted attention map enables an attention-guided fusion strategy during inference, greatly preserving background details. Extensive experiments demonstrate that ObjectClear outperforms existing methods, achieving improved object-effect removal quality and background fidelity, especially in complex scenarios.
Abstract (translated)
对象移除不仅需要消除目标物体，还需要处理其影响，如阴影和反射。然而，基于扩散的修复方法常常会产生伪影、生成虚构内容、改变背景，并且难以准确去除物体效果。为了解决这一挑战，我们引入了一个新的数据集OBject-Effect Removal（OBER），该数据集提供了包含与不包含物体效应的成对图像，以及精确的对象及其相关视觉瑕疵掩码。此数据集包括高质量的真实拍摄和模拟数据，涵盖了多种对象类别及复杂的多对象场景。 基于OBER数据集，我们提出了一个创新框架ObjectClear，它结合了对象影响注意机制，通过学习关注掩模来指导模型聚焦于前景移除区域，从而有效地将前景移除与背景重建解耦。此外，在推理阶段利用预测的注意力图能够实现一种引导融合策略，大大保留了背景细节。 广泛实验表明，ObjectClear在复杂场景中显著优于现有方法，实现了改进的对象影响去除质量和背景保真度。
URL
https://arxiv.org/abs/2505.22636
PDF
-
Identity-Preserving Text-to-Image Generation via Dual-Level Feature Decoupling and Expert-Guided Fusion
Abstract
Recent advances in large-scale text-to-image generation models have led to a surge in subject-driven text-to-image generation, which aims to produce customized images that align with textual descriptions while preserving the identity of specific subjects. Despite significant progress, current methods struggle to disentangle identity-relevant information from identity-irrelevant details in the input images, resulting in overfitting or failure to maintain subject identity. In this work, we propose a novel framework that improves the separation of identity-related and identity-unrelated features and introduces an innovative feature fusion mechanism to improve the quality and text alignment of generated images. Our framework consists of two key components: an Implicit-Explicit foreground-background Decoupling Module (IEDM) and a Feature Fusion Module (FFM) based on a Mixture of Experts (MoE). IEDM combines learnable adapters for implicit decoupling at the feature level with inpainting techniques for explicit foreground-background separation at the image level. FFM dynamically integrates identity-irrelevant features with identity-related features, enabling refined feature representations even in cases of incomplete decoupling. In addition, we introduce three complementary loss functions to guide the decoupling process. Extensive experiments demonstrate the effectiveness of our proposed method in enhancing image generation quality, improving flexibility in scene adaptation, and increasing the diversity of generated outputs across various textual descriptions.
Abstract (translated)
近期在大规模文本到图像生成模型方面的进展引发了以主题为导向的文本到图像生成领域的激增，该领域旨在根据文本描述生成定制化且与特定主体身份一致的图像。尽管已有显著进步，但当前方法仍难以从输入图像中分离出与身份相关的信息和无关信息，导致过度拟合或无法保持主体的身份特征。为此，我们提出了一种新的框架，以改进身份相关和不相关的特性的分离，并引入了一个基于专家混合（MoE）的创新特性融合机制来提高生成图像的质量及其与文本描述的一致性。 该框架包含两个关键组件：隐式-显式前景背景解耦模块（IEDM）以及特征融合模块（FFM）。IEDM结合了可学习适配器用于在特征级别上进行隐式分离，同时采用修复技术进行图像级别的显式前景背景分割。FFM动态地将身份无关的特性与身份相关的特性相结合，在不完全分离的情况下也能实现精细的特征求解表示。 此外，我们还引入了三种互补的损失函数来指导整个解耦过程。大量的实验展示了我们的方法在提高图像生成质量、增强场景适应性灵活性以及增加根据各种文本描述产生的输出多样性方面具有显著效果。
URL
https://arxiv.org/abs/2505.22360
PDF
-
DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation
2025-05-28 01:25:27Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, Shuran SongAbstract
We present DexUMI - a data collection and policy learning framework that uses the human hand as the natural interface to transfer dexterous manipulation skills to various robot hands. DexUMI includes hardware and software adaptations to minimize the embodiment gap between the human hand and various robot hands. The hardware adaptation bridges the kinematics gap using a wearable hand exoskeleton. It allows direct haptic feedback in manipulation data collection and adapts human motion to feasible robot hand motion. The software adaptation bridges the visual gap by replacing the human hand in video data with high-fidelity robot hand inpainting. We demonstrate DexUMI's capabilities through comprehensive real-world experiments on two different dexterous robot hand hardware platforms, achieving an average task success rate of 86%.
Abstract (translated)
我们介绍了DexUMI——一个数据采集和策略学习框架，该框架利用人手作为自然接口，将灵巧的操作技能转移到各种机器人手中。DexUMI 包括硬件和软件的适应性调整，以最小化人手与不同机器人手之间的实体差距。硬件方面的调整通过穿戴式手外骨骼来弥补运动学上的差异，它允许在操作数据收集过程中直接进行触觉反馈，并将人类的动作转换为机器人手可执行的动作。软件方面的调整则通过在视频数据中用高保真的机器人手图像替换人手来弥合视觉差距。我们通过在两种不同的灵巧机器人手上进行了全面的现实世界实验，展示了DexUMI 的功能，实现了86% 的平均任务成功率。
URL
https://arxiv.org/abs/2505.21864
PDF
-
OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers
2025-05-27 17:20:38Ziqiao Peng, Jiwen Liu, Haoxian Zhang, Xiaoqiang Liu, Songlin Tang, Pengfei Wan, Di Zhang, Hongyan Liu, Jun HeAbstract
Lip synchronization is the task of aligning a speaker's lip movements in video with corresponding speech audio, and it is essential for creating realistic, expressive video content. However, existing methods often rely on reference frames and masked-frame inpainting, which limit their robustness to identity consistency, pose variations, facial occlusions, and stylized content. In addition, since audio signals provide weaker conditioning than visual cues, lip shape leakage from the original video will affect lip sync quality. In this paper, we present OmniSync, a universal lip synchronization framework for diverse visual scenarios. Our approach introduces a mask-free training paradigm using Diffusion Transformer models for direct frame editing without explicit masks, enabling unlimited-duration inference while maintaining natural facial dynamics and preserving character identity. During inference, we propose a flow-matching-based progressive noise initialization to ensure pose and identity consistency, while allowing precise mouth-region editing. To address the weak conditioning signal of audio, we develop a Dynamic Spatiotemporal Classifier-Free Guidance (DS-CFG) mechanism that adaptively adjusts guidance strength over time and space. We also establish the AIGC-LipSync Benchmark, the first evaluation suite for lip synchronization in diverse AI-generated videos. Extensive experiments demonstrate that OmniSync significantly outperforms prior methods in both visual quality and lip sync accuracy, achieving superior results in both real-world and AI-generated videos.
Abstract (translated)
嘴唇同步是指将视频中说话者的唇部动作与相应的语音音频对齐的任务，对于创建真实且具有表现力的视频内容至关重要。然而，现有的方法往往依赖于参考帧和掩码帧修复技术（如填补缺失或损坏的信息），这限制了它们在身份一致性、姿态变化、面部遮挡以及风格化内容方面的鲁棒性。此外，由于音频信号提供的引导信息弱于视觉线索，来自原始视频的唇形泄漏将影响嘴唇同步的质量。 本文介绍了OmniSync——一个用于多种视觉场景的通用嘴唇同步框架。我们的方法引入了一种无需掩码的训练范式，利用扩散变换模型直接进行帧编辑，而不需要明确的掩码指示，从而支持不限时长的推理，并保持自然的面部动态并保留人物身份。在推理阶段，我们提出了一种基于流匹配的渐进噪声初始化方法，以确保姿态和身份的一致性，同时允许精确调整嘴部区域。 为了应对音频信号较弱引导信息的问题，我们开发了动态时空无分类器指引（Dynamic Spatiotemporal Classifier-Free Guidance, DS-CFG）机制，该机制能够随时间和空间自适应地调节指导强度。此外，我们建立了AIGC-LipSync基准测试集，这是第一个用于评估多样化AI生成视频中嘴唇同步效果的评测套件。 大量的实验表明，OmniSync在视觉质量和嘴唇同步准确度方面显著优于先前的方法，在现实世界和AI生成视频中都表现出色。
URL
https://arxiv.org/abs/2505.21448
PDF
-
Facial Attribute Based Text Guided Face Anonymization
2025-05-27 10:36:35Mustafa \.Izzet Mu\c{s}tu, Haz{\i}m Kemal EkenelAbstract
The increasing prevalence of computer vision applications necessitates handling vast amounts of visual data, often containing personal information. While this technology offers significant benefits, it should not compromise privacy. Data privacy regulations emphasize the need for individual consent for processing personal data, hindering researchers' ability to collect high-quality datasets containing the faces of the individuals. This paper presents a deep learning-based face anonymization pipeline to overcome this challenge. Unlike most of the existing methods, our method leverages recent advancements in diffusion-based inpainting models, eliminating the need for training Generative Adversarial Networks. The pipeline employs a three-stage approach: face detection with RetinaNet, feature extraction with VGG-Face, and realistic face generation using the state-of-the-art BrushNet diffusion model. BrushNet utilizes the entire image, face masks, and text prompts specifying desired facial attributes like age, ethnicity, gender, and expression. This enables the generation of natural-looking images with unrecognizable individuals, facilitating the creation of privacy-compliant datasets for computer vision research.
Abstract (translated)
计算机视觉应用程序的日益普及需要处理大量包含个人信息的视觉数据。虽然这项技术提供了许多好处，但不应以牺牲隐私为代价。数据隐私法规强调了在处理个人信息时获取个体同意的重要性，这限制了研究人员收集高质量的人脸数据集的能力。本文提出了一种基于深度学习的脸部匿名化管道，旨在克服这一挑战。 与大多数现有方法不同，我们的方法利用了基于扩散的图像修复模型的最新进展，从而无需训练生成对抗网络（GAN）。该管道采用三阶段方法：使用RetinaNet进行脸部检测、用VGG-Face提取特征以及利用最先进的BrushNet扩散模型生成逼真的面部图像。BrushNet利用整个图像、脸部掩码和文本提示（包括所需的面部属性如年龄、种族、性别和表情）来生成自然且无法识别个人身份的图像，从而促进计算机视觉研究中符合隐私合规的数据集的创建。
URL
https://arxiv.org/abs/2505.21002
PDF
-
What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models
2025-05-26 18:00:10Lorenzo Baraldi, Davide Bucciarelli, Federico Betti, Marcella Cornia, Lorenzo Baraldi, Nicu Sebe, Rita CucchiaraAbstract
Instruction-based image editing models offer increased personalization opportunities in generative tasks. However, properly evaluating their results is challenging, and most of the existing metrics lag in terms of alignment with human judgment and explainability. To tackle these issues, we introduce DICE (DIfference Coherence Estimator), a model designed to detect localized differences between the original and the edited image and to assess their relevance to the given modification request. DICE consists of two key components: a difference detector and a coherence estimator, both built on an autoregressive Multimodal Large Language Model (MLLM) and trained using a strategy that leverages self-supervision, distillation from inpainting networks, and full supervision. Through extensive experiments, we evaluate each stage of our pipeline, comparing different MLLMs within the proposed framework. We demonstrate that DICE effectively identifies coherent edits, effectively evaluating images generated by different editing models with a strong correlation with human judgment. We publicly release our source code, models, and data.
Abstract (translated)
指令驱动的图像编辑模型为生成任务提供了更多的个性化机会。然而，对其结果进行恰当评估颇具挑战性，并且现有的大多数指标在与人类判断的一致性和可解释性方面存在不足。为了应对这些问题，我们引入了DICE（DIfference Coherence Estimator），这是一种旨在检测原始图像和编辑后图像之间的局部差异并评估这些差异对给定修改请求的相关性的模型。DICE由两个关键组件组成：一个差分检测器和一个一致性估算器，它们都基于自回归多模态大型语言模型（MLLM）构建，并通过一种利用自我监督、从修复网络的蒸馏以及完全监督的方法进行训练。 通过广泛的实验，我们评估了管道中的每个阶段，并在所提出的框架内比较了不同的MLLM。结果显示，DICE能够有效识别连贯的编辑，并且可以有效地对由不同编辑模型生成的图像进行评价，其结果与人类判断有很强的相关性。我们将源代码、模型和数据公开发布。
URL
https://arxiv.org/abs/2505.20405
PDF
-
In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
2025-05-26 17:49:10Yu Xu, Fan Tang, You Wu, Lin Gao, Oliver Deussen, Hongbin Yan, Jintao Li, Juan Cao, Tong-Yee LeeAbstract
Recent advances in diffusion models have enhanced multimodal-guided visual generation, enabling customized subject insertion that seamlessly "brushes" user-specified objects into a given image guided by textual prompts. However, existing methods often struggle to insert customized subjects with high fidelity and align results with the user's intent through textual prompts. In this work, we propose "In-Context Brush", a zero-shot framework for customized subject insertion by reformulating the task within the paradigm of in-context learning. Without loss of generality, we formulate the object image and the textual prompts as cross-modal demonstrations, and the target image with the masked region as the query. The goal is to inpaint the target image with the subject aligning textual prompts without model tuning. Building upon a pretrained MMDiT-based inpainting network, we perform test-time enhancement via dual-level latent space manipulation: intra-head "latent feature shifting" within each attention head that dynamically shifts attention outputs to reflect the desired subject semantics and inter-head "attention reweighting" across different heads that amplifies prompt controllability through differential attention prioritization. Extensive experiments and applications demonstrate that our approach achieves superior identity preservation, text alignment, and image quality compared to existing state-of-the-art methods, without requiring dedicated training or additional data collection.
Abstract (translated)
最近的扩散模型在多模态引导视觉生成方面的进步，增强了定制对象插入功能，使得用户可以通过文本提示将特定物体无缝“刷入”给定图像。然而，现有的方法通常难以以高保真度插入定制对象，并且很难通过文本提示使结果与用户的意图对齐。为此，我们提出了“In-Context Brush”，这是一种零样本框架，用于基于上下文学习范式重新定义任务的定制对象插入。 具体来说，在不失一般性的前提下，我们将目标图像及其掩码区域作为查询（query），将对象图像和文本提示视为跨模态演示（cross-modal demonstrations）。我们的目标是在不调整模型的情况下，使用这些跨模态信息来填充目标图像中的空缺部分，并确保生成的对象与文本提示对齐。 基于预训练的MMDiT基础填补网络，我们通过双层隐空间操作在测试时间进行增强：在同一注意头内的“隐特征偏移”（latent feature shifting），动态调整注意力输出以反映所需对象语义；以及跨不同头部间的“注意重新加权”（attention reweighting），通过差异化注意力优先级来放大提示控制能力。 广泛的实验和应用表明，与现有最先进的方法相比，我们的方法在身份保留、文本对齐和图像质量方面均表现更优，并且无需专门训练或额外的数据收集。
URL
https://arxiv.org/abs/2505.20271
PDF
-
Deep Spectral Prior
2025-05-26 12:00:37Yanqi Cheng, Tieyong Zeng, Pietro Lio, Carola-Bibiane Sch\"onlieb, Angelica I Aviles-RiveroAbstract
We introduce Deep Spectral Prior (DSP), a new formulation of Deep Image Prior (DIP) that redefines image reconstruction as a frequency-domain alignment problem. Unlike traditional DIP, which relies on pixel-wise loss and early stopping to mitigate overfitting, DSP directly matches Fourier coefficients between the network output and observed measurements. This shift introduces an explicit inductive bias towards spectral coherence, aligning with the known frequency structure of images and the spectral bias of convolutional neural networks. We provide a rigorous theoretical framework demonstrating that DSP acts as an implicit spectral regulariser, suppressing high-frequency noise by design and eliminating the need for early stopping. Our analysis spans four core dimensions establishing smooth convergence dynamics, local stability, and favourable bias-variance tradeoffs. We further show that DSP naturally projects reconstructions onto a frequency-consistent manifold, enhancing interpretability and robustness. These theoretical guarantees are supported by empirical results across denoising, inpainting, and super-resolution tasks, where DSP consistently outperforms classical DIP and other unsupervised baselines.
Abstract (translated)
我们介绍了深度频谱先验（DSP），这是一种新的深度图像先验（DIP）公式，它将图像重建定义为一个频率域对齐问题。与传统的DIP依赖于像素级损失和提前停止以减少过拟合不同，DSP直接匹配网络输出和观测测量之间的傅里叶系数。这种转变引入了明确的归纳偏差，倾向于频谱一致性，这符合已知的图像频率结构以及卷积神经网络的频谱偏置。我们提供了一个严格的理论框架，证明DSP作为隐式频谱正则化器的作用，在设计上抑制高频噪声，并消除了提前停止的需求。我们的分析涵盖了四个核心维度，建立了平滑收敛动力学、局部稳定性及有利偏差-方差权衡。此外，我们还展示了DSP自然地将重建投影到一致的频率流形上，增强了可解释性和鲁棒性。这些理论保证得到了去噪、修复和超分辨率任务中经验结果的支持，在这些任务中，DSP始终优于传统的DIP和其他无监督基准方法。
URL
https://arxiv.org/abs/2505.19873
PDF
-
Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation
2025-05-26 02:32:22Yuhao He, Jinyu Tian, Haiwei Wu, Jianqing LiAbstract
The rapid advancement of diffusion models has enhanced their image inpainting and editing capabilities but also introduced significant societal risks. Adversaries can exploit user images from social media to generate misleading or harmful content. While adversarial perturbations can disrupt inpainting, global perturbation-based methods fail in mask-guided editing tasks due to spatial constraints. To address these challenges, we propose Structure Disruption Attack (SDA), a powerful protection framework for safeguarding sensitive image regions against inpainting-based editing. Building upon the contour-focused nature of self-attention mechanisms of diffusion models, SDA optimizes perturbations by disrupting queries in self-attention during the initial denoising step to destroy the contour generation process. This targeted interference directly disrupts the structural generation capability of diffusion models, effectively preventing them from producing coherent images. We validate our motivation through visualization techniques and extensive experiments on public datasets, demonstrating that SDA achieves state-of-the-art (SOTA) protection performance while maintaining strong robustness.
Abstract (translated)
扩散模型的迅速发展增强了其图像修复和编辑能力，但也带来了重要的社会风险。对手可以利用社交媒体上的用户图片生成误导性或有害内容。虽然对抗性扰动能够干扰图像修复过程，但基于全局扰动的方法在引导式编辑任务中由于空间约束而失效。为了解决这些挑战，我们提出了结构破坏攻击（SDA），这是一种强大的保护框架，用于防御扩散模型对敏感图像区域进行修复式编辑的威胁。基于扩散模型中的自注意力机制以轮廓为重点的特点，SDA通过在初始去噪步骤中干扰自注意力中的查询来优化扰动，从而破坏轮廓生成过程。这种有针对性的干扰直接中断了扩散模型的结构生成能力，有效防止其产生连贯图像。我们通过可视化技术和在公开数据集上的广泛实验验证了我们的动机，证明SDA在保持强大鲁棒性的同时达到了最先进的（SOTA）保护性能。
URL
https://arxiv.org/abs/2505.19425
PDF
-
Advancing Limited-Angle CT Reconstruction Through Diffusion-Based Sinogram Completion
2025-05-26 00:59:58Jiaqi Guo, Santiago Lopez-Tapia, Aggelos K. KatsaggelosAbstract
Limited Angle Computed Tomography (LACT) often faces significant challenges due to missing angular information. Unlike previous methods that operate in the image domain, we propose a new method that focuses on sinogram inpainting. We leverage MR-SDEs, a variant of diffusion models that characterize the diffusion process with mean-reverting stochastic differential equations, to fill in missing angular data at the projection level. Furthermore, by combining distillation with constraining the output of the model using the pseudo-inverse of the inpainting matrix, the diffusion process is accelerated and done in a step, enabling efficient and accurate sinogram completion. A subsequent post-processing module back-projects the inpainted sinogram into the image domain and further refines the reconstruction, effectively suppressing artifacts while preserving critical structural details. Quantitative experimental results demonstrate that the proposed method achieves state-of-the-art performance in both perceptual and fidelity quality, offering a promising solution for LACT reconstruction in scientific and clinical applications.
Abstract (translated)
限制角度计算机断层扫描（LACT）常常由于缺少角度信息而面临重大挑战。与以往在图像域操作的方法不同，我们提出了一种新的方法，专注于正弦图的修补（sinogram inpainting）。我们利用MR-SDEs（一种用均值回复随机微分方程描述扩散过程的扩散模型变体），在投影层面填充缺失的角度数据。此外，通过结合蒸馏技术和使用正弦图修补矩阵伪逆来约束模型输出，可以加速扩散过程并在一步内完成，从而实现高效且精确的正弦图补全。随后的一个后处理模块将修补后的正弦图反投射到图像域，并进一步优化重建结果，有效抑制了伪影并保留了关键结构细节。定量实验结果显示，该方法在感知和保真度质量方面均达到了最先进的性能，为LACT重建在科研和临床应用中提供了一种有前景的解决方案。
URL
https://arxiv.org/abs/2505.19385
PDF
-
What Do You Need for Diverse Trajectory Stitching in Diffusion Planning?
2025-05-23 16:41:08Quentin Clark, Florian ShkurtiAbstract
In planning, stitching is an ability of algorithms to piece together sub-trajectories of data they are trained on to generate new and diverse behaviours. While stitching is historically a strength of offline reinforcement learning, recent generative behavioural cloning (BC) methods have also shown proficiency at stitching. However, the main factors behind this are poorly understood, hindering the development of new algorithms that can reliably stitch. Focusing on diffusion planners trained via BC, we find two properties are needed to compose: \emph{positional equivariance} and \emph{local receptiveness}. We use these two properties to explain architecture, data, and inference choices in existing generative BC methods based on diffusion planning, including replanning frequency, data augmentation, and data scaling. Experimental comparisions show that (1) while locality is more important than positional equivariance in creating a diffusion planner capable of composition, both are crucial (2) enabling these properties through relatively simple architecture choices can be competitive with more computationally expensive methods such as replanning or scaling data, and (3) simple inpainting-based guidance can guide architecturally compositional models to enable generalization in goal-conditioned settings.
Abstract (translated)
在规划过程中，缝合（stitching）是算法将它们训练时所使用的数据子轨迹拼接起来以生成新的、多样的行为的能力。虽然历史上离线强化学习在缝合方面表现出色，但最近的生成式行为克隆（Behavior Cloning, BC）方法也展示了其在这方面的能力。然而，导致这种能力的主要因素尚不明确，这阻碍了开发能够可靠地进行缝合的新算法的发展。 通过研究基于行为克隆训练的扩散规划器，我们发现两种属性是必要的：\emph{位置等变性}（positional equivariance）和\emph{局部响应性}（local receptiveness）。这两种性质被用来解释现有的基于扩散规划的生成式BC方法中的架构、数据集及推理选择，包括重新计划频率、数据增强以及数据缩放。实验比较表明： 1. 尽管在创建能够进行组合的扩散规划器时，局部响应性比位置等变性更为重要，但两者都至关重要。 2. 通过简单的架构选择来实现这些性质可以与更耗费计算资源的方法（如重新计划或扩大数据集）相竞争。 3. 简单的修补指导法可以帮助具有可组合架构的模型在目标条件设定下进行泛化。 总的来说，这项研究揭示了基于扩散规划器的生成式行为克隆方法中的关键性质，并为如何通过简单的架构设计来实现这些特性以增强算法的性能提供了见解。
URL
https://arxiv.org/abs/2505.18083
PDF
-
M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversion
2025-05-22 11:58:54Nina Shvetsova, Goutam Bhat, Prune Truong, Hilde Kuehne, Federico TombariAbstract
We tackle the problem of monocular-to-stereo video conversion and propose a novel architecture for inpainting and refinement of the warped right view obtained by depth-based reprojection of the input left view. We extend the Stable Video Diffusion (SVD) model to utilize the input left video, the warped right video, and the disocclusion masks as conditioning input to generate a high-quality right camera view. In order to effectively exploit information from neighboring frames for inpainting, we modify the attention layers in SVD to compute full attention for discoccluded pixels. Our model is trained to generate the right view video in an end-to-end manner by minimizing image space losses to ensure high-quality generation. Our approach outperforms previous state-of-the-art methods, obtaining an average rank of 1.43 among the 4 compared methods in a user study, while being 6x faster than the second placed method.
Abstract (translated)
我们解决了单目到立体视频转换的问题，并提出了一种新颖的架构，用于对通过基于深度的重投影从输入左视图获得的扭曲右视图进行修复和细化。我们将Stable Video Diffusion (SVD) 模型扩展为利用输入左视视频、扭曲后的右视图以及遮挡排除掩码作为条件输入来生成高质量的右摄像机视图。为了有效利用相邻帧的信息来进行修补，我们修改了SVD中的注意力层以对被遮挡像素计算完全注意（full attention）。我们的模型通过最小化图像空间损失，在端到端的方式下训练以生成高质量的右视视频。在用户研究中，与四个比较方法相比，我们的方法平均排名为1.43，同时比第二名的方法快6倍。
URL
https://arxiv.org/abs/2505.16565
PDF
-
Constructing a 3D Town from a Single Image
2025-05-21 17:10:47Kaizhi Zheng, Ruijian Zhang, Jing Gu, Jie Yang, Xin Eric WangAbstract
Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes. In this work, we introduce 3DTown, a training-free framework designed to synthesize realistic and coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning. Extensive experiments across diverse scenes show that 3DTown outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, and TripoSG, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality 3D town generation is achievable from a single image using a principled, training-free approach.
Abstract (translated)
获取详细的三维场景通常需要昂贵的设备、多视角数据或复杂的建模过程。因此，一种轻量级的方法——从单一顶视图图像生成复杂三维场景，在实际应用中扮演着重要角色。尽管最近的三维生成模型在物体级别的表现非常出色，但它们扩展到全场景生成时往往会导致不一致的几何结构、布局幻觉以及低质量的网格。为此，我们在本研究中引入了3DTown，这是一个无需训练框架，旨在从单一顶视图图像合成真实且连贯的三维场景。 我们的方法基于两个原则：区域化生成以提高二维到三维的一致性和分辨率，以及空间感知的三维修复填充来确保全局场景的一致性及高质量几何结构生成。具体而言，我们将输入图像分解为重叠的区域，并使用预训练的三维对象生成器生成每个区域；随后进行掩码修正流修复填充过程，以填补缺失的几何信息并保持结构连续性。 这种模块化设计允许我们克服分辨率瓶颈问题，并在无需三维监督或微调的情况下保存空间结构。通过在各种场景中进行广泛的实验表明，3DTown 在几何质量、空间一致性以及纹理保真度方面均优于当前最佳基线模型（包括 Trellis, Hunyuan3D-2 和 TripoSG）。 我们的研究成果展示了基于单一图像生成高质量三维城镇的可行性，并且采用了一种无需训练的原理性方法。
URL
https://arxiv.org/abs/2505.15765
PDF
-
Anti-Inpainting: A Proactive Defense against Malicious Diffusion-based Inpainters under Unknown Conditions
2025-05-19 12:07:29Yimao Guo, Zuomin Qu, Wei Lu, Xiangyang LuoAbstract
As diffusion-based malicious image manipulation becomes increasingly prevalent, multiple proactive defense methods are developed to safeguard images against unauthorized tampering. However, most proactive defense methods only can safeguard images against manipulation under known conditions, and fail to protect images from manipulations guided by tampering conditions crafted by malicious users. To tackle this issue, we propose Anti-Inpainting, a proactive defense method that achieves adequate protection under unknown conditions through a triple mechanism to address this challenge. Specifically, a multi-level deep feature extractor is presented to obtain intricate features during the diffusion denoising process to improve protective effectiveness. We design multi-scale semantic-preserving data augmentation to enhance the transferability of adversarial perturbations across unknown conditions by multi-scale transformations while preserving semantic integrity. In addition, we propose a selection-based distribution deviation optimization strategy to improve the protection of adversarial perturbation against manipulation under diverse random seeds. Extensive experiments indicate the proactive defensive performance of Anti-Inpainting against diffusion-based inpainters guided by unknown conditions in InpaintGuardBench and CelebA-HQ. At the same time, we also demonstrate the proposed approach's robustness under various image purification methods and its transferability across different versions of diffusion models.
Abstract (translated)
随着基于扩散的恶意图像篡改变得越来越普遍，已经开发出多种主动防御方法来保护图像免受未经授权的修改。然而，大多数主动防御方法仅能在已知条件下保护图像不受操纵，并且无法防止由恶意用户定制篡改条件所导致的图像被操控。为了解决这一问题，我们提出了Anti-Inpainting，这是一种通过三重机制实现未知条件下充分保护的主动防御方法。 具体来说，我们提出了一种多级深度特征提取器，在扩散去噪过程中获取复杂的特征以提高防护效果。此外，我们设计了多层次语义保持数据增强技术，通过多种尺度变换增强了对抗性扰动在未知条件下的迁移能力，并同时保证了语义的完整性。另外，我们还提出了一种基于选择的分布偏差优化策略，以提升对不同随机种子条件下对抗性扰动防护的效果。 广泛的实验表明，Anti-Inpainting 在 InpaintGuardBench 和 CelebA-HQ 数据集上针对由未知条件引导的扩散式修复程序展示了积极防御性能。同时，我们还证明了所提出的方法在各种图像净化方法下具有鲁棒性，并且能够在不同的扩散模型版本之间实现迁移能力。 此研究工作对于增强基于人工智能技术的安全防护有着重要的贡献和应用价值。
URL
https://arxiv.org/abs/2505.13023
PDF
