Inpainting
-
AUTHENTICATION: Identifying Rare Failure Modes in Autonomous Vehicle Perception Systems using Adversarially Guided Diffusion Models
2025-04-24 01:31:13Mohammad Zarei, Melanie A Jutras, Eliana Evans, Mike Tan, Omid AaramoonAbstract
Autonomous Vehicles (AVs) rely on artificial intelligence (AI) to accurately detect objects and interpret their surroundings. However, even when trained using millions of miles of real-world data, AVs are often unable to detect rare failure modes (RFMs). The problem of RFMs is commonly referred to as the "long-tail challenge", due to the distribution of data including many instances that are very rarely seen. In this paper, we present a novel approach that utilizes advanced generative and explainable AI techniques to aid in understanding RFMs. Our methods can be used to enhance the robustness and reliability of AVs when combined with both downstream model training and testing. We extract segmentation masks for objects of interest (e.g., cars) and invert them to create environmental masks. These masks, combined with carefully crafted text prompts, are fed into a custom diffusion model. We leverage the Stable Diffusion inpainting model guided by adversarial noise optimization to generate images containing diverse environments designed to evade object detection models and expose vulnerabilities in AI systems. Finally, we produce natural language descriptions of the generated RFMs that can guide developers and policymakers to improve the safety and reliability of AV systems.
Abstract (translated)
自主车辆（AVs）依赖于人工智能（AI）来准确检测物体并解释周围环境。然而，即使经过数百万英里的实际道路数据训练后，自动驾驶汽车仍然经常无法识别罕见故障模式（RFMs）。由于数据分布中包含许多极少出现的实例，这个问题被称为“长尾挑战”。在本文中，我们提出了一种新颖的方法，该方法利用先进的生成式和可解释的人工智能技术来帮助理解RFMs。我们的方法可以与下游模型训练和测试相结合，增强AVs的鲁棒性和可靠性。 我们提取感兴趣物体（例如汽车）的分割掩码，并将其反转以创建环境掩码。然后，我们将这些掩码结合精心设计的文本提示输入自定义扩散模型中。我们利用由对抗性噪声优化引导的Stable Diffusion修复模型生成包含多样环境的图像，这些环境旨在避开目标检测模型并揭示AI系统的漏洞。 最后，我们为生成的RFMs提供自然语言描述，以指导开发人员和政策制定者提高AV系统的安全性和可靠性。
URL
https://arxiv.org/abs/2504.17179
PDF
-
Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detection
2025-04-23 19:52:47Jens Petersen, Davide Abati, Amirhossein Habibian, Auke WiggersAbstract
Generative image models are increasingly being used for training data augmentation in vision tasks. In the context of automotive object detection, methods usually focus on producing augmented frames that look as realistic as possible, for example by replacing real objects with generated ones. Others try to maximize the diversity of augmented frames, for example by pasting lots of generated objects onto existing backgrounds. Both perspectives pay little attention to the locations of objects in the scene. Frame layouts are either reused with little or no modification, or they are random and disregard realism entirely. In this work, we argue that optimal data augmentation should also include realistic augmentation of layouts. We introduce a scene-aware probabilistic location model that predicts where new objects can realistically be placed in an existing scene. By then inpainting objects in these locations with a generative model, we obtain much stronger augmentation performance than existing approaches. We set a new state of the art for generative data augmentation on two automotive object detection tasks, achieving up to $2.8\times$ higher gains than the best competing approach ($+1.4$ vs. $+0.5$ mAP boost). We also demonstrate significant improvements for instance segmentation.
Abstract (translated)
生成图像模型在视觉任务的数据增强中应用日益增多。特别是在汽车目标检测领域，通常的方法主要集中在产生尽可能逼真的数据增强帧上，比如用生成的物体替换真实的物体。另一些方法则试图最大化增强帧的多样性，例如在现有背景上粘贴大量生成的物体。然而，这两种方法都很少关注场景中物体的位置。现有的布局要么是简单重复使用而几乎不作修改，要么就是随机放置且完全不顾及现实性。 本文提出了一种观点：最优的数据增强应包括对布局的真实感增强。我们引入了一个场景感知的概率位置模型，该模型可以预测在现有场景中新的物体能被合理放置的位置。通过在这个位置上用生成模型进行图像修复（inpainting），我们得到了比现有方法更强的数据增强效果。 我们在两个汽车目标检测任务上的生成数据增强方面设立了新的技术标准，并取得了高达2.8倍于最佳竞争方法的增益（+1.4 vs. +0.5 mAP提升）。此外，我们也展示了实例分割方面的显著改进。
URL
https://arxiv.org/abs/2504.17076
PDF
-
Beyond Anonymization: Object Scrubbing for Privacy-Preserving 2D and 3D Vision Tasks
2025-04-23 09:33:10Murat Bilgehan Ertan, Ronak Sahu, Phuong Ha Nguyen, Kaleel Mahmood, Marten van DijkAbstract
We introduce ROAR (Robust Object Removal and Re-annotation), a scalable framework for privacy-preserving dataset obfuscation that eliminates sensitive objects instead of modifying them. Our method integrates instance segmentation with generative inpainting to remove identifiable entities while preserving scene integrity. Extensive evaluations on 2D COCO-based object detection show that ROAR achieves 87.5% of the baseline detection average precision (AP), whereas image dropping achieves only 74.2% of the baseline AP, highlighting the advantage of scrubbing in preserving dataset utility. The degradation is even more severe for small objects due to occlusion and loss of fine-grained details. Furthermore, in NeRF-based 3D reconstruction, our method incurs a PSNR loss of at most 1.66 dB while maintaining SSIM and improving LPIPS, demonstrating superior perceptual quality. Our findings establish object removal as an effective privacy framework, achieving strong privacy guarantees with minimal performance trade-offs. The results highlight key challenges in generative inpainting, occlusion-robust segmentation, and task-specific scrubbing, setting the foundation for future advancements in privacy-preserving vision systems.
Abstract (translated)
我们介绍了ROAR（Robust Object Removal and Re-annotation），这是一个用于隐私保护的数据集模糊处理的可扩展框架，它通过消除敏感对象而不是修改它们来实现这一目标。我们的方法结合了实例分割和生成式填补技术，以去除可识别实体的同时保持场景完整性。在基于2D COCO的对象检测上的广泛评估表明，ROAR实现了基准检测平均精度（AP）的87.5%，相比之下，图像删除仅达到基准AP的74.2%，突显了擦除方法在保留数据集效用方面的优势。对于小对象来说，由于遮挡和细微特征丢失的问题，这种退化现象更为严重。 此外，在基于NeRF的3D重建中，我们的方法最多导致1.66 dB的PSNR损失，同时保持SSIM并提高LPIPS，展示了卓越的感知质量。我们的研究成果确立了物体移除作为有效的隐私框架的地位，实现了强有力的隐私保证，并在性能妥协方面达到最小化。结果突显了生成式填补、抗遮挡分割和任务特定擦除的关键挑战，为未来隐私保护视觉系统的进步奠定了基础。
URL
https://arxiv.org/abs/2504.16557
PDF
-
Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning
2025-04-23 06:21:11Mingxuan Cui, Qing Guo, Yuyi Wang, Hongkai Yu, Di Lin, Qin Zou, Ming-Ming Cheng, Xi LiAbstract
3D Gaussian Splatting (3DGS) has emerged as a powerful and efficient 3D representation for novel view synthesis. This paper extends 3DGS capabilities to inpainting, where masked objects in a scene are replaced with new contents that blend seamlessly with the surroundings. Unlike 2D image inpainting, 3D Gaussian inpainting (3DGI) is challenging in effectively leveraging complementary visual and semantic cues from multiple input views, as occluded areas in one view may be visible in others. To address this, we propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI in utilizing complementary visual cues. We also employ uncertainties to learn a semantic concept of scene without the masked object and use a diffusion model to fill masked objects in input images based on the learned concept. Finally, we build a novel 3DGI framework, VISTA, by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning. VISTA generates high-quality 3DGS models capable of synthesizing artifact-free and naturally inpainted novel views. Furthermore, our approach extends to handling dynamic distractors arising from temporal object changes, enhancing its versatility in diverse scene reconstruction scenarios. We demonstrate the superior performance of our method over state-of-the-art techniques using two challenging datasets: the SPIn-NeRF dataset, featuring 10 diverse static 3D inpainting scenes, and an underwater 3D inpainting dataset derived from UTB180, including fast-moving fish as inpainting targets.
Abstract (translated)
3D高斯点云（3D Gaussian Splatting，3DGS）作为一种强大的、高效的三维表示方法，在新颖视图合成中崭露头角。本文将3DGS的能力扩展到图像修复领域，即替换场景中被遮挡对象为与周围环境无缝融合的新内容。不同于2D图像修复，3D高斯点云修复（3DGI）在有效利用来自多个输入视角的互补视觉和语义线索方面更具挑战性，因为在一个视角下被遮挡的区域可能在其他视图中可见。为此，我们提出了一种方法来测量不同输入视角之间三维点的可视不确定性，并使用这些信息指导3DGI充分利用互补的视觉提示。 同时，利用这种不确定性的特性，我们还开发了一个学习场景概念（去除遮罩物体后）的方法，并通过扩散模型根据所学的概念填充输入图像中的被遮掩对象。最后，我们构建了一个新的3DGI框架VISTA，该框架将基于可视不确定性引导的3DGI与场景概念化学习相结合。 VISTA能够生成高质量的3DGS模型，这些模型可以合成无伪影且自然修复的新视图。此外，我们的方法还扩展到了处理动态干扰物的问题，这些问题来源于物体随时间的变化，这增强了其在各种场景重建情况下的适应性。我们通过两个具有挑战性的数据集验证了该方法相对于现有技术的优越性能：一个是在10个多样化静态3D修复场景中进行测试的SPIn-NeRF数据集，另一个是从UTB180数据集中衍生出的水下3D修复数据集，其中包括快速移动的鱼作为修复目标。
URL
https://arxiv.org/abs/2504.17815
PDF
-
DiTPainter: Efficient Video Inpainting with Diffusion Transformers
2025-04-22 07:36:45Xian Wu, Chang LiuAbstract
Many existing video inpainting algorithms utilize optical flows to construct the corresponding maps and then propagate pixels from adjacent frames to missing areas by mapping. Despite the effectiveness of the propagation mechanism, they might encounter blurry and inconsistencies when dealing with inaccurate optical flows or large masks. Recently, Diffusion Transformer (DiT) has emerged as a revolutionary technique for video generation tasks. However, pretrained DiT models for video generation all contain a large amount of parameters, which makes it very time consuming to apply to video inpainting tasks. In this paper, we present DiTPainter, an end-to-end video inpainting model based on Diffusion Transformer (DiT). DiTPainter uses an efficient transformer network designed for video inpainting, which is trained from scratch instead of initializing from any large pretrained models. DiTPainter can address videos with arbitrary lengths and can be applied to video decaptioning and video completion tasks with an acceptable time cost. Experiments show that DiTPainter outperforms existing video inpainting algorithms with higher quality and better spatial-temporal consistency.
Abstract (translated)
许多现有的视频修复算法利用光流来构建相应的映射，然后通过映射将相邻帧的像素传播到缺失区域。虽然这种传播机制非常有效，但在处理不准确的光流或大面积遮罩时可能会导致模糊和不一致的问题。最近，扩散变换器（Diffusion Transformer, DiT）作为视频生成任务中的革命性技术崭露头角。然而，用于视频生成的预训练DiT模型参数量巨大，这使得它们在应用于视频修复任务时非常耗时。 在这篇论文中，我们介绍了DiTPainter，这是一个基于扩散变换器（Diffusion Transformer, DiT）的端到端视频修复模型。DiTPainter使用了一种专门设计用于视频修复的有效Transformer网络，并且是从头开始训练的，而不是从任何大型预训练模型初始化的。DiTPainter可以处理任意长度的视频，并能以可接受的时间成本应用于视频去遮挡和视频完成任务。实验表明，DiTPainter在质量和时空一致性方面优于现有的视频修复算法。
URL
https://arxiv.org/abs/2504.15661
PDF
-
MirrorVerse: Pushing Diffusion Models to Realistically Reflect the World
2025-04-21 19:01:02Ankit Dhiman, Manan Shah, R Venkatesh BabuAbstract
Diffusion models have become central to various image editing tasks, yet they often fail to fully adhere to physical laws, particularly with effects like shadows, reflections, and occlusions. In this work, we address the challenge of generating photorealistic mirror reflections using diffusion-based generative models. Despite extensive training data, existing diffusion models frequently overlook the nuanced details crucial to authentic mirror reflections. Recent approaches have attempted to resolve this by creating synhetic datasets and framing reflection generation as an inpainting task; however, they struggle to generalize across different object orientations and positions relative to the mirror. Our method overcomes these limitations by introducing key augmentations into the synthetic data pipeline: (1) random object positioning, (2) randomized rotations, and (3) grounding of objects, significantly enhancing generalization across poses and placements. To further address spatial relationships and occlusions in scenes with multiple objects, we implement a strategy to pair objects during dataset generation, resulting in a dataset robust enough to handle these complex scenarios. Achieving generalization to real-world scenes remains a challenge, so we introduce a three-stage training curriculum to develop the MirrorFusion 2.0 model to improve real-world performance. We provide extensive qualitative and quantitative evaluations to support our approach. The project page is available at: this https URL.
Abstract (translated)
扩散模型已经成为各种图像编辑任务的核心，但它们往往未能完全遵循物理定律，尤其是在阴影、反射和遮挡等效果上。在这项工作中，我们解决了使用基于扩散的生成模型生成逼真的镜面反射的问题。尽管训练数据量很大，现有的扩散模型经常会忽略生成真实镜像反射所必需的细微细节。最近的方法试图通过创建合成数据集并将反射生成视为一个像素填充任务来解决这个问题；然而，它们在处理不同对象相对于镜子的位置和朝向时难以泛化。 我们的方法通过引入三个关键增强到合成数据管道中来克服这些限制：（1）随机物体定位，（2）随机旋转以及（3）将物体固定于地面，从而显著增强了跨姿态和位置的泛化能力。为了进一步解决多对象场景中的空间关系和遮挡问题，我们在数据集生成期间实施了一种配对对象的方法，最终形成一个能够处理这些复杂情况的数据集。 实现真实世界场景的泛化仍然是一个挑战，因此我们引入了一个三阶段训练课程来发展MirrorFusion 2.0模型，以提高其在现实环境中的表现。为了支持我们的方法，我们提供了详尽的质量和数量评价结果。项目主页位于：此链接（请根据实际情况替换为实际URL）。
URL
https://arxiv.org/abs/2504.15397
PDF
-
SMPL-GPTexture: Dual-View 3D Human Texture Estimation using Text-to-Image Generation Models
2025-04-17 23:28:38Mingxiao Tu, Shuchang Ye, Hoijoon Jung, Jinman KimAbstract
Generating high-quality, photorealistic textures for 3D human avatars remains a fundamental yet challenging task in computer vision and multimedia field. However, real paired front and back images of human subjects are rarely available with privacy, ethical and cost of acquisition, which restricts scalability of the data. Additionally, learning priors from image inputs using deep generative models, such as GANs or diffusion models, to infer unseen regions such as the human back often leads to artifacts, structural inconsistencies, or loss of fine-grained detail. To address these issues, we present SMPL-GPTexture (skinned multi-person linear model - general purpose Texture), a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model to produce paired high-resolution front and back images of a human subject as the starting point for texture estimation. Using the generated paired dual-view images, we first employ a human mesh recovery model to obtain a robust 2D-to-3D SMPL alignment between image pixels and the 3D model's UV coordinates for each views. Second, we use an inverted rasterization technique that explicitly projects the observed colour from the input images into the UV space, thereby producing accurate, complete texture maps. Finally, we apply a diffusion-based inpainting module to fill in the missing regions, and the fusion mechanism then combines these results into a unified full texture map. Extensive experiments shows that our SMPL-GPTexture can generate high resolution texture aligned with user's prompts.
Abstract (translated)
生成高质量、逼真的3D人体头像纹理对于计算机视觉和多媒体领域而言仍然是一个基础而具有挑战性的任务。然而，由于隐私、伦理以及获取成本的限制，真实的人体前后配对图像非常难以获得，这在很大程度上限制了数据集的规模。此外，使用深度生成模型（如GAN或扩散模型）从图像输入中学习先验知识以推断未见过的区域（例如人体背面），往往会导致伪影、结构不一致或细节损失等问题。 为了解决这些问题，我们提出了SMPL-GPTexture（具有通用纹理的多个人体线性模型），这是一种创新的工作流程。该工作流程使用自然语言提示作为输入，并利用最先进的文本到图像生成模型来创建配对的高分辨率前后人体视图图像，以用作纹理估计的基础。通过使用生成的双视角图像，我们首先采用一个人体网格恢复模型，以获得在每个视图中将图像像素与3D模型UV坐标进行可靠2D至3D SMPL对齐的方法。其次，我们利用一种逆向光栅化技术，该技术明确地将输入图像中的观察颜色投影到UV空间中，从而生成准确、完整的纹理贴图。最后，我们采用基于扩散的填补模块来填充缺失区域，并使用融合机制将这些结果结合成一个统一的整体纹理贴图。 广泛的实验表明，我们的SMPL-GPTexture能够根据用户的提示生成高质量并与其描述相匹配的高分辨率纹理。
URL
https://arxiv.org/abs/2504.13378
PDF
-
MathPhys-Guided Coarse-to-Fine Anomaly Synthesis with SQE-Driven Bi-Level Optimization for Anomaly Detection
2025-04-17 14:22:27Long Qian, Bingke Zhu, Yingying Chen, Ming Tang, Jinqiao WangAbstract
Anomaly detection is a crucial task in computer vision, yet collecting real-world defect images is inherently difficult due to the rarity and unpredictability of anomalies. Consequently, researchers have turned to synthetic methods for training data augmentation. However, existing synthetic strategies (e.g., naive cut-and-paste or inpainting) overlook the underlying physical causes of defects, leading to inconsistent, low-fidelity anomalies that hamper model generalization to real-world complexities. In this thesis, we introduced a novel pipeline that generates synthetic anomalies through Math-Physics model guidance, refines them via a Coarse-to-Fine approach and employs a bi-level optimization strategy with a Synthesis Quality Estimator(SQE). By incorporating physical modeling of cracks, corrosion, and deformation, our method produces realistic defect masks, which are subsequently enhanced in two phases. The first stage (npcF) enforces a PDE-based consistency to achieve a globally coherent anomaly structure, while the second stage (npcF++) further improves local fidelity using wavelet transforms and boundary synergy blocks. Additionally, we leverage SQE-driven weighting, ensuring that high-quality synthetic samples receive greater emphasis during training. To validate our approach, we conducted comprehensive experiments on three widely adopted industrial anomaly detection benchmarks: MVTec AD, VisA, and BTAD. Across these datasets, the proposed pipeline achieves state-of-the-art (SOTA) results in both image-AUROC and pixel-AUROC, confirming the effectiveness of our MaPhC2F and BiSQAD.
Abstract (translated)
异常检测是计算机视觉中的一个重要任务，但由于缺陷图像在现实世界中极为稀有且难以预测，收集这些图像非常困难。因此，研究人员转向合成方法来增强训练数据。然而，现有的合成策略（如简单的剪切粘贴或修复）忽略了缺陷的物理成因，导致生成的异常不一致且质量低劣，限制了模型对现实复杂情况的泛化能力。在本论文中，我们引入了一种新型管道，该管道通过数学-物理模型指导生成合成异常，并采用粗到细的方法进行优化，同时利用双层级优化策略和合成质量评估器（SQE）。我们的方法通过对裂缝、腐蚀和变形等缺陷进行物理建模来产生现实的缺陷掩膜，在两个阶段进一步增强它们。第一阶段(npcF)通过基于PDE的一致性实现全局连贯的异常结构，而第二阶段(npcF++)则利用小波变换和边界协同块进一步提升局部细节的真实性。 此外，我们还采用了SQE驱动的加权机制，确保在训练过程中对高质量合成样本给予更多关注。为了验证我们的方法的有效性，在三个广泛采用的工业异常检测基准数据集（MVTec AD、VisA 和 BTAD）上进行了全面实验。在这些数据集中，所提出的管道在图像AUROC和像素AUROC方面均达到了最先进的（SOTA）结果，这证实了我们MaPhC2F和BiSQAD的有效性。
URL
https://arxiv.org/abs/2504.12970
PDF
-
High-Fidelity Image Inpainting with Multimodal Guided GAN Inversion
2025-04-17 10:58:45Libo Zhang, Yongsheng Yu, Jiali Yao, Heng FanAbstract
Generative Adversarial Network (GAN) inversion have demonstrated excellent performance in image inpainting that aims to restore lost or damaged image texture using its unmasked content. Previous GAN inversion-based methods usually utilize well-trained GAN models as effective priors to generate the realistic regions for missing holes. Despite excellence, they ignore a hard constraint that the unmasked regions in the input and the output should be the same, resulting in a gap between GAN inversion and image inpainting and thus degrading the performance. Besides, existing GAN inversion approaches often consider a single modality of the input image, neglecting other auxiliary cues in images for improvements. Addressing these problems, we propose a novel GAN inversion approach, dubbed MMInvertFill, for image inpainting. MMInvertFill contains primarily a multimodal guided encoder with a pre-modulation and a GAN generator with F&W+ latent space. Specifically, the multimodal encoder aims to enhance the multi-scale structures with additional semantic segmentation edge texture modalities through a gated mask-aware attention module. Afterwards, a pre-modulation is presented to encode these structures into style vectors. To mitigate issues of conspicuous color discrepancy and semantic inconsistency, we introduce the F&W+ latent space to bridge the gap between GAN inversion and image inpainting. Furthermore, in order to reconstruct faithful and photorealistic images, we devise a simple yet effective Soft-update Mean Latent module to capture more diversified in-domain patterns for generating high-fidelity textures for massive corruptions. In our extensive experiments on six challenging datasets, we show that our MMInvertFill qualitatively and quantitatively outperforms other state-of-the-arts and it supports the completion of out-of-domain images effectively.
Abstract (translated)
生成对抗网络（GAN）逆向技术在图像修复领域中展示了卓越的性能，其目标是利用未遮挡的内容恢复丢失或损坏的纹理。之前的基于 GAN 逆向的方法通常会使用经过良好训练的 GAN 模型作为有效的先验条件来生成缺失区域的真实感部分。尽管这些方法表现优秀，但它们忽略了输入图像和输出图像中未遮挡区域应当保持一致这一严格的约束条件，这导致了 GAN 逆向与图像修复之间的差距，并因此降低了性能。此外，现有的 GAN 逆向方法通常仅考虑输入图像的单一模式，忽视了其他在改进方面有帮助的辅助线索。 为了应对这些挑战，我们提出了一种新颖的 GAN 逆向方法，称为 MMInvertFill，用于图像修复。MMInvertFill 主要包含一个多模态引导编码器和一个 F&W+ 潜空间中的 GAN 发生器。具体来说，多模态编码器旨在通过门控掩码感知注意模块增强多层次结构，并且引入了额外的语义分割边缘纹理模式。随后，我们提出预调制技术将这些结构编码为样式向量。为了缓解明显的颜色差异和语义不一致的问题，我们引进 F&W+ 潜空间来弥合 GAN 逆向与图像修复之间的差距。 进一步地，为了重构忠实且逼真的图像，我们设计了一个简单而有效的软更新均值潜空间模块，以捕捉更多的域内模式，并为大量损坏生成高质量的纹理。在六个具有挑战性的数据集上的广泛实验中，我们的 MMInvertFill 从定性和定量上都超越了当前最佳方法，并支持跨域图像的有效完成任务。
URL
https://arxiv.org/abs/2504.12844
PDF
-
SmartFreeEdit: Mask-Free Spatial-Aware Image Editing with Complex Instruction Understanding
2025-04-17 07:17:49Qianqian Sun, Jixiang Luo, Dell Zhang, Xuelong LiAbstract
Recent advancements in image editing have utilized large-scale multimodal models to enable intuitive, natural instruction-driven interactions. However, conventional methods still face significant challenges, particularly in spatial reasoning, precise region segmentation, and maintaining semantic consistency, especially in complex scenes. To overcome these challenges, we introduce SmartFreeEdit, a novel end-to-end framework that integrates a multimodal large language model (MLLM) with a hypergraph-enhanced inpainting architecture, enabling precise, mask-free image editing guided exclusively by natural language instructions. The key innovations of SmartFreeEdit include:(1)the introduction of region aware tokens and a mask embedding paradigm that enhance the spatial understanding of complex scenes;(2) a reasoning segmentation pipeline designed to optimize the generation of editing masks based on natural language instructions;and (3) a hypergraph-augmented inpainting module that ensures the preservation of both structural integrity and semantic coherence during complex edits, overcoming the limitations of local-based image generation. Extensive experiments on the Reason-Edit benchmark demonstrate that SmartFreeEdit surpasses current state-of-the-art methods across multiple evaluation metrics, including segmentation accuracy, instruction adherence, and visual quality preservation, while addressing the issue of local information focus and improving global consistency in the edited image. Our project will be available at this https URL.
Abstract (translated)
近期的图像编辑技术利用大规模多模态模型实现了直观且自然的语言驱动交互。然而，传统方法在空间推理、精确区域分割以及保持语义一致性方面仍面临重大挑战，尤其是在复杂场景中更为明显。为了解决这些问题，我们引入了SmartFreeEdit——一种结合了超图增强的图像修复架构和多模态大型语言模型（MLLM）的全新端到端框架，它支持仅通过自然语言指令进行无遮罩、精确的图像编辑。SmartFreeEdit的主要创新点包括： 1. 引入区域感知令牌与掩码嵌入范式，以增强对复杂场景的空间理解； 2. 设计了一种推理分割流水线，旨在根据自然语言指令优化生成编辑掩码的过程； 3. 采用超图增强的图像修复模块，在进行复杂编辑时确保结构完整性和语义连贯性的保持，从而克服基于局部信息生成图像的局限性。 在Reason-Edit基准测试中的广泛实验表明，SmartFreeEdit在多个评估指标（包括分割准确性、指令遵循度和视觉质量保留）上均超越了当前最先进的方法，并解决了局部信息关注不足及改进编辑后图像全局一致性的难题。有关我们项目的更多信息可在此URL访问：[https://此链接待补充](https://此链接待补充)。
URL
https://arxiv.org/abs/2504.12704
PDF
-
Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpainting
2025-04-15 11:38:14Jiaxin Huang, Sheng Miao, BangBnag Yang, Yuewen Ma, Yiyi LiaoAbstract
Reconstructing 4D dynamic scenes from casually captured monocular videos is valuable but highly challenging, as each timestamp is observed from a single viewpoint. We introduce Vivid4D, a novel approach that enhances 4D monocular video synthesis by augmenting observation views - synthesizing multi-view videos from a monocular input. Unlike existing methods that either solely leverage geometric priors for supervision or use generative priors while overlooking geometry, we integrate both. This reformulates view augmentation as a video inpainting task, where observed views are warped into new viewpoints based on monocular depth priors. To achieve this, we train a video inpainting model on unposed web videos with synthetically generated masks that mimic warping occlusions, ensuring spatially and temporally consistent completion of missing regions. To further mitigate inaccuracies in monocular depth priors, we introduce an iterative view augmentation strategy and a robust reconstruction loss. Experiments demonstrate that our method effectively improves monocular 4D scene reconstruction and completion.
Abstract (translated)
从随意捕捉的单目视频中重建动态的四维场景具有很高的价值，但同时也极具挑战性，因为每个时间戳只能从单一视角观察。为此，我们提出了Vivid4D这一创新方法，它通过增强观测视角来提升四维单目视频合成的质量——即从单目输入生成多视点视频。与现有的要么单纯依赖几何先验进行监督、要么利用生成式先验而忽视了几何特性的方法不同，我们的方法整合了这两者。 我们将视角增强重新定义为一个视频修复任务，在此过程中，根据单目深度先验将已观测到的视角扭曲至新的视点。为了实现这一点，我们训练了一个视频修复模型，并在未经过预设处理的网络视频上进行训练，同时生成模仿视差遮挡效果的合成掩码，以确保缺失区域的空间和时间一致性得到妥善完成。 为进一步减少单目深度先验中的不准确性，我们引入了一种迭代视角增强策略及一种鲁棒重建损失函数。实验结果表明，我们的方法能够有效提升基于单目的四维场景重建与填补的效果。
URL
https://arxiv.org/abs/2504.11092
PDF
-
Cross-Frequency Implicit Neural Representation with Self-Evolving Parameters
2025-04-15 07:14:35Chang Yu, Yisi Luo, Kai Ye, Xile Zhao, Deyu MengAbstract
Implicit neural representation (INR) has emerged as a powerful paradigm for visual data representation. However, classical INR methods represent data in the original space mixed with different frequency components, and several feature encoding parameters (e.g., the frequency parameter $\omega$ or the rank $R$) need manual configurations. In this work, we propose a self-evolving cross-frequency INR using the Haar wavelet transform (termed CF-INR), which decouples data into four frequency components and employs INRs in the wavelet space. CF-INR allows the characterization of different frequency components separately, thus enabling higher accuracy for data representation. To more precisely characterize cross-frequency components, we propose a cross-frequency tensor decomposition paradigm for CF-INR with self-evolving parameters, which automatically updates the rank parameter $R$ and the frequency parameter $\omega$ for each frequency component through self-evolving optimization. This self-evolution paradigm eliminates the laborious manual tuning of these parameters, and learns a customized cross-frequency feature encoding configuration for each dataset. We evaluate CF-INR on a variety of visual data representation and recovery tasks, including image regression, inpainting, denoising, and cloud removal. Extensive experiments demonstrate that CF-INR outperforms state-of-the-art methods in each case.
Abstract (translated)
隐式神经表示（INR）作为一种强大的视觉数据表达范式已经出现。然而，传统的INR方法在原始空间中用不同的频率成分混合表示数据，并且需要手动配置若干特征编码参数（如频率参数$\omega$或秩$R$）。在这项工作中，我们提出了一种使用Haar小波变换的自进化跨频带隐式神经表示（CF-INR），该方法将数据分解为四个频率分量并在小波空间中应用INRs。通过这种方式，CF-INR能够分别表征不同的频率成分，从而提高数据表示的准确性。 为了更精确地描述跨频带组件，我们提出了一个适用于CF-INR的自进化跨频带张量分解范式，该方法自动更新每个频率分量的秩参数$R$和频率参数$\omega$。通过自我进化的优化过程，这种自进化范例消除了这些参数的手动调整需求，并为每种数据集学习定制化的跨频带特征编码配置。 我们在多种视觉数据表示与恢复任务上评估了CF-INR的表现，包括图像回归、修复（inpainting）、去噪和去除云层。广泛的实验结果表明，在所有情况下，CF-INR均优于当前最佳方法。
URL
https://arxiv.org/abs/2504.10929
PDF
-
Investigating the Role of Bilateral Symmetry for Inpainting Brain MRI
2025-04-14 09:41:47Sergey Kuznetsov, Sanduni Pinnawala, Peter A. Wijeratne, Ivor J. A. SimpsonAbstract
Inpainting has recently emerged as a valuable and interesting technology to employ in the analysis of medical imaging data, in particular brain MRI. A wide variety of methodologies for inpainting MRI have been proposed and demonstrated on tasks including anomaly detection. In this work we investigate the statistical relationship between inpainted brain structures and the amount of subject-specific conditioning information, i.e. the other areas of the image that are masked. In particular, we analyse the distribution of inpainting results when masking additional regions of the image, specifically the contra-lateral structure. This allows us to elucidate where in the brain the model is drawing information from, and in particular, what is the importance of hemispherical symmetry? Our experiments interrogate a diffusion inpainting model through analysing the inpainting of subcortical brain structures based on intensity and estimated area change. We demonstrate that some structures show a strong influence of symmetry in the conditioning of the inpainting process.
Abstract (translated)
最近，图像修复（Inpainting）技术在医学影像数据分析中，尤其是脑部MRI分析方面展现出了其价值和趣味性。许多用于MRI图像修复的方法已经被提出，并且在包括异常检测在内的任务中得到了验证。在这项工作中，我们调查了修复后的脑结构与其特定受试者的条件信息之间的统计关系，即那些被遮蔽的其他区域。具体来说，我们分析了当屏蔽额外区域（特别是对侧结构）时图像修复结果的分布情况。这使我们能够揭示模型从大脑何处获取信息，并特别指出半球对称性的重要性是什么。 我们的实验通过分析基于强度和估计面积变化进行弥散MRI图像修复的过程来探究一个图像修复模型。结果显示，某些结构在图像修复过程中的条件化过程中表现出强烈的对称性影响。
URL
https://arxiv.org/abs/2504.10039
PDF
-
GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting
2025-04-14 09:04:01Junlin Hao, Peiheng Wang, Haoyang Wang, Xinggong Zhang, Zongming GuoAbstract
Single-image 3D scene reconstruction presents significant challenges due to its inherently ill-posed nature and limited input constraints. Recent advances have explored two promising directions: multiview generative models that train on 3D consistent datasets but struggle with out-of-distribution generalization, and 3D scene inpainting and completion frameworks that suffer from cross-view inconsistency and suboptimal error handling, as they depend exclusively on depth data or 3D smoothness, which ultimately degrades output quality and computational performance. Building upon these approaches, we present GaussVideoDreamer, which advances generative multimedia approaches by bridging the gap between image, video, and 3D generation, integrating their strengths through two key innovations: (1) A progressive video inpainting strategy that harnesses temporal coherence for improved multiview consistency and faster convergence. (2) A 3D Gaussian Splatting consistency mask to guide the video diffusion with 3D consistent multiview evidence. Our pipeline combines three core components: a geometry-aware initialization protocol, Inconsistency-Aware Gaussian Splatting, and a progressive video inpainting strategy. Experimental results demonstrate that our approach achieves 32% higher LLaVA-IQA scores and at least 2x speedup compared to existing methods while maintaining robust performance across diverse scenes.
Abstract (translated)
单张图像的三维场景重建面临巨大的挑战，由于其本质上是一个病态问题，并且输入信息有限。最近的研究主要探索了两个有前景的方向：多视图生成模型训练在3D一致的数据集上，但难以处理出分布数据的一般化；以及依赖于深度数据或三维平滑性的3D场景修复和补全框架，在跨视角一致性方面存在不足，并且误差处理欠佳，最终导致输出质量降低及计算性能下降。基于这些方法的进展，我们提出了GaussVideoDreamer，该模型通过连接图像、视频与三维生成之间的差距，利用两个关键创新来推动生成多媒体的方法：（1）渐进式视频修复策略，借助时间一致性实现多视图一致性的改进和更快收敛。（2）3D高斯点阵一致性掩模，为视频扩散过程提供基于3D一致性多视角证据的指导。我们的流程整合了三个核心组件：几何感知初始化协议、跨一致性意识高斯点阵以及渐进式视频修复策略。实验结果表明，相较于现有的方法，我们提出的方法在LLaVA-IQA评分上提高了32%，同时至少提升了两倍的速度，并保持了在各种场景中的稳定性能。
URL
https://arxiv.org/abs/2504.10001
PDF
-
Learning Object Focused Attention
2025-04-10 23:23:26Vivek Trivedy, Amani Almalki, Longin Jan LateckiAbstract
We propose an adaptation to the training of Vision Transformers (ViTs) that allows for an explicit modeling of objects during the attention computation. This is achieved by adding a new branch to selected attention layers that computes an auxiliary loss which we call the object-focused attention (OFA) loss. We restrict the attention to image patches that belong to the same object class, which allows ViTs to gain a better understanding of configural (or holistic) object shapes by focusing on intra-object patches instead of other patches such as those in the background. Our proposed inductive bias fits easily into the attention framework of transformers since it only adds an auxiliary loss over selected attention layers. Furthermore, our approach has no additional overhead during inference. We also experiment with multiscale masking to further improve the performance of our OFA model and give a path forward for self-supervised learning with our method. Our experimental results demonstrate that ViTs with OFA achieve better classification results than their base models, exhibit a stronger generalization ability to out-of-distribution (OOD) and adversarially corrupted images, and learn representations based on object shapes rather than spurious correlations via general textures. For our OOD setting, we generate a novel dataset using the COCO dataset and Stable Diffusion inpainting which we plan to share with the community.
Abstract (translated)
我们提出了一种对视觉变换器（ViT）训练的适应性调整，该调整允许在注意力计算过程中显式地对物体进行建模。通过向选定的关注层添加一个新的分支来实现这一点，这个新分支计算一个辅助损失，即所谓的对象聚焦关注（OFA）损失。我们将注意力限制在属于同一物体类别的图像补丁上，这使ViT能够更好地理解配置图（或整体）的物体形状，方法是将重点放在物体内部的补丁上，而不是背景中的其他补丁。我们提出的归纳偏差很容易融入到变换器的关注框架中，因为它仅向选定的关注层添加了一个辅助损失，并且在推理过程中没有额外的开销。此外，我们还通过多尺度掩码实验来进一步提高我们的OFA模型的表现，并为使用我们方法进行自监督学习提供了一条路径。实验证明了，带有OFA的ViT比其基础模型有更好的分类效果，更强地泛化到分布外（OOD）和对抗性损坏的图像上，并且基于物体形状而非偶然关联的一般纹理来学习表示。对于我们的OOD设置，我们使用COCO数据集和Stable Diffusion inpainting生成了一个新的数据集，计划与社区分享。
URL
https://arxiv.org/abs/2504.08166
PDF
-
A Meaningful Perturbation Metric for Evaluating Explainability Methods
2025-04-09 11:46:41Danielle Cohen, Hila Chefer, Lior WolfAbstract
Deep neural networks (DNNs) have demonstrated remarkable success, yet their wide adoption is often hindered by their opaque decision-making. To address this, attribution methods have been proposed to assign relevance values to each part of the input. However, different methods often produce entirely different relevance maps, necessitating the development of standardized metrics to evaluate them. Typically, such evaluation is performed through perturbation, wherein high- or low-relevance regions of the input image are manipulated to examine the change in prediction. In this work, we introduce a novel approach, which harnesses image generation models to perform targeted perturbation. Specifically, we focus on inpainting only the high-relevance pixels of an input image to modify the model's predictions while preserving image fidelity. This is in contrast to existing approaches, which often produce out-of-distribution modifications, leading to unreliable results. Through extensive experiments, we demonstrate the effectiveness of our approach in generating meaningful rankings across a wide range of models and attribution methods. Crucially, we establish that the ranking produced by our metric exhibits significantly higher correlation with human preferences compared to existing approaches, underscoring its potential for enhancing interpretability in DNNs.
Abstract (translated)
深度神经网络（DNN）展现了显著的成功，但其广泛采用往往受到其不透明决策过程的阻碍。为了解决这一问题，已经提出了归因方法来给输入的每一部分分配相关性值。然而，不同的方法常常产生完全不同的相关性图谱，这需要开发标准化的评估指标来进行评价。通常，这种评估是通过扰动进行的，即操纵输入图像中的高或低相关区域以观察预测的变化。 在本文中，我们引入了一种新颖的方法，利用图像生成模型执行定向扰动。具体来说，我们将仅对输入图像中的高相关像素进行修复（inpainting），同时修改模型的预测并保持图像的真实性。这与现有方法形成对比，后者通常会产生出分布的数据修改，导致结果不可靠。 通过广泛的实验，我们展示了该方法在广泛的不同模型和归因方法中生成有意义排名的有效性。尤其重要的是，我们证明了我们的度量所生成的排序与人类偏好之间的相关性显著高于现有的方法，强调了其增强DNN可解释性的潜力。
URL
https://arxiv.org/abs/2504.06800
PDF
-
FANeRV: Frequency Separation and Augmentation based Neural Representation for Video
2025-04-09 10:19:35Li Yu, Zhihui Li, Jimin Xiao, Moncef GabboujAbstract
Neural representations for video (NeRV) have gained considerable attention for their strong performance across various video tasks. However, existing NeRV methods often struggle to capture fine spatial details, resulting in vague reconstructions. In this paper, we present a Frequency Separation and Augmentation based Neural Representation for video (FANeRV), which addresses these limitations with its core Wavelet Frequency Upgrade this http URL block explicitly separates input frames into high and low-frequency components using discrete wavelet transform, followed by targeted enhancement using specialized modules. Finally, a specially designed gated network effectively fuses these frequency components for optimal reconstruction. Additionally, convolutional residual enhancement blocks are integrated into the later stages of the network to balance parameter distribution and improve the restoration of high-frequency details. Experimental results demonstrate that FANeRV significantly improves reconstruction performance and excels in multiple tasks, including video compression, inpainting, and interpolation, outperforming existing NeRV methods.
Abstract (translated)
基于神经表示的视频方法（NeRV）因其在各种视频任务中的出色表现而获得了广泛的关注。然而，现有的NeRV方法常常难以捕捉精细的空间细节，导致重建效果模糊不清。为此，在本文中我们提出了一种新的方法——基于频率分离和增强的神经视频表示（FANeRV），它通过核心的小波频谱升级模块解决了这些问题。 FANeRV的核心在于其采用了离散小波变换来将输入帧显式地分离为高频与低频成分，随后利用专门设计的模块对特定成分进行针对性提升。最终，一个特别设计的门控网络有效融合了这些频率组件以实现最优重建效果。此外，在网络后期阶段还集成了卷积残差增强块，这有助于平衡参数分布并进一步改善高频细节的恢复。 实验结果表明，FANeRV在视频压缩、修复和插值等多任务上显著提升了重建性能，并超越了现有的NeRV方法的表现。
URL
https://arxiv.org/abs/2504.06755
PDF
-
PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Rendering
2025-04-09 07:13:08Yifan Gao, Zihang Lin, Chuanbin Liu, Min Zhou, Tiezheng Ge, Bo Zheng, Hongtao XieAbstract
Product posters, which integrate subject, scene, and text, are crucial promotional tools for attracting customers. Creating such posters using modern image generation methods is valuable, while the main challenge lies in accurately rendering text, especially for complex writing systems like Chinese, which contains over 10,000 individual characters. In this work, we identify the key to precise text rendering as constructing a character-discriminative visual feature as a control signal. Based on this insight, we propose a robust character-wise representation as control and we develop TextRenderNet, which achieves a high text rendering accuracy of over 90%. Another challenge in poster generation is maintaining the fidelity of user-specific products. We address this by introducing SceneGenNet, an inpainting-based model, and propose subject fidelity feedback learning to further enhance fidelity. Based on TextRenderNet and SceneGenNet, we present PosterMaker, an end-to-end generation framework. To optimize PosterMaker efficiently, we implement a two-stage training strategy that decouples text rendering and background generation learning. Experimental results show that PosterMaker outperforms existing baselines by a remarkable margin, which demonstrates its effectiveness.
Abstract (translated)
产品海报集成了主题、场景和文本，是吸引顾客的关键促销工具。利用现代图像生成方法创建这样的海报非常有价值，然而主要挑战在于准确渲染文字，尤其是对于包含超过10,000个独特字符的复杂书写系统如中文来说更是如此。在这项工作中，我们确定了精确文字渲染的关键在于构建具有区分度的字符视觉特征作为控制信号。基于这一洞察，我们提出了一种稳健的逐字符表示形式作为控制，并开发了TextRenderNet，实现了超过90%的文字渲染准确率。 海报生成中的另一个挑战是保持用户特定产品的忠实性（保真度）。为了解决这个问题，我们引入了SceneGenNet，这是一个基于图像修复的方法模型，并提出了主题忠实反馈学习以进一步提升保真度。结合TextRenderNet和SceneGenNet，我们推出了PosterMaker，一个端到端的生成框架。 为了高效地优化PosterMaker，我们实施了一种两阶段训练策略，将文字渲染与背景生成的学习过程解耦。实验结果表明，PosterMaker显著优于现有的基准模型，这证明了其有效性。
URL
https://arxiv.org/abs/2504.06632
PDF
-
Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization
2025-04-07 16:12:05Zeqin Yu, Jiangqun Ni, Jian Zhang, Haoyi Deng, Yuzhen LinAbstract
Image forgery detection and localization (IFDL) is of vital importance as forged images can spread misinformation that poses potential threats to our daily lives. However, previous methods still struggled to effectively handle forged images processed with diverse forgery operations in real-world scenarios. In this paper, we propose a novel Reinforced Multi-teacher Knowledge Distillation (Re-MTKD) framework for the IFDL task, structured around an encoder-decoder \textbf{C}onvNeXt-\textbf{U}perNet along with \textbf{E}dge-Aware Module, named Cue-Net. First, three Cue-Net models are separately trained for the three main types of image forgeries, i.e., copy-move, splicing, and inpainting, which then serve as the multi-teacher models to train the target student model with Cue-Net through self-knowledge distillation. A Reinforced Dynamic Teacher Selection (Re-DTS) strategy is developed to dynamically assign weights to the involved teacher models, which facilitates specific knowledge transfer and enables the student model to effectively learn both the common and specific natures of diverse tampering traces. Extensive experiments demonstrate that, compared with other state-of-the-art methods, the proposed method achieves superior performance on several recently emerged datasets comprised of various kinds of image forgeries.
Abstract (translated)
图像伪造检测和定位（IFDL）至关重要，因为伪造的图片可以传播错误信息，对我们的日常生活造成潜在威胁。然而，先前的方法在处理真实场景中经过多种伪造操作处理的伪造图片时仍然存在困难。本文提出了一种新的强化多教师知识蒸馏（Re-MTKD）框架来解决IFDL任务，该框架围绕编码器-解码器**C**onvNeXt-**U**perNet以及**E**dge-Aware Module构建，命名为Cue-Net。 首先，分别训练三个Cue-Net模型以处理三种主要类型的图像伪造：复制粘贴、拼接和填充。这些模型作为多教师模型参与自我知识蒸馏过程来培训目标学生模型。此外，开发了一种强化动态教师选择（Re-DTS）策略，用于根据具体情况为涉及的教师模型分配权重，这有助于特定的知识转移，并使学生模型能够有效地学习各种篡改痕迹中普遍和特定的性质。 大量的实验表明，与现有的其他先进方法相比，所提出的方法在包含多种类型图像伪造的几个最近出现的数据集上取得了优越的性能。
URL
https://arxiv.org/abs/2504.05224
PDF
-
Bridging Knowledge Gap Between Image Inpainting and Large-Area Visible Watermark Removal
2025-04-07 02:37:14Yicheng Leng, Chaowei Fang, Junye Chen, Yixiang Fang, Sheng Li, Guanbin LiAbstract
Visible watermark removal which involves watermark cleaning and background content restoration is pivotal to evaluate the resilience of watermarks. Existing deep neural network (DNN)-based models still struggle with large-area watermarks and are overly dependent on the quality of watermark mask prediction. To overcome these challenges, we introduce a novel feature adapting framework that leverages the representation modeling capacity of a pre-trained image inpainting model. Our approach bridges the knowledge gap between image inpainting and watermark removal by fusing information of the residual background content beneath watermarks into the inpainting backbone model. We establish a dual-branch system to capture and embed features from the residual background content, which are merged into intermediate features of the inpainting backbone model via gated feature fusion modules. Moreover, for relieving the dependence on high-quality watermark masks, we introduce a new training paradigm by utilizing coarse watermark masks to guide the inference process. This contributes to a visible image removal model which is insensitive to the quality of watermark mask during testing. Extensive experiments on both a large-scale synthesized dataset and a real-world dataset demonstrate that our approach significantly outperforms existing state-of-the-art methods. The source code is available in the supplementary materials.
Abstract (translated)
可见水印去除技术，包括水印清洗和背景内容恢复，在评估水印的鲁棒性方面至关重要。现有的基于深度神经网络（DNN）的模型在处理大面积水印时仍然面临挑战，并且过度依赖于水印掩码预测的质量。为了解决这些问题，我们引入了一个新颖的功能适应框架，该框架利用了预训练图像修复模型的表示建模能力。我们的方法通过将残余背景内容的信息融入到图像修复主干模型中，填补了图像修复和水印去除之间的知识空白。我们建立了一个双分支系统来捕获并嵌入来自水印下的剩余背景内容的功能，并通过门控特征融合模块将其合并到图像修复主干模型的中间特性中。此外，为了减轻对高质量水印掩码的依赖性，我们引入了一种新的训练范式，利用粗略水印掩码来指导推理过程。这有助于创建一个在测试过程中对水印掩码质量不敏感的可见图片去除模型。广泛的实验结果表明，在大规模合成数据集和真实世界数据集上，我们的方法显著优于现有的最先进的方法。源代码可在补充材料中找到。
URL
https://arxiv.org/abs/2504.04687
PDF
