Inpainting
-
Training-free Geometric Image Editing on Diffusion Models
2025-07-31 07:36:00Hanshen Zhu, Zhen Zhu, Kaile Zhang, Yiming Gong, Yuliang Liu, Xiang BaiAbstract
We tackle the task of geometric image editing, where an object within an image is repositioned, reoriented, or reshaped while preserving overall scene coherence. Previous diffusion-based editing methods often attempt to handle all relevant subtasks in a single step, proving difficult when transformations become large or structurally complex. We address this by proposing a decoupled pipeline that separates object transformation, source region inpainting, and target region refinement. Both inpainting and refinement are implemented using a training-free diffusion approach, FreeFine. In experiments on our new GeoBench benchmark, which contains both 2D and 3D editing scenarios, FreeFine outperforms state-of-the-art alternatives in image fidelity, and edit precision, especially under demanding transformations. Code and benchmark are available at: this https URL
Abstract (translated)
我们解决了几何图像编辑任务，其中目标是在保持整体场景连贯性的同时，重新定位、旋转或改变图像中某个对象的位置和形状。以往基于扩散的编辑方法通常试图一次性处理所有相关子任务，在进行较大或结构复杂的转换时难度会增加。为了解决这个问题，我们提出了一种解耦管道，将对象变换、源区域修复（inpainting）以及目标区域优化分离出来处理。修复和优化过程均采用无需训练的扩散方法FreeFine实现。在我们的新GeoBench基准测试中进行了实验，该测试包含2D和3D编辑场景，在此基准下，FreeFine在图像保真度和编辑精度上优于现有的最先进替代方案，尤其是在进行具有挑战性的转换时效果更加突出。代码及基准可以在以下网址获取：this https URL
URL
https://arxiv.org/abs/2507.23300
PDF
-
Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation
2025-07-30 19:43:47Alexandru BuburuzanAbstract
Safety-critical applications, such as autonomous driving and medical image analysis, require extensive multimodal data for rigorous testing. Synthetic data methods are gaining prominence due to the cost and complexity of gathering real-world data, but they demand a high degree of realism and controllability to be useful. This work introduces two novel methods for synthetic data generation in autonomous driving and medical image analysis, namely MObI and AnydoorMed, respectively. MObI is a first-of-its-kind framework for Multimodal Object Inpainting that leverages a diffusion model to produce realistic and controllable object inpaintings across perceptual modalities, demonstrated simultaneously for camera and lidar. Given a single reference RGB image, MObI enables seamless object insertion into existing multimodal scenes at a specified 3D location, guided by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, this approach uses 3D bounding box conditioning to ensure accurate spatial positioning and realistic scaling. AnydoorMed extends this paradigm to the medical imaging domain, focusing on reference-guided inpainting for mammography scans. It leverages a diffusion-based model to inpaint anomalies with impressive detail preservation, maintaining the reference anomaly's structural integrity while semantically blending it with the surrounding tissue. Together, these methods demonstrate that foundation models for reference-guided inpainting in natural images can be readily adapted to diverse perceptual modalities, paving the way for the next generation of systems capable of constructing highly realistic, controllable and multimodal counterfactual scenarios.
Abstract (translated)
安全关键型应用，如自动驾驶和医学影像分析，需要大量的多模态数据来进行严格测试。由于收集现实世界数据的成本高昂且复杂性大，合成数据方法正日益受到重视，但这些方法要求高度的真实感和可控性才能发挥作用。这项工作介绍了两种用于生成自动驾驶和医学影像分析领域合成数据的新方法，分别是MObI（Multimodal Object Inpainting）和AnydoorMed。 MObI是一个开创性的多模态对象修复框架，利用扩散模型跨感知模式产生真实且可控制的对象修复效果，在相机图像和激光雷达数据中同时展示。给定单个参考RGB图像，MObI能够在指定的三维位置将对象无缝插入现有多种模态场景中，并通过边界框指导这一过程，确保语义一致性和多模态连贯性。与仅依赖编辑掩码的传统修复方法不同，这种方法使用3D边界框条件来保证准确的空间定位和逼真的缩放。 AnydoorMed则将这种范式扩展到医学成像领域，专注于基于参考图像的乳腺X光片异常修复。它利用扩散模型在保留参考异常结构完整性的前提下以惊人的细节保持能力对异常进行修复，并使其语义上与周围组织融合一致。 这两种方法共同表明，用于自然图像参考指导修复的基础模型可以轻松适应各种感知模式，为构建高度逼真、可控和多模态的反事实场景铺平了道路。
URL
https://arxiv.org/abs/2507.23058
PDF
-
Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards
2025-07-30 10:06:38Song Yan, Hui Wei, Jinlong Fei, Guoliang Yang, Zhengyu Zhao, Zheng WamgAbstract
Various (text) prompt filters and (image) safety checkers have been implemented to mitigate the misuse of Text-to-Image (T2I) models in creating Not-Safe-For-Work (NSFW) this http URL order to expose potential security vulnerabilities of such safeguards, multimodal jailbreaks have been this http URL, existing jailbreaks are limited to prompt-specific and image-specific perturbations, which suffer from poor scalability and time-consuming this http URL address these limitations, we propose Universally Unfiltered and Unseen (U3)-Attack, a multimodal jailbreak attack method against T2I this http URL, U3-Attack optimizes an adversarial patch on the image background to universally bypass safety checkers and optimizes a safe paraphrase set from a sensitive word to universally bypass prompt filters while eliminating redundant this http URL experimental results demonstrate the superiority of our U3-Attack on both open-source and commercial T2I this http URL example, on the commercial Runway-inpainting model with both prompt filter and safety checker, our U3-Attack achieves $~4\times$ higher success rates than the state-of-the-art multimodal jailbreak attack, this http URL Warning: This paper includes examples of NSFW content.
Abstract (translated)
为了减少文本到图像（Text-to-Image，T2I）模型被滥用以生成不适合工作的内容（Not-Safe-For-Work，NSFW），已经实施了多种文本提示过滤器和图像安全检查措施。然而，为揭示这些防护措施的安全漏洞，多模态越狱攻击已经被提出。现有的越狱攻击方法局限于针对特定提示或特定图像的扰动处理，这导致了可扩展性差且耗时的问题。 为了克服这些问题，我们提出了一个名为“Universally Unfiltered and Unseen (U3)-Attack”的新多模态越狱攻击方法。该方法通过优化背景中的对抗贴图来普遍绕过安全检查器，并从敏感词汇中优化出一个无害的同义替换集合以普遍避开提示过滤器，同时消除了冗余。 实验结果显示，在开源和商用的文本到图像模型上，我们的U3-Attack表现出了优越性。例如，在使用了提示过滤器和安全性检查器的商用Runway-inpainting模型中，我们的U3-Attack比当前最先进的多模态越狱攻击方法获得了高出约4倍的成功率。 **警告：该论文包含不适合工作的内容示例。**
URL
https://arxiv.org/abs/2508.05658
PDF
-
GuidPaint: Class-Guided Image Inpainting with Diffusion Models
2025-07-29 09:36:52Qimin Wang, Xinda Liu, Guohua GengAbstract
In recent years, diffusion models have been widely adopted for image inpainting tasks due to their powerful generative capabilities, achieving impressive results. Existing multimodal inpainting methods based on diffusion models often require architectural modifications and retraining, resulting in high computational cost. In contrast, context-aware diffusion inpainting methods leverage the model's inherent priors to adjust intermediate denoising steps, enabling high-quality inpainting without additional training and significantly reducing computation. However, these methods lack fine-grained control over the masked regions, often leading to semantically inconsistent or visually implausible content. To address this issue, we propose GuidPaint, a training-free, class-guided image inpainting framework. By incorporating classifier guidance into the denoising process, GuidPaint enables precise control over intermediate generations within the masked areas, ensuring both semantic consistency and visual realism. Furthermore, it integrates stochastic and deterministic sampling, allowing users to select preferred intermediate results and deterministically refine them. Experimental results demonstrate that GuidPaint achieves clear improvements over existing context-aware inpainting methods in both qualitative and quantitative evaluations.
Abstract (translated)
近年来，扩散模型因其强大的生成能力而在图像修复任务中得到了广泛的应用，并取得了令人印象深刻的结果。然而，基于扩散模型的现有多模态修复方法通常需要对架构进行修改和重新训练，导致计算成本高昂。相比之下，感知上下文的扩散修复方法利用了模型本身的先验知识来调整中间去噪步骤，在无需额外训练的情况下实现了高质量的图像修复，并显著降低了计算需求。然而，这些方法缺乏对遮罩区域的精细控制，往往会导致语义不一致或视觉上不可信的内容。 为了解决这个问题，我们提出了GuidPaint，这是一种无须重新训练、基于类别指导的图像修复框架。通过在去噪过程中加入分类器引导，GuidPaint能够在遮盖区域内精确地控制中间生成过程，确保内容既语义连贯又视觉真实。此外，它还结合了随机和确定性采样方法，允许用户选择所需的中间结果，并以确定性方式对其进行优化。 实验结果显示，在定性和定量评估中，GuidPaint在感知上下文的修复方法上均取得了显著改进。
URL
https://arxiv.org/abs/2507.21627
PDF
-
Reconstruct or Generate: Exploring the Spectrum of Generative Modeling for Cardiac MRI
2025-07-25 11:53:50Niklas Bubeck, Yundi Zhang, Suprosanna Shit, Daniel Rueckert, Jiazhen PanAbstract
In medical imaging, generative models are increasingly relied upon for two distinct but equally critical tasks: reconstruction, where the goal is to restore medical imaging (usually inverse problems like inpainting or superresolution), and generation, where synthetic data is created to augment datasets or carry out counterfactual analysis. Despite shared architecture and learning frameworks, they prioritize different goals: generation seeks high perceptual quality and diversity, while reconstruction focuses on data fidelity and faithfulness. In this work, we introduce a "generative model zoo" and systematically analyze how modern latent diffusion models and autoregressive models navigate the reconstruction-generation spectrum. We benchmark a suite of generative models across representative cardiac medical imaging tasks, focusing on image inpainting with varying masking ratios and sampling strategies, as well as unconditional image generation. Our findings show that diffusion models offer superior perceptual quality for unconditional generation but tend to hallucinate as masking ratios increase, whereas autoregressive models maintain stable perceptual performance across masking levels, albeit with generally lower fidelity.
Abstract (translated)
在医学影像领域，生成模型越来越被用于两种不同但同样重要的任务：重建和生成。在重建任务中，目标是恢复医学影像（通常涉及逆问题如图像插值或超分辨率），而在生成任务中，则是为了创建合成数据以扩充数据集或进行反事实分析。尽管这些模型共享相似的架构和学习框架，但在优先目标上有所不同：生成任务追求高质量且多样化的视觉效果，而重建则侧重于数据的真实性和准确性。在这项工作中，我们引入了一个“生成模型动物园”，并系统地分析了现代潜在扩散模型和自回归模型在重建与生成之间的表现差异。我们在代表性的心脏医学影像任务中对一系列生成模型进行了基准测试，重点关注不同遮挡比例下的图像插值及无条件的图像生成。我们的研究结果表明，扩散模型在无条件生成方面提供了更好的视觉质量，但在遮挡比例增加时容易产生幻觉；而自回归模型则在整个遮挡级别上保持稳定的视觉性能，尽管其总体真实度较低。
URL
https://arxiv.org/abs/2507.19186
PDF
-
U-Net Based Healthy 3D Brain Tissue Inpainting
Abstract
This paper introduces a novel approach to synthesize healthy 3D brain tissue from masked input images, specifically focusing on the task of 'ASNR-MICCAI BraTS Local Synthesis of Tissue via Inpainting'. Our proposed method employs a U-Net-based architecture, which is designed to effectively reconstruct the missing or corrupted regions of brain MRI scans. To enhance our model's generalization capabilities and robustness, we implement a comprehensive data augmentation strategy that involves randomly masking healthy images during training. Our model is trained on the BraTS-Local-Inpainting dataset and demonstrates the exceptional performance in recovering healthy brain tissue. The evaluation metrics employed, including Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Mean Squared Error (MSE), consistently yields impressive results. On the BraTS-Local-Inpainting validation set, our model achieved an SSIM score of 0.841, a PSNR score of 23.257, and an MSE score of 0.007. Notably, these evaluation metrics exhibit relatively low standard deviations, i.e., 0.103 for SSIM score, 4.213 for PSNR score and 0.007 for MSE score, which indicates that our model's reliability and consistency across various input scenarios. Our method also secured first place in the challenge.
Abstract (translated)
本文提出了一种新颖的方法，用于从带有遮罩的输入图像中合成健康的人脑组织，特别关注的是“ASNR-MICCAI BraTS 局部组织插值”任务。我们提出的模型采用基于U-Net架构的设计，旨在有效重建大脑MRI扫描中缺失或损坏的部分区域。为了增强模型的泛化能力和鲁棒性，我们在训练过程中实施了一种全面的数据增强策略，即在随机遮罩健康图像时进行训练。 我们的模型是在BraTS-Local-Inpainting数据集上进行训练的，并且在恢复健康的脑组织方面表现出色。评估指标包括结构相似度指数（SSIM）、峰值信噪比（PSNR）和均方误差（MSE），这些指标持续显示出色的结果。在BraTS-Local-Inpainting验证集中，我们的模型达到了0.841的SSIM分数、23.257的PSNR分数以及0.007的MSE分数。 值得注意的是，这些评估指标的标准偏差相对较低，即SSIM得分为0.103，PSNR得分为4.213，MSE得分为0.007，这表明我们的模型在各种输入场景中的可靠性和一致性。此外，我们的方法在这次挑战赛中获得了第一名的成绩。
URL
https://arxiv.org/abs/2507.18126
PDF
-
High-fidelity 3D Gaussian Inpainting: preserving multi-view consistency and photorealistic details
2025-07-24 01:48:50Jun Zhou, Dinghao Li, Nannan Li, Mingjie WangAbstract
Recent advancements in multi-view 3D reconstruction and novel-view synthesis, particularly through Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have greatly enhanced the fidelity and efficiency of 3D content creation. However, inpainting 3D scenes remains a challenging task due to the inherent irregularity of 3D structures and the critical need for maintaining multi-view consistency. In this work, we propose a novel 3D Gaussian inpainting framework that reconstructs complete 3D scenes by leveraging sparse inpainted views. Our framework incorporates an automatic Mask Refinement Process and region-wise Uncertainty-guided Optimization. Specifically, we refine the inpainting mask using a series of operations, including Gaussian scene filtering and back-projection, enabling more accurate localization of occluded regions and realistic boundary restoration. Furthermore, our Uncertainty-guided Fine-grained Optimization strategy, which estimates the importance of each region across multi-view images during training, alleviates multi-view inconsistencies and enhances the fidelity of fine details in the inpainted results. Comprehensive experiments conducted on diverse datasets demonstrate that our approach outperforms existing state-of-the-art methods in both visual quality and view consistency.
Abstract (translated)
最近，在多视角3D重建和新视角合成领域，特别是在使用神经辐射场（NeRF）和3D高斯点阵（3DGS）的技术进步极大地提高了三维内容创作的准确性和效率。然而，由于三维结构本身的不规则性以及维护多视图一致性的关键需求，对3D场景进行修复填充仍是一项具有挑战性的任务。 在这项工作中，我们提出了一种新颖的3D高斯修复框架，该框架通过利用稀疏的修复视图来重建完整的3D场景。我们的框架集成了自动掩码精炼过程和区域导向的不确定性指导优化策略。具体来说，我们使用一系列操作（包括高斯场景过滤和反向投影）来细化修复掩码，这使得能够更准确地定位被遮挡区域并恢复现实边界。 此外，我们在训练过程中提出了一种基于不确定性引导的细粒度优化策略，它估计了多视图图像中每个区域的重要性。这种方法减少了多视图之间的不一致性，并提高了修复结果中的细节准确性。 在多种数据集上进行的全面实验表明，我们的方法在视觉质量和视图一致性方面都优于现有的最先进的技术。
URL
https://arxiv.org/abs/2507.18023
PDF
-
Hierarchical Diffusion Framework for Pseudo-Healthy Brain MRI Inpainting with Enhanced 3D Consistency
2025-07-23 20:21:29Dou Hoon Kwark, Shirui Luo, Xiyue Zhu, Yudu Li, Zhi-Pei Liang, Volodymyr KindratenkoAbstract
Pseudo-healthy image inpainting is an essential preprocessing step for analyzing pathological brain MRI scans. Most current inpainting methods favor slice-wise 2D models for their high in-plane fidelity, but their independence across slices produces discontinuities in the volume. Fully 3D models alleviate this issue, but their high model capacity demands extensive training data for reliable, high-fidelity synthesis -- often impractical in medical settings. We address these limitations with a hierarchical diffusion framework by replacing direct 3D modeling with two perpendicular coarse-to-fine 2D stages. An axial diffusion model first yields a coarse, globally consistent inpainting; a coronal diffusion model then refines anatomical details. By combining perpendicular spatial views with adaptive resampling, our method balances data efficiency and volumetric consistency. Our experiments show our approach outperforms state-of-the-art baselines in both realism and volumetric consistency, making it a promising solution for pseudo-healthy image inpainting. Code is available at this https URL.
Abstract (translated)
伪健康图像修复是分析病理脑MRI扫描的重要预处理步骤。目前大多数的图像修复方法偏向于使用片状二维模型，因其在平面内的高保真度而被广泛采用；然而，这些模型独立处理每一层会带来体素间不连续性的问题。完全三维的模型可以解决这个问题，但其巨大的模型容量需要大量的训练数据来实现可靠且高质量的合成——这在医学环境中往往是不可行的。我们通过引入一个分层扩散框架解决了这些问题，在该框架中用两个相互垂直的从粗到细二维阶段替代了直接使用三维建模。首先采用轴向（axial）扩散模型进行初步、全局一致性的修复；随后，采用冠状面（coronal）扩散模型来细化解剖细节。通过结合不同空间视角和自适应重采样，我们的方法在数据效率与体素一致性之间取得了平衡。 实验表明，相较于现有的基准方法，我们的方法无论是在逼真度还是体积一致性上都表现更佳，为伪健康图像修复提供了有前景的解决方案。代码可在以下链接获取：[请在此处插入实际URL]。
URL
https://arxiv.org/abs/2507.17911
PDF
-
Diffusion Models for Solving Inverse Problems via Posterior Sampling with Piecewise Guidance
2025-07-22 19:35:14Saeed Mohseni-Sehdeh, Walid Saad, Kei Sakaguchi, Tao YuAbstract
Diffusion models are powerful tools for sampling from high-dimensional distributions by progressively transforming pure noise into structured data through a denoising process. When equipped with a guidance mechanism, these models can also generate samples from conditional distributions. In this paper, a novel diffusion-based framework is introduced for solving inverse problems using a piecewise guidance scheme. The guidance term is defined as a piecewise function of the diffusion timestep, facilitating the use of different approximations during high-noise and low-noise phases. This design is shown to effectively balance computational efficiency with the accuracy of the guidance term. Unlike task-specific approaches that require retraining for each problem, the proposed method is problem-agnostic and readily adaptable to a variety of inverse problems. Additionally, it explicitly incorporates measurement noise into the reconstruction process. The effectiveness of the proposed framework is demonstrated through extensive experiments on image restoration tasks, specifically image inpainting and super-resolution. Using a class conditional diffusion model for recovery, compared to the \pgdm baseline, the proposed framework achieves a reduction in inference time of \(25\%\) for inpainting with both random and center masks, and \(23\%\) and \(24\%\) for \(4\times\) and \(8\times\) super-resolution tasks, respectively, while incurring only negligible loss in PSNR and SSIM.
Abstract (translated)
扩散模型是一种强大的工具，能够通过逐步将纯噪声转化为结构化数据的去噪过程从高维分布中采样。当配备有引导机制时，这些模型也可以生成条件分布中的样本。在这篇论文中，提出了一种新的基于扩散框架的方法，用于采用分段引导方案解决逆向问题。该引导项被定义为扩散时间步长的分段函数，从而可以在高噪声和低噪声阶段使用不同的近似值。这种设计能够有效平衡计算效率与指导项的准确性之间的关系。不同于需要针对每个任务重新训练的任务特定方法，所提出的方法是无关任务的，并且可以轻松适应各种逆向问题。此外，它明确将测量噪声纳入到重建过程中。 通过广泛的图像恢复实验（具体为图像修复和超分辨率）展示了该框架的有效性。使用类条件扩散模型进行恢复时，与\pgdm基准方法相比，在随机掩码和中心掩码下的图像修补任务中，所提出的框架分别减少了25%的推理时间；在4倍和8倍超分辨率任务中，分别减少了23%和24%，同时仅造成了PSNR（峰值信噪比）和SSIM（结构相似性指数）上微不足道的质量损失。
URL
https://arxiv.org/abs/2507.18654
PDF
-
HarmonPaint: Harmonized Training-Free Diffusion Inpainting
2025-07-22 16:14:35Ying Li, Xinzhe Li, Yong Du, Yangyang Xu, Junyu Dong, Shengfeng HeAbstract
Existing inpainting methods often require extensive retraining or fine-tuning to integrate new content seamlessly, yet they struggle to maintain coherence in both structure and style between inpainted regions and the surrounding background. Motivated by these limitations, we introduce HarmonPaint, a training-free inpainting framework that seamlessly integrates with the attention mechanisms of diffusion models to achieve high-quality, harmonized image inpainting without any form of training. By leveraging masking strategies within self-attention, HarmonPaint ensures structural fidelity without model retraining or fine-tuning. Additionally, we exploit intrinsic diffusion model properties to transfer style information from unmasked to masked regions, achieving a harmonious integration of styles. Extensive experiments demonstrate the effectiveness of HarmonPaint across diverse scenes and styles, validating its versatility and performance.
Abstract (translated)
现有的图像修复方法通常需要大量的重新训练或微调，才能将新的内容无缝地整合进来，但它们在保持修补区域与周围背景之间结构和风格的一致性方面存在困难。鉴于这些限制，我们引入了HarmonPaint，这是一个无需训练的图像修复框架，它通过利用扩散模型中的注意力机制来实现高质量且一致的图像修复。HarmonPaint通过在自注意力中使用掩码策略，在不进行任何重新训练或微调的情况下保证结构的一致性。此外，该方法还利用扩散模型的内在属性将风格信息从未被遮罩的部分传递到被遮罩的部分，从而实现了风格的一致融合。广泛的实验验证了HarmonPaint在各种场景和风格中的有效性和通用性。
URL
https://arxiv.org/abs/2507.16732
PDF
-
Blended Point Cloud Diffusion for Localized Text-guided Shape Editing
2025-07-21 09:00:19Etai Sella, Noam Atia, Ron Mokady, Hadar Averbuch-ElorAbstract
Natural language offers a highly intuitive interface for enabling localized fine-grained edits of 3D shapes. However, prior works face challenges in preserving global coherence while locally modifying the input 3D shape. In this work, we introduce an inpainting-based framework for editing shapes represented as point clouds. Our approach leverages foundation 3D diffusion models for achieving localized shape edits, adding structural guidance in the form of a partial conditional shape, ensuring that other regions correctly preserve the shape's identity. Furthermore, to encourage identity preservation also within the local edited region, we propose an inference-time coordinate blending algorithm which balances reconstruction of the full shape with inpainting at a progression of noise levels during the inference process. Our coordinate blending algorithm seamlessly blends the original shape with its edited version, enabling a fine-grained editing of 3D shapes, all while circumventing the need for computationally expensive and often inaccurate inversion. Extensive experiments show that our method outperforms alternative techniques across a wide range of metrics that evaluate both fidelity to the original shape and also adherence to the textual description.
Abstract (translated)
自然语言提供了一种高度直观的界面，用于对3D形状进行本地化的精细编辑。然而，之前的工作在局部修改输入3D形状的同时保持全局一致性方面面临挑战。在这项工作中，我们提出了一种基于图像修复（inpainting）的框架，用于编辑表示为点云的形状。我们的方法利用基础3D扩散模型来实现局部形状编辑，并以部分条件形状的形式添加结构指导，确保其他区域正确地保留形状的身份特征。 为了进一步鼓励在本地编辑区域内保持身份特征，我们提出了一种推理时间坐标融合算法，在推理过程中通过逐步调整噪声水平平衡完整形状的重建和图像修复。我们的坐标融合算法可以无缝地将原始形状与其编辑版本结合在一起，从而实现对3D形状进行精细控制的编辑，并且避免了计算成本高昂且通常不准确的逆向过程。 广泛的实验表明，我们提出的方法在评估原始形状保真度和文本描述符合性的广泛指标上均优于其他技术。
URL
https://arxiv.org/abs/2507.15399
PDF
-
Latent Space Synergy: Text-Guided Data Augmentation for Direct Diffusion Biomedical Segmentation
2025-07-21 08:15:17Muhammad Aqeel, Maham Nazir, Zanxi Ruan, Francesco SettiAbstract
Medical image segmentation suffers from data scarcity, particularly in polyp detection where annotation requires specialized expertise. We present SynDiff, a framework combining text-guided synthetic data generation with efficient diffusion-based segmentation. Our approach employs latent diffusion models to generate clinically realistic synthetic polyps through text-conditioned inpainting, augmenting limited training data with semantically diverse samples. Unlike traditional diffusion methods requiring iterative denoising, we introduce direct latent estimation enabling single-step inference with T x computational speedup. On CVC-ClinicDB, SynDiff achieves 96.0% Dice and 92.9% IoU while maintaining real-time capability suitable for clinical deployment. The framework demonstrates that controlled synthetic augmentation improves segmentation robustness without distribution shift. SynDiff bridges the gap between data-hungry deep learning models and clinical constraints, offering an efficient solution for deployment in resourcelimited medical settings.
Abstract (translated)
医学图像分割面临着数据稀缺的问题，尤其是在息肉检测中，标注需要专门的知识和技能。我们提出了SynDiff框架，结合了文本引导的合成数据生成与高效的扩散基序分割方法。我们的方法利用潜在扩散模型通过基于文本条件的修复技术来生成临床上真实的合成息肉样本，从而用语义多样化的样本来增强有限的训练数据集。 不同于传统的需要迭代去噪处理的扩散方法，我们引入了直接潜在估计的方法，能够在单步推理中实现T x（其中T为传统模型的迭代次数）的加速效果。在CVC-ClinicDB数据集中，SynDiff达到了96.0%的Dice系数和92.9%的IoU值，并且保持了实时处理能力，适用于临床应用部署。 该框架表明，在不产生分布偏移的情况下，受控合成增强可以提高分割模型的鲁棒性。SynDiff弥合了对大量数据有需求的深度学习模型与临床约束之间的差距，为资源有限的医疗环境提供了一种高效的解决方案。
URL
https://arxiv.org/abs/2507.15361
PDF
-
FreeCus: Free Lunch Subject-driven Customization in Diffusion Transformers
2025-07-21 05:15:45Yanbing Zhang, Zhe Wang, Qin Zhou, Mengping YangAbstract
In light of recent breakthroughs in text-to-image (T2I) generation, particularly with diffusion transformers (DiT), subject-driven technologies are increasingly being employed for high-fidelity customized production that preserves subject identity from reference inputs, enabling thrilling design workflows and engaging entertainment. Existing alternatives typically require either per-subject optimization via trainable text embeddings or training specialized encoders for subject feature extraction on large-scale datasets. Such dependencies on training procedures fundamentally constrain their practical applications. More importantly, current methodologies fail to fully leverage the inherent zero-shot potential of modern diffusion transformers (e.g., the Flux series) for authentic subject-driven synthesis. To bridge this gap, we propose FreeCus, a genuinely training-free framework that activates DiT's capabilities through three key innovations: 1) We introduce a pivotal attention sharing mechanism that captures the subject's layout integrity while preserving crucial editing flexibility. 2) Through a straightforward analysis of DiT's dynamic shifting, we propose an upgraded variant that significantly improves fine-grained feature extraction. 3) We further integrate advanced Multimodal Large Language Models (MLLMs) to enrich cross-modal semantic representations. Extensive experiments reflect that our method successfully unlocks DiT's zero-shot ability for consistent subject synthesis across diverse contexts, achieving state-of-the-art or comparable results compared to approaches that require additional training. Notably, our framework demonstrates seamless compatibility with existing inpainting pipelines and control modules, facilitating more compelling experiences. Our code is available at: this https URL.
Abstract (translated)
鉴于近期在文本到图像（T2I）生成方面，特别是扩散变压器（DiT）方面的突破，基于主题的驱动技术越来越多地被用于高保真定制化生产，该过程从参考输入中保持主体身份的同时支持令人兴奋的设计工作流程和娱乐活动。现有的替代方法通常需要通过可训练的文本嵌入进行每主体优化或在大规模数据集上对特定于主体特征提取的编码器进行专门培训。这种依赖于训练程序的因素从根本上限制了它们的实际应用范围。更重要的是，当前的方法未能充分利用现代扩散变压器（例如Flux系列）固有的零样本潜力来进行真实的主题驱动合成。 为了解决这一差距，我们提出了FreeCus，这是一个真正的无需训练的框架，通过三个关键创新来激活DiT的能力：1）我们引入了一个至关重要的注意力共享机制，该机制在捕捉主体布局完整性的同时保持了必要的编辑灵活性。2）通过对DiT动态偏移的简单分析，我们提出了一种改进的变体，显著提升了细粒度特征提取能力。3）我们进一步整合了先进的多模态大型语言模型（MLLM），以丰富跨模式语义表示。 广泛的实验表明，我们的方法成功地解锁了DiT在各种上下文中的零样本能力，进行一致的主题合成，并且与需要额外训练的方法相比取得了最先进的或可比较的结果。值得注意的是，我们的框架展示了与其现有的修补管线和控制模块无缝兼容的特性，从而促进了更加引人入胜的体验。我们的代码可在以下网址获取：this https URL。
URL
https://arxiv.org/abs/2507.15249
PDF
-
Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimization
2025-07-20 06:59:42Xiang Tang, Ruotong Li, Xiaopeng FanAbstract
In recent years, 3D generation has made great strides in both academia and industry. However, generating 3D scenes from a single RGB image remains a significant challenge, as current approaches often struggle to ensure both object generation quality and scene coherence in multi-object scenarios. To overcome these limitations, we propose a novel three-stage framework for 3D scene generation with explicit geometric representations and high-quality textural details via single image-guided model generation and spatial layout optimization. Our method begins with an image instance segmentation and inpainting phase, which recovers missing details of occluded objects in the input images, thereby achieving complete generation of foreground 3D assets. Subsequently, our approach captures the spatial geometry of reference image by constructing pseudo-stereo viewpoint for camera parameter estimation and scene depth inference, while employing a model selection strategy to ensure optimal alignment between the 3D assets generated in the previous step and the input. Finally, through model parameterization and minimization of the Chamfer distance between point clouds in 3D and 2D space, our approach optimizes layout parameters to produce an explicit 3D scene representation that maintains precise alignment with input guidance image. Extensive experiments on multi-object scene image sets have demonstrated that our approach not only outperforms state-of-the-art methods in terms of geometric accuracy and texture fidelity of individual generated 3D models, but also has significant advantages in scene layout synthesis.
Abstract (translated)
近年来，3D生成技术在学术界和工业界都取得了重大进展。然而，从单张RGB图像生成3D场景仍然是一个巨大的挑战，因为现有方法常常难以同时保证多对象场景中的物体生成质量和场景一致性。为了克服这些限制，我们提出了一种新颖的三阶段框架，用于通过单一图像引导模型生成和空间布局优化来创建具有显式几何表示和高质量纹理细节的3D场景。 我们的方法首先进行图像实例分割与修复阶段，在此过程中恢复输入图片中被遮挡物体缺失的详细信息，从而实现前景3D资产的完整生成。随后，我们通过构建伪立体视角来进行相机参数估计和场景深度推断，以此捕捉参考图的空间几何结构，并采用模型选择策略确保先前步骤产生的3D资产与输入图像的最佳对齐。最后，通过对模型进行参数化处理并最小化3D和2D空间中点云之间的Chamfer距离，我们的方法优化布局参数，从而生成一个显式的3D场景表示，保持精确地与指导输入图一致。 在多对象场景图像集上的广泛实验表明，我们所提出的方法不仅在单个生成的3D模型的几何准确性和纹理保真度方面超越了最先进的技术，还在场景布局合成方面具有显著优势。
URL
https://arxiv.org/abs/2507.14841
PDF
-
Clutter Detection and Removal by Multi-Objective Analysis for Photographic Guidance
2025-07-19 09:15:17Xiaoran WuAbstract
Clutter in photos is a distraction preventing photographers from conveying the intended emotions or stories to the audience. Photography amateurs frequently include clutter in their photos due to unconscious negligence or the lack of experience in creating a decluttered, aesthetically appealing scene for shooting. We are thus motivated to develop a camera guidance system that provides solutions and guidance for clutter identification and removal. We estimate and visualize the contribution of objects to the overall aesthetics and content of a photo, based on which users can interactively identify clutter. Suggestions on getting rid of clutter, as well as a tool that removes cluttered objects computationally, are provided to guide users to deal with different kinds of clutter and improve their photographic work. Two technical novelties underpin interactions in our system: a clutter distinguishment algorithm with aesthetics evaluations for objects and an iterative image inpainting algorithm based on generative adversarial nets that reconstructs missing regions of removed objects for high-resolution images. User studies demonstrate that our system provides flexible interfaces and accurate algorithms that allow users to better identify distractions and take higher quality images within less time.
Abstract (translated)
照片中的杂乱是妨碍摄影师向观众传达意图情感或故事的障碍。摄影初学者常常因为无意识的疏忽或缺乏创建整洁、美观拍摄场景的经验而将杂物纳入照片中。因此，我们致力于开发一个相机引导系统，为用户提供解决和指导杂乱识别与去除的方法。该系统基于对物体在整体美学和内容中的贡献进行估计和可视化，帮助用户交互式地识别杂物，并提供有关如何消除杂乱的建议以及一种通过计算移除杂乱对象的工具，从而引导用户处理不同类型的杂乱并改进他们的摄影作品。 我们系统的两个技术亮点支持了这些互动：一个具有美学评估功能的杂物区分算法和基于生成对抗网络（GAN）的迭代图像修复算法。后者用于重建被删除物体后的缺失区域，适用于高分辨率图片。用户体验研究表明，我们的系统提供了灵活的界面和准确的算法，使用户能够更有效地识别干扰因素，并在较短的时间内拍摄出更高品质的照片。
URL
https://arxiv.org/abs/2507.14553
PDF
-
fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting
2025-07-17 14:10:51Alicia Durrer, Florentin Bieder, Paul Friedrich, Bjoern Menze, Philippe C. Cattin, Florian KoflerAbstract
Healthy tissue inpainting has significant applications, including the generation of pseudo-healthy baselines for tumor growth models and the facilitation of image registration. In previous editions of the BraTS Local Synthesis of Healthy Brain Tissue via Inpainting Challenge, denoising diffusion probabilistic models (DDPMs) demonstrated qualitatively convincing results but suffered from low sampling speed. To mitigate this limitation, we adapted a 2D image generation approach, combining DDPMs with generative adversarial networks (GANs) and employing a variance-preserving noise schedule, for the task of 3D inpainting. Our experiments showed that the variance-preserving noise schedule and the selected reconstruction losses can be effectively utilized for high-quality 3D inpainting in a few time steps without requiring adversarial training. We applied our findings to a different architecture, a 3D wavelet diffusion model (WDM3D) that does not include a GAN component. The resulting model, denoted as fastWDM3D, obtained a SSIM of 0.8571, a MSE of 0.0079, and a PSNR of 22.26 on the BraTS inpainting test set. Remarkably, it achieved these scores using only two time steps, completing the 3D inpainting process in 1.81 s per image. When compared to other DDPMs used for healthy brain tissue inpainting, our model is up to 800 x faster while still achieving superior performance metrics. Our proposed method, fastWDM3D, represents a promising approach for fast and accurate healthy tissue inpainting. Our code is available at this https URL.
Abstract (translated)
健康组织修复技术在多种领域有着重要的应用，包括生成肿瘤生长模型的伪正常基准以及促进图像配准。在之前的BraTS局部合成健康脑组织通过修复竞赛中，去噪扩散概率模型（DDPMs）虽然展示出了定性上令人信服的结果，但因其采样速度慢而存在局限性。为了克服这一限制，我们结合了二维图像生成方法，将DDPMs与生成对抗网络（GANs）相结合，并采用了一种保方差噪声时间表进行三维修复任务。我们的实验表明，在不使用对抗训练的情况下，这种保方差的噪声时间表和所选的重构损失可以在几步时间内有效用于高质量的三维修复。 我们还将研究结果应用于一种不同的架构——无GAN组件的三维小波扩散模型（WDM3D），从而获得了名为fastWDM3D的结果模型。该模型在BraTS修复测试集上的SSIM值为0.8571，MSE值为0.0079，PSNR值为22.26，并且仅用了两个时间步骤即完成了三维修复过程，每张图像的处理时间为1.81秒。与用于健康脑组织修复的其他DDPMs相比，我们的模型在速度上快了多达800倍，同时仍保持优异的性能指标。 我们提出的快速且准确的健康组织修复方法fastWDM3D，在三维组织修复方面展现出了巨大潜力。相关代码可在以下链接找到：[此处插入实际链接地址]
URL
https://arxiv.org/abs/2507.13146
PDF
-
Reconstruct, Inpaint, Finetune: Dynamic Novel-view Synthesis from Monocular Videos
2025-07-16 21:40:29Kaihua Chen, Tarasha Khurana, Deva RamananAbstract
We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be "inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.
Abstract (translated)
我们研究了从单目视频生成动态场景的新视角合成方法。先前的方法依赖于昂贵的测试时间4D表示优化，或者在前馈训练方式下不能保持场景几何结构。我们的方法基于三个关键见解：(1) 对于在输入和目标视图中都可见的共视像素（covisible pixels），可以通过首先重建动态3D场景然后从新视角进行渲染来生成；(2) 新视角中的隐藏像素可以使用前馈2D视频扩散模型（即填充缺失信息）来“修复”。特别地，我们的视频修复扩散模型(CogNVS)可以从2D视频中自我监督学习，这使我们能够在其上训练大量野外视频数据集。进而使得CogNVS可以通过测试时间微调的方式零样本应用于新的测试视频。(3) 我们通过实证验证发现，CogNVS在从单目视频生成动态场景的新视角合成方面几乎超越了所有先前的方法。
URL
https://arxiv.org/abs/2507.12646
PDF
-
Revealing the Ancient Beauty: Digital Reconstruction of Temple Tiles using Computer Vision
2025-07-16 12:46:04Arkaprabha BasuAbstract
Modern digitised approaches have dramatically changed the preservation and restoration of cultural treasures, integrating computer scientists into multidisciplinary projects with ease. Machine learning, deep learning, and computer vision techniques have revolutionised developing sectors like 3D reconstruction, picture inpainting,IoT-based methods, genetic algorithms, and image processing with the integration of computer scientists into multidisciplinary initiatives. We suggest three cutting-edge techniques in recognition of the special qualities of Indian monuments, which are famous for their architectural skill and aesthetic appeal. First is the Fractal Convolution methodology, a segmentation method based on image processing that successfully reveals subtle architectural patterns within these irreplaceable cultural buildings. The second is a revolutionary Self-Sensitive Tile Filling (SSTF) method created especially for West Bengal's mesmerising Bankura Terracotta Temples with a brand-new data augmentation method called MosaicSlice on the third. Furthermore, we delve deeper into the Super Resolution strategy to upscale the images without losing significant amount of quality. Our methods allow for the development of seamless region-filling and highly detailed tiles while maintaining authenticity using a novel data augmentation strategy within affordable costs introducing automation. By providing effective solutions that preserve the delicate balance between tradition and innovation, this study improves the subject and eventually ensures unrivalled efficiency and aesthetic excellence in cultural heritage protection. The suggested approaches advance the field into an era of unmatched efficiency and aesthetic quality while carefully upholding the delicate equilibrium between tradition and innovation.
Abstract (translated)
现代数字化方法已经极大地改变了文化遗产的保护和修复，将计算机科学家轻松地融入多学科项目中。机器学习、深度学习以及计算机视觉技术在3D重建、图像修补、基于物联网的方法、遗传算法及图像处理等领域革新了发展中的行业，尤其是在计算机科学家参与多学科合作的情况下。 我们针对印度纪念碑的独特特点提出了三种前沿技术，这些纪念碑以其建筑技艺和美学吸引力而闻名于世。第一项是分形卷积方法（Fractal Convolution methodology），这是一种基于图像处理的分割方法，能够成功揭示这些不可替代的文化建筑中的细微建筑模式。第二种是一项革命性的自适应填砖法（Self-Sensitive Tile Filling, SSTF），专门为西孟加拉邦令人着迷的班库拉陶制寺庙设计，并引入了一种全新的数据增强技术——马赛克切片（MosaicSlice）。第三，我们深入探讨了超分辨率策略，在不影响图像质量的前提下提升图像分辨率。我们的方法允许在保持真实性的同时发展无缝区域填充和高度详细的砖块，通过一种新颖的数据增强策略以经济实惠的方式实现自动化。 本研究提供有效的解决方案来维护传统与创新之间的微妙平衡，并最终确保文化遗产保护的无与伦比效率及美学卓越性。所提出的方案将该领域推进到一个前所未有的高效性和美学质量的时代，同时谨慎地保持了传统与创新之间微妙的均衡。
URL
https://arxiv.org/abs/2507.12195
PDF
-
MFGDiffusion: Mask-Guided Smoke Synthesis for Enhanced Forest Fire Detection
2025-07-15 12:25:35Guanghao Wu, Chen Xu, Hai Song, Chong Wang, Qixing ZhangAbstract
Smoke is the first visible indicator of a this http URL the advancement of deep learning, image-based smoke detection has become a crucial method for detecting and preventing forest fires. However, the scarcity of smoke image data from forest fires is one of the significant factors hindering the detection of forest fire smoke. Image generation models offer a promising solution for synthesizing realistic smoke images. However, current inpainting models exhibit limitations in generating high-quality smoke representations, particularly manifesting as inconsistencies between synthesized smoke and background contexts. To solve these problems, we proposed a comprehensive framework for generating forest fire smoke images. Firstly, we employed the pre-trained segmentation model and the multimodal model to obtain smoke masks and image this http URL, to address the insufficient utilization of masks and masked images by inpainting models, we introduced a network architecture guided by mask and masked image features. We also proposed a new loss function, the mask random difference loss, which enhances the consistency of the generated effects around the mask by randomly expanding and eroding the mask this http URL, to generate a smoke image dataset using random masks for subsequent detection tasks, we incorporated smoke characteristics and use a multimodal large language model as a filtering tool to select diverse and reasonable smoke images, thereby improving the quality of the synthetic dataset. Experiments showed that our generated smoke images are realistic and diverse, and effectively enhance the performance of forest fire smoke detection models. Code is available at this https URL.
Abstract (translated)
烟是森林火灾早期最明显的视觉标志之一。随着深度学习的进展，基于图像的烟雾检测已成为发现和预防森林火灾的关键方法。然而，缺乏来自实际森林火灾的烟雾图像数据是阻碍有效检测的一个重要因素。图像生成模型为合成逼真的烟雾图像提供了一个有前景的解决方案。但是，现有的修复模型在生成高质量的烟雾表示时存在局限性，尤其表现为合成出的烟雾与其背景环境之间的一致性不足。 为了解决这些问题，我们提出了一种用于生成森林火灾烟雾图像的全面框架。首先，我们利用预训练的分割模型和多模态模型获取烟雾掩码（即标记了烟雾区域的图像）。为了克服修复模型对这些烟雾掩码及掩码后图像利用不足的问题，我们引入了一个由掩码与掩码后图像特征引导的网络架构。此外，我们提出了一种新的损失函数——随机差异掩码损失，通过随机扩展和腐蚀掩码来增强生成效果在掩码周围的一致性。 为了使用随机掩码生成烟雾图像数据集，以供后续检测任务使用，并提高合成数据集的质量，我们结合了烟雾特征并采用多模态大型语言模型作为过滤工具，选择多样且合理的烟雾图像。 实验结果表明，我们生成的烟雾图像既逼真又多样化，能有效提升森林火灾烟雾检测模型的性能。代码可在[提供的URL]获取。
URL
https://arxiv.org/abs/2507.11252
PDF
-
Warm Starts Accelerate Generative Modelling
Abstract
Iterative generative models, like diffusion and flow-matching, create high-fidelity samples by progressively refining a noise vector into data. However, this process is notoriously slow, often requiring hundreds of function evaluations. We introduce the warm-start model, a simple, deterministic model that dramatically accelerates conditional generation by providing a better starting point. Instead of starting generation from an uninformed N(0, I) prior, our warm-start model predicts an informed prior N(mu, sigma), whose moments are conditioned on the input context. This "warm start" substantially reduces the distance the generative process must traverse, particularly when the conditioning information is strongly informative. On tasks like image inpainting, our method achieves results competitive with a 1000-step DDPM baseline using only 11 total function evaluations (1 for the warm start, 10 for generation). A simple conditional normalization trick makes our method compatible with any standard generative model and sampler without modification, allowing it to be combined with other efficient sampling techniques for further acceleration. Our implementation is available at this https URL.
Abstract (translated)
迭代生成模型，如扩散和流匹配模型，通过逐步将噪声向量细化为数据来创建高保真样本。然而，这一过程通常非常慢，往往需要进行数百次函数评估。我们引入了预热启动模型（warm-start model），这是一种简单且确定性的模型，可以通过提供一个更好的起点显著加速条件生成。我们的预热启动模型预测了一个基于输入上下文的有信息量的先验分布N(mu, sigma)，而不是从无信息的N(0, I)先验开始生成。这种“预热”大幅减少了生成过程必须覆盖的距离，尤其是在上下文信息非常有用的情况下。 在图像修复等任务上，我们的方法仅使用总共11次函数评估（一次用于预热启动，十次用于生成）就能达到与一个需要1000步的DDPM基线相当的结果。一种简单的条件归一化技巧使得该方法可以与其他标准生成模型和采样器兼容，无需进行任何修改，并允许它结合其他高效的采样技术以进一步加速。 我们的实现代码可在[此处](https://this https URL)获得。
URL
https://arxiv.org/abs/2507.09212
PDF
