Inpainting
-
RePaintGS: Reference-Guided Gaussian Splatting for Realistic and View-Consistent 3D Scene Inpainting
2025-07-11 09:26:07Ji Hyun Seo, Byounhyun Yoo, Gerard Jounghyun KimAbstract
Radiance field methods, such as Neural Radiance Field or 3D Gaussian Splatting, have emerged as seminal 3D representations for synthesizing realistic novel views. For practical applications, there is ongoing research on flexible scene editing techniques, among which object removal is a representative task. However, removing objects exposes occluded regions, often leading to unnatural appearances. Thus, studies have employed image inpainting techniques to replace such regions with plausible content - a task referred to as 3D scene inpainting. However, image inpainting methods produce one of many plausible completions for each view, leading to inconsistencies between viewpoints. A widely adopted approach leverages perceptual cues to blend inpainted views smoothly. However, it is prone to detail loss and can fail when there are perceptual inconsistencies across views. In this paper, we propose a novel 3D scene inpainting method that reliably produces realistic and perceptually consistent results even for complex scenes by leveraging a reference view. Given the inpainted reference view, we estimate the inpainting similarity of the other views to adjust their contribution in constructing an accurate geometry tailored to the reference. This geometry is then used to warp the reference inpainting to other views as pseudo-ground truth, guiding the optimization to match the reference appearance. Comparative evaluation studies have shown that our approach improves both the geometric fidelity and appearance consistency of inpainted scenes.
Abstract (translated)
辐射场方法，如神经辐射场或3D高斯点阵，在合成逼真的新视图方面作为重要的三维表示方式而崭露头角。在实际应用中，有关灵活场景编辑技术的研究正在进行之中，其中物体移除是一项代表性的任务。然而，移除物体会暴露被遮挡的区域，这常常会导致不自然的效果。因此，一些研究采用了图像修复技术来用合理的内容替代这些区域——这项工作被称为三维场景修复。但是，基于图像的方法为每个视图生成多种可能的完整内容之一，从而导致不同视角之间的不一致现象。一种广泛采用的方法是利用感知线索来平滑地融合修复后的视图。然而，这种方法容易丢失细节，并且当存在多视角间的感知差异时会失效。 在这篇论文中，我们提出了一种新的三维场景修复方法，它能够可靠地生成现实且在复杂场景中具有感知一致性的结果，通过利用参考视图来实现这一目标。给定修复后的参考视图后，我们会估计其他视图的修复相似度以调整它们对构建准确几何结构（针对参考）的贡献程度。随后使用该几何结构将参考视图中的修复内容转换到其他视角作为伪真实基准，并以此引导优化过程与参考视图外观相匹配。 比较评价研究显示我们的方法提升了修复场景在几何保真度和外观一致性方面的表现。
URL
https://arxiv.org/abs/2507.08434
PDF
-
Audio Inpanting using Discrete Diffusion Model
2025-07-11 06:25:49Tali Dror, Iftach Shoham, Moshe Buchris, Oren Gal, Haim Permuter, Gilad Katz, Eliya NachmaniAbstract
Audio inpainting refers to the task of reconstructing missing segments in corrupted audio recordings. While prior approaches-including waveform and spectrogram-based diffusion models-have shown promising results for short gaps, they often degrade in quality when gaps exceed 100 milliseconds (ms). In this work, we introduce a novel inpainting method based on discrete diffusion modeling, which operates over tokenized audio representations produced by a pre-trained audio tokenizer. Our approach models the generative process directly in the discrete latent space, enabling stable and semantically coherent reconstruction of missing audio. We evaluate the method on the MusicNet dataset using both objective and perceptual metrics across gap durations up to 300 ms. We further evaluated our approach on the MTG dataset, extending the gap duration to 500 ms. Experimental results demonstrate that our method achieves competitive or superior performance compared to existing baselines, particularly for longer gaps, offering a robust solution for restoring degraded musical recordings. Audio examples of our proposed method can be found at this https URL
Abstract (translated)
音频修复（audio inpainting）是指重建被损坏的音频记录中缺失的部分。尽管之前的方法，包括基于波形和频谱图的扩散模型，在处理短间隔时表现出了良好的结果，但当缺口超过100毫秒(ms)时，它们的质量通常会下降。在本研究中，我们介绍了一种新的修复方法，该方法基于离散扩散建模，并且是在预训练音频标记器生成的标记化音频表示上进行操作的。我们的方法直接对离散潜在空间中的生成过程进行建模，从而能够实现稳定而语义一致的缺失音频重建。 我们在MusicNet数据集上使用客观和感知指标评估了该方法，在300毫秒(ms)以内的缺口长度下进行了测试。我们还在MTG数据集上进一步验证了我们的方法，并将缺口持续时间延长至500毫秒。实验结果表明，与现有基准相比，我们的方法在较长的缺口情况下达到了竞争力或优越的表现，为修复受损音乐录音提供了稳健的解决方案。 有关我们提出的方法的音频示例，请参见此链接：[https URL]（此处应填写实际的URL）。
URL
https://arxiv.org/abs/2507.08333
PDF
-
Stable-Hair v2: Real-World Hair Transfer via Multiple-View Diffusion Model
2025-07-10 09:49:34Kuiyuan Sun, Yuxuan Zhang, Jichao Zhang, Jiaming Liu, Wei Wang, Niculae Sebe, Yao ZhaoAbstract
While diffusion-based methods have shown impressive capabilities in capturing diverse and complex hairstyles, their ability to generate consistent and high-quality multi-view outputs -- crucial for real-world applications such as digital humans and virtual avatars -- remains underexplored. In this paper, we propose Stable-Hair v2, a novel diffusion-based multi-view hair transfer framework. To the best of our knowledge, this is the first work to leverage multi-view diffusion models for robust, high-fidelity, and view-consistent hair transfer across multiple perspectives. We introduce a comprehensive multi-view training data generation pipeline comprising a diffusion-based Bald Converter, a data-augment inpainting model, and a face-finetuned multi-view diffusion model to generate high-quality triplet data, including bald images, reference hairstyles, and view-aligned source-bald pairs. Our multi-view hair transfer model integrates polar-azimuth embeddings for pose conditioning and temporal attention layers to ensure smooth transitions between views. To optimize this model, we design a novel multi-stage training strategy consisting of pose-controllable latent IdentityNet training, hair extractor training, and temporal attention training. Extensive experiments demonstrate that our method accurately transfers detailed and realistic hairstyles to source subjects while achieving seamless and consistent results across views, significantly outperforming existing methods and establishing a new benchmark in multi-view hair transfer. Code is publicly available at this https URL.
Abstract (translated)
尽管基于扩散的方法在捕捉多样且复杂的发型方面表现出令人印象深刻的能力，但它们生成一致且高质量的多视角输出（对于数字人类和虚拟角色等现实世界应用至关重要）的能力仍然未被充分探索。在这篇论文中，我们提出了Stable-Hair v2，这是一种新型的基于扩散的多视角发际线转换框架。据我们所知，这是首次利用多视角扩散模型进行稳健、高保真且视图一致的跨多个视角发型转换的工作。 我们的方法包括一个全面的多视角训练数据生成管道，其中包括基于扩散的光头转换器（Bald Converter）、数据增强式图像修复模型以及经过面部微调的多视角扩散模型，用于生成高质量的三元组数据集，包含光头图像、参考发型和视图对齐的源-光头成对图片。 我们的多视角发际线转换模型整合了极地-方位嵌入以进行姿态条件控制，并使用时间注意力层确保在不同视角之间的平滑过渡。为了优化这一模型，我们设计了一种新颖的多层次训练策略，包括可操控姿势的身份网络（IdentityNet）训练、发型提取器训练以及时间注意力训练。 通过广泛的实验表明，我们的方法能够准确地将详细且真实的发型转移到源对象上，并实现视图间无缝和一致的结果，在多视角发际线转换方面显著优于现有方法并建立了新的基准。代码在以下链接公开：[此链接]（请替换为实际的URL）。
URL
https://arxiv.org/abs/2507.07591
PDF
-
Generative Panoramic Image Stitching
2025-07-08 22:07:12Mathieu Tuli, Kaveh Kamali, David B. LindellAbstract
We introduce the task of generative panoramic image stitching, which aims to synthesize seamless panoramas that are faithful to the content of multiple reference images containing parallax effects and strong variations in lighting, camera capture settings, or style. In this challenging setting, traditional image stitching pipelines fail, producing outputs with ghosting and other artifacts. While recent generative models are capable of outpainting content consistent with multiple reference images, they fail when tasked with synthesizing large, coherent regions of a panorama. To address these limitations, we propose a method that fine-tunes a diffusion-based inpainting model to preserve a scene's content and layout based on multiple reference images. Once fine-tuned, the model outpaints a full panorama from a single reference image, producing a seamless and visually coherent result that faithfully integrates content from all reference images. Our approach significantly outperforms baselines for this task in terms of image quality and the consistency of image structure and scene layout when evaluated on captured datasets.
Abstract (translated)
我们介绍了一项名为生成全景图像拼接的任务，其目标是合成无缝的全景图，这些全景图忠实于包含视差效果和强烈光照变化、相机拍摄设置或风格差异的多张参考图像的内容。在这一挑战性的场景中，传统的图像拼接流水线无法正常工作，会产生鬼影等其他伪影。虽然最近的一些生成模型能够根据多张参考图像扩展内容一致的新区域，但在合成大型且连贯的全景图区域时却会失败。为了克服这些限制，我们提出了一种方法，即微调基于扩散的修复模型，以依据多张参考图像来保留场景的内容和布局。一旦经过微调，该模型可以从单个参考图像中扩展出完整的全景图，并生成无缝且视觉连贯的结果，忠实整合所有参考图像的内容。我们的方法在捕获数据集上评估时，在图像质量以及图像结构和场景布局的一致性方面显著优于基准方法。
URL
https://arxiv.org/abs/2507.07133
PDF
-
Diffusion-Based Limited-Angle CT Reconstruction under Noisy Conditions
2025-07-08 03:58:52Jiaqi Guo, Santiago L\'opez-TapiaAbstract
Limited-Angle Computed Tomography (LACT) is a challenging inverse problem where missing angular projections lead to incomplete sinograms and severe artifacts in the reconstructed images. While recent learning-based methods have demonstrated effectiveness, most of them assume ideal, noise-free measurements and fail to address the impact of measurement noise. To overcome this limitation, we treat LACT as a sinogram inpainting task and propose a diffusion-based framework that completes missing angular views using a Mean-Reverting Stochastic Differential Equation (MR-SDE) formulation. To improve robustness under realistic noise, we propose RNSD$^+$, a novel noise-aware rectification mechanism that explicitly models inference-time uncertainty, enabling reliable and robust reconstruction. Extensive experiments demonstrate that our method consistently surpasses baseline models in data consistency and perceptual quality, and generalizes well across varying noise intensity and acquisition scenarios.
Abstract (translated)
限制角度计算机断层扫描（LACT）是一个具有挑战性的逆向问题，其中缺少的角度投影会导致不完整的弦图，并在重建图像中产生严重的伪影。虽然最近基于学习的方法已经证明了其有效性，但大多数方法假设理想且无噪声的测量值，并未能解决测量噪声的影响。为了克服这一限制，我们将LACT视为一个弦图修复任务，并提出了一种基于扩散框架的方法，使用均值复归随机微分方程（MR-SDE）来完成缺失的角度视图。为提高在实际噪声下的鲁棒性，我们提出了RNSD$^+$，这是一种新颖的噪声感知校正机制，它明确地建模了推理时间的不确定性，从而能够实现可靠和稳健的重建。广泛的实验表明，我们的方法在数据一致性以及感知质量方面始终优于基线模型，并且能够在不同的噪声强度和获取场景下良好泛化。
URL
https://arxiv.org/abs/2507.05647
PDF
-
Kernel Density Steering: Inference-Time Scaling via Mode Seeking for Image Restoration
2025-07-08 02:33:44Yuyang Hu, Kangfu Mei, Mojtaba Sahraee-Ardakan, Ulugbek S. Kamilov, Peyman Milanfar, Mauricio DelbracioAbstract
Diffusion models show promise for image restoration, but existing methods often struggle with inconsistent fidelity and undesirable artifacts. To address this, we introduce Kernel Density Steering (KDS), a novel inference-time framework promoting robust, high-fidelity outputs through explicit local mode-seeking. KDS employs an $N$-particle ensemble of diffusion samples, computing patch-wise kernel density estimation gradients from their collective outputs. These gradients steer patches in each particle towards shared, higher-density regions identified within the ensemble. This collective local mode-seeking mechanism, acting as "collective wisdom", steers samples away from spurious modes prone to artifacts, arising from independent sampling or model imperfections, and towards more robust, high-fidelity structures. This allows us to obtain better quality samples at the expense of higher compute by simultaneously sampling multiple particles. As a plug-and-play framework, KDS requires no retraining or external verifiers, seamlessly integrating with various diffusion samplers. Extensive numerical validations demonstrate KDS substantially improves both quantitative and qualitative performance on challenging real-world super-resolution and image inpainting tasks.
Abstract (translated)
扩散模型在图像恢复方面表现出潜力，但现有的方法往往难以保持一致的保真度，并且会产生不希望出现的伪影。为了解决这些问题，我们引入了 Kernel Density Steering (KDS)，这是一种新颖的推理框架，通过显式的局部模式寻求机制来促进稳健、高保真的输出。 KDS 使用一个由 $N$ 个粒子组成的扩散样本集合，在这些样本的集体输出中计算补丁级别的核密度估计梯度。这些梯度将每个粒子中的补丁引导到集合内识别出的共享更高密度区域，从而形成了一种“群体智慧”的机制。这种集体局部模式寻求机制可以将样本从容易产生伪影的虚假模式（由于独立采样或模型不完善）中引导出来，并朝向更稳健、高保真的结构。 通过同时采样多个粒子，我们可以以更高的计算成本换取更好的质量样本。KDS 作为一种即插即用框架，无需重新训练或外部验证器即可与各种扩散采样器无缝集成。 大量的数值验证表明，KDS 在具有挑战性的现实世界超分辨率和图像修复任务中显著提高了定性和定量性能。
URL
https://arxiv.org/abs/2507.05604
PDF
-
Rethinking Layered Graphic Design Generation with a Top-Down Approach
2025-07-08 02:26:08Jingye Chen, Zhaowen Wang, Nanxuan Zhao, Li Zhang, Difan Liu, Jimei Yang, Qifeng ChenAbstract
Graphic design is crucial for conveying ideas and messages. Designers usually organize their work into objects, backgrounds, and vectorized text layers to simplify editing. However, this workflow demands considerable expertise. With the rise of GenAI methods, an endless supply of high-quality graphic designs in pixel format has become more accessible, though these designs often lack editability. Despite this, non-layered designs still inspire human designers, influencing their choices in layouts and text styles, ultimately guiding the creation of layered designs. Motivated by this observation, we propose Accordion, a graphic design generation framework taking the first attempt to convert AI-generated designs into editable layered designs, meanwhile refining nonsensical AI-generated text with meaningful alternatives guided by user prompts. It is built around a vision language model (VLM) playing distinct roles in three curated stages. For each stage, we design prompts to guide the VLM in executing different tasks. Distinct from existing bottom-up methods (e.g., COLE and Open-COLE) that gradually generate elements to create layered designs, our approach works in a top-down manner by using the visually harmonious reference image as global guidance to decompose each layer. Additionally, it leverages multiple vision experts such as SAM and element removal models to facilitate the creation of graphic layers. We train our method using the in-house graphic design dataset Design39K, augmented with AI-generated design images coupled with refined ground truth created by a customized inpainting model. Experimental results and user studies by designers show that Accordion generates favorable results on the DesignIntention benchmark, including tasks such as text-to-template, adding text to background, and text de-rendering, and also excels in creating design variations.
Abstract (translated)
图形设计在传达想法和信息方面至关重要。设计师通常会将他们的作品组织成对象层、背景层和矢量文本层，以此简化编辑过程。然而，这一工作流程需要相当的专业知识。随着生成人工智能（GenAI）方法的兴起，高分辨率像素格式的高质量图形设计变得更加触手可及。尽管如此，这些设计往往缺乏可编辑性。不过，非分层的设计仍然激发着人类设计师的灵感，影响他们的布局和文字风格选择，并最终指导他们创作出分层设计。基于这一观察，我们提出了Accordion框架——这是第一个尝试将AI生成的设计转化为可编辑分层设计的方法，同时根据用户的提示来优化无意义的AI生成文本，使其具有实际含义。 Accordion围绕一个视觉语言模型（VLM）构建而成，在三个精心策划的阶段中扮演不同角色。对于每个阶段，我们设计了引导性提示，帮助VLM执行不同的任务。与现有自下而上的方法（如COLE和Open-COLE）通过逐步生成元素来创建分层设计的方法不同，我们的方法采用一种从上至下的方式，即以视觉和谐的参考图像作为全局指导，将每个层次进行分解。此外，Accordion还利用多种视觉专家模型（例如SAM和去元素除除模型），从而方便图形层级的设计创作。 我们使用内部构建的Design39K图形设计数据集对方法进行了训练，并通过定制的修复完成模型生成了与AI产生的设计方案相匹配的真实地面实况标签。实验结果和设计师进行的用户研究显示，Accordion在DesignIntention基准测试中的表现良好，包括文本到模板、背景加文字以及去渲染等任务，同时在创造设计变体方面也表现出色。
URL
https://arxiv.org/abs/2507.05601
PDF
-
Structure-Guided Diffusion Models for High-Fidelity Portrait Shadow Removal
2025-07-07 06:19:07Wanchang Yu, Qing Zhang, Rongjia Zheng, Wei-Shi ZhengAbstract
We present a diffusion-based portrait shadow removal approach that can robustly produce high-fidelity results. Unlike previous methods, we cast shadow removal as diffusion-based inpainting. To this end, we first train a shadow-independent structure extraction network on a real-world portrait dataset with various synthetic lighting conditions, which allows to generate a shadow-independent structure map including facial details while excluding the unwanted shadow boundaries. The structure map is then used as condition to train a structure-guided inpainting diffusion model for removing shadows in a generative manner. Finally, to restore the fine-scale details (e.g., eyelashes, moles and spots) that may not be captured by the structure map, we take the gradients inside the shadow regions as guidance and train a detail restoration diffusion model to refine the shadow removal result. Extensive experiments on the benchmark datasets show that our method clearly outperforms existing methods, and is effective to avoid previously common issues such as facial identity tampering, shadow residual, color distortion, structure blurring, and loss of details. Our code is available at this https URL.
Abstract (translated)
我们提出了一种基于扩散的肖像阴影移除方法，能够稳健地产生高保真度的结果。与以往的方法不同，我们将阴影移除问题视为一种基于扩散的修复（inpainting）任务。为此，我们首先在一个包含多种合成光照条件的真实世界肖像数据集上训练了一个独立于阴影的结构提取网络，从而生成一个包括面部细节但排除不需要的阴影边界的独立于阴影的结构图。然后使用该结构图为指导来训练一个结构导向的修复扩散模型，以生成方式移除阴影。最后，为了恢复可能未被结构图捕捉到的细小特征（如睫毛、痣和斑点），我们采用阴影区域内的梯度作为引导，并训练一个细节恢复扩散模型对阴影移除结果进行精炼。 在基准数据集上的大量实验表明，我们的方法明显优于现有的方法，并能有效避免诸如面部身份篡改、阴影残留、颜色失真、结构模糊以及细节丢失等以往常见的问题。我们的代码可在以下链接获取：[此链接]（请将方括号中的内容替换为实际的URL）。
URL
https://arxiv.org/abs/2507.04692
PDF
-
Neuroverse3D: Developing In-Context Learning Universal Model for Neuroimaging in 3D
2025-07-04 10:04:18Jiesi Hu, Chenfei Ye, Yanwu Yang, Xutao Guo, Yang Shang, Pengcheng Shi, Hanyang Peng, Ting MaAbstract
In-context learning (ICL), a type of universal model, demonstrates exceptional generalization across a wide range of tasks without retraining by leveraging task-specific guidance from context, making it particularly effective for the intricate demands of neuroimaging. However, current ICL models, limited to 2D inputs and thus exhibiting suboptimal performance, struggle to extend to 3D inputs due to the high memory demands of ICL. In this regard, we introduce Neuroverse3D, an ICL model capable of performing multiple neuroimaging tasks in 3D (e.g., segmentation, denoising, inpainting). Neuroverse3D overcomes the large memory consumption associated with 3D inputs through adaptive parallel-sequential context processing and a U-shaped fusion strategy, allowing it to handle an unlimited number of context images. Additionally, we propose an optimized loss function to balance multi-task training and enhance focus on anatomical boundaries. Our study incorporates 43,674 3D multi-modal scans from 19 neuroimaging datasets and evaluates Neuroverse3D on 14 diverse tasks using held-out test sets. The results demonstrate that Neuroverse3D significantly outperforms existing ICL models and closely matches task-specific models, enabling flexible adaptation to medical center variations without retraining. The code and model weights are publicly available at this https URL.
Abstract (translated)
**翻译** 在上下文学习（ICL）这种通用模型中，通过利用任务特定的指导信息进行泛化处理，在不重新训练的情况下能够在广泛的任务范围内表现出色。特别是在神经成像的复杂需求方面尤为有效。然而，目前的ICL模型仅限于2D输入，因此性能表现不足，并且难以扩展到3D输入，因为这会带来巨大的内存消耗问题。鉴于此，我们引入了Neuroverse3D，这是一种能够执行多种神经成像任务（例如分割、去噪和填充）的ICL模型，在3D环境中运作。 通过自适应并行-串行上下文处理以及U形融合策略，Neuroverse3D克服了与3D输入相关的大量内存消耗问题，使其可以处理无限数量的上下文图像。此外，我们还提出了一种优化过的损失函数来平衡多任务训练，并增强对解剖边界的关注。 我们的研究采用了来自19个神经成像数据集中的43,674个3D多模态扫描，并在包含14项不同任务的保留测试集合上评估了Neuroverse3D的表现。结果表明，Neuroverse3D显著优于现有的ICL模型，并且接近于特定任务模型的效果，这使得它能够灵活地适应医疗中心之间的差异而不必重新训练。 该代码和模型权重可在以下网址公开获取：[https://this-url.com](https://this-url.com)
URL
https://arxiv.org/abs/2503.02410
PDF
-
MAGIC: Mask-Guided Diffusion Inpainting with Multi-Level Perturbations and Context-Aware Alignment for Few-Shot Anomaly Generation
2025-07-03 04:54:37JaeHyuck Choi, MinJun Kim, JeHyeong HongAbstract
Few-shot anomaly generation is emerging as a practical solution for augmenting the scarce anomaly data in industrial quality control settings. An ideal generator would meet three demands at once, namely (i) keep the normal background intact, (ii) inpaint anomalous regions to tightly overlap with the corresponding anomaly masks, and (iii) generate anomalous regions in a semantically valid location, while still producing realistic, diverse appearances from only a handful of real examples. Existing diffusion-based methods usually satisfy at most two of these requirements: global anomaly generators corrupt the background, whereas mask-guided ones often falter when the mask is imprecise or misplaced. We propose MAGIC--Mask-guided inpainting with multi-level perturbations and Context-aware alignment--to resolve all three issues. At its core, MAGIC fine-tunes a Stable Diffusion inpainting backbone that preserves normal regions and ensures strict adherence of the synthesized anomaly to the supplied mask, directly addressing background corruption and misalignment. To offset the diversity loss that fine-tuning can cause, MAGIC adds two complementary perturbation strategies: (i) Gaussian prompt-level perturbation applied during fine-tuning and inference that broadens the global appearance of anomalies while avoiding low-fidelity textual appearances, and (ii) mask-guided spatial noise injection that enriches local texture variations. Additionally, the context-aware mask alignment module forms semantic correspondences and relocates masks so that every anomaly remains plausibly contained within the host object, eliminating out-of-boundary artifacts. Under a consistent identical evaluation protocol on the MVTec-AD dataset, MAGIC outperforms previous state-of-the-arts in downstream anomaly tasks.
Abstract (translated)
少量样本异常生成正逐渐成为解决工业质量控制环境中稀缺异常数据问题的一种实用解决方案。理想中的生成器需要同时满足以下三个需求：（i）保持正常的背景不变，（ii）在相应的异常掩码上精确绘制异常区域，以及（iii）在语义有效的位置生成异常区域，并且仅从少量的真实样本中就能产生逼真且多样化的外观。 现有的基于扩散的方法通常最多只能满足上述两个要求：全局异常生成器会破坏背景，而掩模引导的生成方法往往在掩模不精确或定位错误时表现不佳。我们提出了一种名为MAGIC（Mask-guided Inpainting with Multi-level Perturbations and Context-aware Alignment）的技术来解决所有这些问题。MAGIC的核心在于对Stable Diffusion修复模型进行微调，以保留正常区域，并确保合成的异常严格遵循提供的掩模，从而直接解决了背景破坏和不对齐的问题。 为了抵消微调可能带来的多样性损失，MAGIC增加了两种互补的扰动策略：（i）在微调和推理过程中应用高斯提示级扰动，这可以扩展全局异常外观的同时避免低质量的文字表现；以及（ii）掩模引导的空间噪声注入，以丰富局部纹理变化。此外，上下文感知的掩模对齐模块建立了语义对应关系并重新定位掩模，确保每个异常都合理地包含在宿主对象内，从而消除了边界外的伪影。 在MVTec-AD数据集上进行一致且相同的评估协议下，MAGIC在下游异常检测任务中超越了之前的最先进方法。
URL
https://arxiv.org/abs/2507.02314
PDF
-
DiGA3D: Coarse-to-Fine Diffusional Propagation of Geometry and Appearance for Versatile 3D Inpainting
2025-07-01 04:57:08Jingyi Pan, Dan Xu, Qiong LuoAbstract
Developing a unified pipeline that enables users to remove, re-texture, or replace objects in a versatile manner is crucial for text-guided 3D inpainting. However, there are still challenges in performing multiple 3D inpainting tasks within a unified framework: 1) Single reference inpainting methods lack robustness when dealing with views that are far from the reference view. 2) Appearance inconsistency arises when independently inpainting multi-view images with 2D diffusion priors; 3) Geometry inconsistency limits performance when there are significant geometric changes in the inpainting regions. To tackle these challenges, we introduce DiGA3D, a novel and versatile 3D inpainting pipeline that leverages diffusion models to propagate consistent appearance and geometry in a coarse-to-fine manner. First, DiGA3D develops a robust strategy for selecting multiple reference views to reduce errors during propagation. Next, DiGA3D designs an Attention Feature Propagation (AFP) mechanism that propagates attention features from the selected reference views to other views via diffusion models to maintain appearance consistency. Furthermore, DiGA3D introduces a Texture-Geometry Score Distillation Sampling (TG-SDS) loss to further improve the geometric consistency of inpainted 3D scenes. Extensive experiments on multiple 3D inpainting tasks demonstrate the effectiveness of our method. The project page is available at this https URL.
Abstract (translated)
开发一个统一的管道，使用户能够以灵活的方式移除、重新着色或替换对象，对于文本引导的3D修复（inpainting）至关重要。然而，在单一框架内执行多个3D修复任务仍然存在挑战： 1) 单一参考修复方法在处理远离参考视图的视角时缺乏鲁棒性。 2) 独立地使用2D扩散先验修复多视图图像会导致外观不一致。 3) 几何变化显著的情况下，几何一致性限制了修复区域的表现。 为了应对这些挑战，我们引入了DiGA3D，这是一种新颖且多功能的3D修复管道，它利用扩散模型以粗到细的方式传播一致的外观和几何结构。首先，DiGA3D开发了一种稳健的方法来选择多个参考视图，从而减少在传播过程中产生的误差。其次，DiGA3D设计了一个注意力特征传播（AFP）机制，该机制通过扩散模型将选定参考视图的注意特征传递到其他视图中以保持外观一致性。此外，DiGA3D还引入了纹理-几何评分蒸馏采样（TG-SDS）损失函数，进一步提高了修复后的3D场景中的几何一致性。 在多个3D修复任务上的广泛实验表明了我们方法的有效性。该项目页面可在此[URL]访问。请注意，在实际引用时，请确保使用正确的项目网址链接，因为上述提示中提供的“https URL”需要被具体的网页地址所替换。
URL
https://arxiv.org/abs/2507.00429
PDF
-
Can We Challenge Open-Vocabulary Object Detectors with Generated Content in Street Scenes?
2025-06-30 11:48:44Annika M\"utze, Sadia Ilyas, Christian D\"orpelkus, Matthias RottmannAbstract
Open-vocabulary object detectors such as Grounding DINO are trained on vast and diverse data, achieving remarkable performance on challenging datasets. Due to that, it is unclear where to find their limitations, which is of major concern when using in safety-critical applications. Real-world data does not provide sufficient control, required for a rigorous evaluation of model generalization. In contrast, synthetically generated data allows to systematically explore the boundaries of model competence/generalization. In this work, we address two research questions: 1) Can we challenge open-vocabulary object detectors with generated image content? 2) Can we find systematic failure modes of those models? To address these questions, we design two automated pipelines using stable diffusion to inpaint unusual objects with high diversity in semantics, by sampling multiple substantives from WordNet and ChatGPT. On the synthetically generated data, we evaluate and compare multiple open-vocabulary object detectors as well as a classical object detector. The synthetic data is derived from two real-world datasets, namely LostAndFound, a challenging out-of-distribution (OOD) detection benchmark, and the NuImages dataset. Our results indicate that inpainting can challenge open-vocabulary object detectors in terms of overlooking objects. Additionally, we find a strong dependence of open-vocabulary models on object location, rather than on object semantics. This provides a systematic approach to challenge open-vocabulary models and gives valuable insights on how data could be acquired to effectively improve these models.
Abstract (translated)
开放式词汇对象检测器（如Grounding DINO）在海量且多样化的数据上进行训练，因此它们在具有挑战性的数据集上表现出色。然而，这种优秀的性能也使得很难找到这些模型的局限性，这在安全关键的应用中尤为重要。现实世界的数据无法提供严格的评估所需的那种控制，而合成生成的数据则允许系统地探索模型能力与泛化边界的极限。 在这项工作中，我们探讨了两个研究问题： 1) 我们能否用生成的内容挑战开放式词汇对象检测器？ 2) 我们能否找到这些模型的系统性失败模式？ 为了解决这些问题，我们设计了两条自动化流水线，使用Stable Diffusion来对合成图像中的异常物体进行多样化的语义填充。我们在两个真实世界的数据集（LostAndFound和NuImages）的基础上生成合成数据，并评估多个开放式词汇对象检测器以及经典的物体检测器。 我们的实验结果表明，修补技术可以挑战开放式词汇模型在忽略对象方面的能力。此外，我们发现这些开放式词汇模型对对象位置的依赖性远远大于语义上的依赖性。 这提供了一种系统化的方法来挑战开放词汇模型，并为如何获取有效的改进数据提供了宝贵的见解。
URL
https://arxiv.org/abs/2506.23751
PDF
-
JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching
2025-06-30 06:51:40Mingi Kwon, Joonghyuk Shin, Jaeseok Jung, Jaesik Park, Youngjung UhAbstract
The intrinsic link between facial motion and speech is often overlooked in generative modeling, where talking head synthesis and text-to-speech (TTS) are typically addressed as separate tasks. This paper introduces JAM-Flow, a unified framework to simultaneously synthesize and condition on both facial motion and speech. Our approach leverages flow matching and a novel Multi-Modal Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT and Audio-DiT modules. These are coupled via selective joint attention layers and incorporate key architectural choices, such as temporally aligned positional embeddings and localized joint attention masking, to enable effective cross-modal interaction while preserving modality-specific strengths. Trained with an inpainting-style objective, JAM-Flow supports a wide array of conditioning inputs-including text, reference audio, and reference motion-facilitating tasks such as synchronized talking head generation from text, audio-driven animation, and much more, within a single, coherent model. JAM-Flow significantly advances multi-modal generative modeling by providing a practical solution for holistic audio-visual synthesis. project page: this https URL
Abstract (translated)
在生成模型中，面部动作与语音之间的内在联系常常被忽视，通常会将说话头部合成（Talking Head Synthesis）和文本转语音（Text-to-Speech, TTS）视为两个独立的任务。本文介绍了一种名为JAM-Flow的统一框架，它可以同时综合并基于面部运动和语音进行条件设定。我们的方法利用了流匹配技术和一种新颖的多模态扩散变压器（Multi-Modal Diffusion Transformer, MM-DiT）架构，该架构结合了专门设计的动作-DiT和音频-DiT模块。这些模块通过选择性联合注意力层相互连接，并采用了关键的架构决策，如时间对齐的位置嵌入和局部联合注意力掩码，以实现有效的跨模态交互并保留特定模式的优势。 JAM-Flow采用一种补全风格的目标进行训练，支持多种条件输入形式——包括文本、参考音频以及参考动作——这使它能够在一个单一的连贯模型中处理诸如从文本生成同步说话头部、由音频驱动的动画等任务，并且还有更多可能的应用场景。通过为整体音视频合成提供实用解决方案，JAM-Flow显著推进了多模态生成建模技术的发展。 项目主页：[此 URL](https://this-url.com/)（请将链接替换为您提供的实际网址）
URL
https://arxiv.org/abs/2506.23552
PDF
-
MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting
2025-06-30 03:06:54Jun Huang, Ting Liu, Yihang Wu, Xiaochao Qu, Luoqi Liu, Xiaolin HuAbstract
Advancements in generative models have enabled image inpainting models to generate content within specific regions of an image based on provided prompts and masks. However, existing inpainting methods often suffer from problems such as semantic misalignment, structural distortion, and style inconsistency. In this work, we present MTADiffusion, a Mask-Text Alignment diffusion model designed for object inpainting. To enhance the semantic capabilities of the inpainting model, we introduce MTAPipeline, an automatic solution for annotating masks with detailed descriptions. Based on the MTAPipeline, we construct a new MTADataset comprising 5 million images and 25 million mask-text pairs. Furthermore, we propose a multi-task training strategy that integrates both inpainting and edge prediction tasks to improve structural stability. To promote style consistency, we present a novel inpainting style-consistency loss using a pre-trained VGG network and the Gram matrix. Comprehensive evaluations on BrushBench and EditBench demonstrate that MTADiffusion achieves state-of-the-art performance compared to other methods.
Abstract (translated)
生成模型的进步使图像修复（inpainting）模型能够在特定区域根据提供的提示和蒙版生成内容。然而，现有的修复方法常常面临语义错配、结构扭曲和风格不一致等问题。为此，我们提出了一种名为MTADiffusion的掩码-文本对齐扩散模型，专为对象修复设计。 为了增强修复模型的语义能力，我们引入了自动解决方案MTAPipeline，用于用详细的描述来标注蒙版。基于MTAPipeline，我们构建了一个新的MTADataset，包括500万张图像和2500万个掩码-文本对。 此外，我们提出了一种多任务训练策略，该策略结合修复和边缘预测任务以提高结构稳定性。为了促进风格一致性，我们提出了一种新颖的修复风格一致性损失方法，使用预训练的VGG网络和Gram矩阵来实现。 在BrushBench和EditBench上的全面评估表明，MTADiffusion相对于其他方法实现了最先进的性能。
URL
https://arxiv.org/abs/2506.23482
PDF
-
Time-variant Image Inpainting via Interactive Distribution Transition Estimation
2025-06-30 01:45:33Yun Xing, Qing Guo, Xiaoguang Li, Yihao Huang, Xiaofeng Cao, Di Lin, Ivor Tsang, Lei MaAbstract
In this work, we focus on a novel and practical task, i.e., Time-vAriant iMage inPainting (TAMP). The aim of TAMP is to restore a damaged target image by leveraging the complementary information from a reference image, where both images captured the same scene but with a significant time gap in between, i.e., time-variant images. Different from conventional reference-guided image inpainting, the reference image under TAMP setup presents significant content distinction to the target image and potentially also suffers from damages. Such an application frequently happens in our daily lives to restore a damaged image by referring to another reference image, where there is no guarantee of the reference image's source and quality. In particular, our study finds that even state-of-the-art (SOTA) reference-guided image inpainting methods fail to achieve plausible results due to the chaotic image complementation. To address such an ill-posed problem, we propose a novel Interactive Distribution Transition Estimation (InDiTE) module which interactively complements the time-variant images with adaptive semantics thus facilitate the restoration of damaged regions. To further boost the performance, we propose our TAMP solution, namely Interactive Distribution Transition Estimation-driven Diffusion (InDiTE-Diff), which integrates InDiTE with SOTA diffusion model and conducts latent cross-reference during sampling. Moreover, considering the lack of benchmarks for TAMP task, we newly assembled a dataset, i.e., TAMP-Street, based on existing image and mask datasets. We conduct experiments on the TAMP-Street datasets under two different time-variant image inpainting settings, which show our method consistently outperform SOTA reference-guided image inpainting methods for solving TAMP.
Abstract (translated)
在这项工作中，我们专注于一个新颖且实用的任务，即时变图像修补（Time-vAriant iMage inPainting，TAMP）。TAMP的目标是利用参考图像中的互补信息来修复受损的目标图像，其中两幅图像是在不同时间点拍摄的同一场景，具有显著的时间差。与传统的基于参考引导的图像修补任务不同，在TAMP设定中，参考图像与目标图像之间存在显著的内容差异，并且可能也遭受了损坏。这种应用经常发生在我们的日常生活中，通过引用另一张参考图像来修复受损图像时会出现这种情况，而此时无法保证参考图的质量和来源。 特别地，我们发现即使是当前最先进的（SOTA）基于参考引导的图像修补方法，在面对混沌的图像补全时也无法达到令人信服的结果。为了解决这一病态问题，我们提出了一种新的交互式分布转换估计(InDiTE)模块，该模块能够适应性地补充具有时间变化性的图像，并利用其语义信息来促进受损区域的修复。为了进一步提高性能，我们提出了我们的TAMP解决方案——基于InDiTE驱动扩散模型的方法（InDiTE-Diff），它将InDiTE与最先进的扩散模型相结合，并在采样过程中进行潜在的跨参考操作。 此外，鉴于缺少针对TAMP任务的基准数据集，我们基于现有的图像和蒙版数据集新构建了一个名为TAMP-Street的数据集。我们在两个不同的时变图像修补设置下对TAMP-Street数据集进行了实验，结果表明我们的方法在解决TAMP问题方面始终优于当前最先进的基于参考引导的图像修补方法。
URL
https://arxiv.org/abs/2506.23461
PDF
-
Single Image Inpainting and Super-Resolution with Simultaneous Uncertainty Guarantees by Universal Reproducing Kernels
2025-06-29 13:12:23B\'alint Horv\'ath, Bal\'azs Csan\'ad Cs\'ajiAbstract
The paper proposes a statistical learning approach to the problem of estimating missing pixels of images, crucial for image inpainting and super-resolution problems. One of the main novelties of the method is that it also provides uncertainty quantifications together with the estimated values. Our core assumption is that the underlying data-generating function comes from a Reproducing Kernel Hilbert Space (RKHS). A special emphasis is put on band-limited functions, central to signal processing, which form Paley-Wiener type RKHSs. The proposed method, which we call Simultaneously Guaranteed Kernel Interpolation (SGKI), is an extension and refinement of a recently developed kernel method. An advantage of SGKI is that it not only estimates the missing pixels, but also builds non-asymptotic confidence bands for the unobserved values, which are simultaneously guaranteed for all missing pixels. We also show how to compute these bands efficiently using Schur complements, we discuss a generalization to vector-valued functions, and we present a series of numerical experiments on various datasets containing synthetically generated and benchmark images, as well.
Abstract (translated)
该论文提出了一种统计学习方法，用于解决图像中缺失像素的估计问题，这对于图像修复（inpainting）和超分辨率问题至关重要。这种方法的主要创新之一是它不仅提供了估计值，还给出了不确定性量化。我们假设底层的数据生成函数来自一个再生核希尔伯特空间（RKHS）。特别强调的是限带信号处理中的限带函数，它们构成了帕莱-维纳类型RKHS。 所提出的方法，即同时保证的核插值法（Simultaneously Guaranteed Kernel Interpolation, SGKI），是对最近开发出的一种核方法的扩展和改进。SGKI的一个优点是它不仅能估计缺失像素，还能为未观测到的所有像素构建非渐近置信区间，并且这些置信区间对所有缺失像素同时有效。我们还展示了如何使用舒尔补来高效地计算这些区间，并讨论了向向量值函数的一般化方法。此外，我们通过一系列在包含合成数据和基准图像的多种数据集上的数值实验来验证该方法的有效性。
URL
https://arxiv.org/abs/2506.23221
PDF
-
Inpainting is All You Need: A Diffusion-based Augmentation Method for Semi-supervised Medical Image Segmentation
2025-06-28 23:44:18Xinrong Hu, Yiyu ShiAbstract
Collecting pixel-level labels for medical datasets can be a laborious and expensive process, and enhancing segmentation performance with a scarcity of labeled data is a crucial challenge. This work introduces AugPaint, a data augmentation framework that utilizes inpainting to generate image-label pairs from limited labeled data. AugPaint leverages latent diffusion models, known for their ability to generate high-quality in-domain images with low overhead, and adapts the sampling process for the inpainting task without need for retraining. Specifically, given a pair of image and label mask, we crop the area labeled with the foreground and condition on it during reversed denoising process for every noise level. Masked background area would gradually be filled in, and all generated images are paired with the label mask. This approach ensures the accuracy of match between synthetic images and label masks, setting it apart from existing dataset generation methods. The generated images serve as valuable supervision for training downstream segmentation models, effectively addressing the challenge of limited annotations. We conducted extensive evaluations of our data augmentation method on four public medical image segmentation datasets, including CT, MRI, and skin imaging. Results across all datasets demonstrate that AugPaint outperforms state-of-the-art label-efficient methodologies, significantly improving segmentation performance.
Abstract (translated)
为医疗数据集收集像素级别的标签可能是一个繁琐且昂贵的过程，而在标记数据稀缺的情况下提高分割性能是一项关键挑战。这项工作引入了一种名为AugPaint的数据增强框架，该框架利用图像修复技术从有限的标注数据中生成图像-标签对。AugPaint采用潜在扩散模型，这种模型因其能够以低计算开销生成高质量、领域内的图像而著称，并且无需重新训练即可适应图像修复任务中的采样过程。 具体而言，给定一个包含图像和标签掩码的数据对，我们裁剪出被标记为前景的区域，并在反向去噪过程中针对每个噪声水平进行条件化处理。背景遮罩下的区域将逐渐被填充完成，所有生成的图像都会与标签掩码配对。这种做法确保了合成图像与标签掩模之间匹配的准确性，从而区别于现有的数据集生成方法。 通过这种方式生成的图像可以作为有价值的监督信号用于下游分割模型的训练，有效地解决了标注不足的问题。我们在四个公开的医疗图像分割数据集中进行了广泛的数据增强方法评估，包括CT、MRI和皮肤成像。在所有数据集上的结果表明，AugPaint优于当前最先进的标记效率方法，在所有情况下都显著提升了分割性能。
URL
https://arxiv.org/abs/2506.23038
PDF
-
Low-Rank Implicit Neural Representation via Schatten-p Quasi-Norm and Jacobian Regularization
2025-06-27 11:23:10Zhengyun Cheng, Changhao Wang, Guanwen Zhang, Yi Xu, Wei Zhou, Xiangyang JiAbstract
Higher-order tensors are well-suited for representing multi-dimensional data, such as color images and videos. Low-rank tensor representation has become essential in machine learning and computer vision, but existing methods like Tucker decomposition offer flexibility at the expense of interpretability. In contrast, while the CANDECOMP/PARAFAC (CP) decomposition provides a more natural and interpretable tensor structure, obtaining sparse solutions remains challenging. Leveraging the rich properties of CP decomposition, we propose a CP-based low-rank tensor function parameterized by neural networks for implicit neural representation (CP-INR). This approach enables continuous data representation beyond structured grids, fully exploiting the non-linearity of tensor data with theoretical guarantees on excess risk bounds. To achieve a sparse CP decomposition, we introduce a variational form of the Schatten-p quasi-norm and prove its relationship to multilinear rank minimization. For smoothness, we propose a regularization term based on the spectral norm of the Jacobian and Hutchinson's trace estimator. Our proposed smoothness regularization is SVD-free and avoids explicit chain rule derivations. It can serve as an alternative to Total Variation (TV) regularization in image denoising tasks and is naturally applicable to continuous data. Extensive experiments on multi-dimensional data recovery tasks, including image inpainting, denoising, and point cloud upsampling, demonstrate the superiority and versatility of our method compared to state-of-the-art approaches.
Abstract (translated)
高阶张量非常适合表示多维数据，如彩色图像和视频。低秩张量表示在机器学习和计算机视觉中变得至关重要，但现有的方法如Tucker分解以牺牲可解释性为代价提供了灵活性。相比之下，尽管CANDECOMP/PARAFAC（CP）分解提供了一种更加自然且易于解释的张量结构，但在获得稀疏解方面仍存在挑战。利用CP分解丰富的特性，我们提出了一种基于CP分解并由神经网络参数化的低秩张量函数，用于隐式神经表示（CP-INR）。这种方法使连续数据表示超越了有结构的网格，并充分利用张量数据的非线性，同时在理论上保证了超额风险界限。为了实现稀疏CP分解，我们引入了Schatten-p准范数的一种变分形式，并证明它与多线性秩最小化之间的关系。对于平滑度，我们提出了一种基于雅可比矩阵谱范数和Hutchinson迹估计器的正则化项。我们的提出的平滑正则化方法无需SVD计算且避免了显式的链式法则导数推导，可以作为图像去噪任务中总变差（TV）正则化的替代方案，并自然适用于连续数据。在多维数据恢复任务上的广泛实验，包括图像修复、去噪和点云上采样，展示了我们方法相对于最先进的方法的优越性和灵活性。
URL
https://arxiv.org/abs/2506.22134
PDF
-
PrefPaint: Enhancing Image Inpainting through Expert Human Feedback
2025-06-27 00:47:07Duy-Bao Bui, Hoang-Khang Nguyen, Trung-Nghia LeAbstract
Inpainting, the process of filling missing or corrupted image parts, has broad applications, including medical imaging. However, in specialized fields like medical polyps imaging, where accuracy and reliability are critical, inpainting models can generate inaccurate images, leading to significant errors in medical diagnosis and treatment. To ensure reliability, medical images should be annotated by experts like oncologists for effective model training. We propose PrefPaint, an approach that incorporates human feedback into the training process of Stable Diffusion Inpainting, bypassing the need for computationally expensive reward models. In addition, we develop a web-based interface streamlines training, fine-tuning, and inference. This interactive interface provides a smooth and intuitive user experience, making it easier to offer feedback and manage the fine-tuning process. User study on various domains shows that PrefPaint outperforms existing methods, reducing visual inconsistencies and improving image rendering, particularly in medical contexts, where our model generates more realistic polyps images.
Abstract (translated)
图像修复（Inpainting），即填充缺失或损坏的图像部分，具有广泛的应用领域，包括医学成像。然而，在如医疗息肉成像这样对准确性和可靠性要求极高的专业领域中，图像修复模型可能会生成不准确的图像，这会导致严重的误诊和治疗错误。为了确保医学图像的可靠性和准确性，应该由放射科医生等专家对其进行标注，以进行有效的模型训练。 我们提出了一种名为PrefPaint的方法，该方法将人类反馈整合到Stable Diffusion Inpainting（稳定扩散修复）模型的训练过程中，并且无需使用计算成本高昂的奖励模型。此外，我们还开发了一个基于网络的界面，用于简化训练、微调和推理过程。这个交互式界面提供了流畅直观的用户体验，使得提供反馈和管理微调过程变得更加容易。 在多个领域的用户研究中显示，PrefPaint方法超越了现有的技术，在减少视觉不一致性和提高图像渲染质量方面表现出色，特别是在医学背景下，我们的模型能够生成更加逼真的息肉图像。
URL
https://arxiv.org/abs/2506.21834
PDF
-
DeOcc-1-to-3: 3D De-Occlusion from a Single Image via Self-Supervised Multi-View Diffusion
2025-06-26 17:58:26Yansong Qu, Shaohui Dai, Xinyang Li, Yuze Wang, You Shen, Liujuan Cao, Rongrong JiAbstract
Reconstructing 3D objects from a single image is a long-standing challenge, especially under real-world occlusions. While recent diffusion-based view synthesis models can generate consistent novel views from a single RGB image, they generally assume fully visible inputs and fail when parts of the object are occluded. This leads to inconsistent views and degraded 3D reconstruction quality. To overcome this limitation, we propose an end-to-end framework for occlusion-aware multi-view generation. Our method directly synthesizes six structurally consistent novel views from a single partially occluded image, enabling downstream 3D reconstruction without requiring prior inpainting or manual annotations. We construct a self-supervised training pipeline using the Pix2Gestalt dataset, leveraging occluded-unoccluded image pairs and pseudo-ground-truth views to teach the model structure-aware completion and view consistency. Without modifying the original architecture, we fully fine-tune the view synthesis model to jointly learn completion and multi-view generation. Additionally, we introduce the first benchmark for occlusion-aware reconstruction, encompassing diverse occlusion levels, object categories, and mask patterns. This benchmark provides a standardized protocol for evaluating future methods under partial occlusions. Our code is available at this https URL.
Abstract (translated)
从单张图像重建3D物体长期以来一直是一个挑战，尤其是在现实世界的遮挡情况下。虽然最近基于扩散的视图合成模型能够从单一RGB图像生成一致的新视角，但它们通常假设输入是完全可见的，并且在对象的部分被遮挡时会失效。这会导致不一致的视图和降低的3D重建质量。为了解决这一限制，我们提出了一种端到端框架，用于感知遮挡的多视角生成。我们的方法直接从单张部分被遮挡的图像中合成六个结构上一致的新视角，从而在不需要预先进行图像修复或手动注释的情况下支持下游的3D重建工作。我们使用Pix2Gestalt数据集构建了一个自我监督训练管道，利用有无遮挡的图像对和伪真实视图来教导模型感知结构完成及视图一致性。在不修改原始架构的情况下，我们将视角合成模型完全微调以同时学习补全和多视角生成任务。 此外，我们还推出了首个用于感知遮挡重建的基准测试，涵盖了多样化的遮挡级别、物体类别以及掩码模式。这一基准为评估未来方法在部分被遮挡情况下的表现提供了标准化协议。我们的代码可在此链接获取（原链接已提供）。
URL
https://arxiv.org/abs/2506.21544
PDF
