Inpainting
-
Improved 3D Scene Stylization via Text-Guided Generative Image Editing with Region-Based Control
2025-09-04 15:01:01Haruo Fujiwara, Yusuke Mukuta, Tatsuya HaradaAbstract
Recent advances in text-driven 3D scene editing and stylization, which leverage the powerful capabilities of 2D generative models, have demonstrated promising outcomes. However, challenges remain in ensuring high-quality stylization and view consistency simultaneously. Moreover, applying style consistently to different regions or objects in the scene with semantic correspondence is a challenging task. To address these limitations, we introduce techniques that enhance the quality of 3D stylization while maintaining view consistency and providing optional region-controlled style transfer. Our method achieves stylization by re-training an initial 3D representation using stylized multi-view 2D images of the source views. Therefore, ensuring both style consistency and view consistency of stylized multi-view images is crucial. We achieve this by extending the style-aligned depth-conditioned view generation framework, replacing the fully shared attention mechanism with a single reference-based attention-sharing mechanism, which effectively aligns style across different viewpoints. Additionally, inspired by recent 3D inpainting methods, we utilize a grid of multiple depth maps as a single-image reference to further strengthen view consistency among stylized images. Finally, we propose Multi-Region Importance-Weighted Sliced Wasserstein Distance Loss, allowing styles to be applied to distinct image regions using segmentation masks from off-the-shelf models. We demonstrate that this optional feature enhances the faithfulness of style transfer and enables the mixing of different styles across distinct regions of the scene. Experimental evaluations, both qualitative and quantitative, demonstrate that our pipeline effectively improves the results of text-driven 3D stylization.
Abstract (translated)
近期在文本驱动的三维场景编辑和风格化方面取得的进步，利用了二维生成模型的强大能力，并展示了有希望的结果。然而，在确保高质量的风格化的同时保持视角一致性仍然存在挑战。此外，将样式一致地应用于具有语义对应的不同区域或物体是一项艰巨的任务。为了解决这些限制，我们引入了一系列技术来增强3D风格化的质量，同时保持视角一致性并提供可选的区域控制式样传输。我们的方法通过使用源视图的风格化多视角2D图像重新训练初始3D表示来实现样式化。因此，确保风格化多视角图像的风格一致性和视角一致性至关重要。 为了达到这一目标，我们扩展了与风格对齐的深度条件视角生成框架，并用单一参考基础注意力共享机制替换了完全共享注意力机制，这有效地将不同视点上的风格进行了对齐。此外，受最近的3D修复方法启发，我们利用多个深度图网格作为单张图像的参考，进一步加强了风格化图像之间的视角一致性。 最后，我们提出了一种多区域重要性加权切片Wasserstein距离损失，通过使用现成模型生成的分割掩码，允许不同的样式应用于独特的图像区域。证明这一可选特性增强了样式的传输保真度，并使不同区域混合不同样式成为可能。实验评估（定性和定量）表明，我们的流程有效地提高了文本驱动3D风格化的结果。 翻译总结了原文献的主要贡献和创新点，展示了如何通过一系列技术改进来解决当前三维场景编辑和风格化中的主要挑战，从而提升图像的质量、一致性和灵活性。
URL
https://arxiv.org/abs/2509.05285
PDF
-
2D Gaussian Splatting with Semantic Alignment for Image Inpainting
2025-09-02 05:12:52Hongyu Li, Chaofeng Chen, Xiaoming Li, Guangming LuAbstract
Gaussian Splatting (GS), a recent technique for converting discrete points into continuous spatial representations, has shown promising results in 3D scene modeling and 2D image super-resolution. In this paper, we explore its untapped potential for image inpainting, which demands both locally coherent pixel synthesis and globally consistent semantic restoration. We propose the first image inpainting framework based on 2D Gaussian Splatting, which encodes incomplete images into a continuous field of 2D Gaussian splat coefficients and reconstructs the final image via a differentiable rasterization process. The continuous rendering paradigm of GS inherently promotes pixel-level coherence in the inpainted results. To improve efficiency and scalability, we introduce a patch-wise rasterization strategy that reduces memory overhead and accelerates inference. For global semantic consistency, we incorporate features from a pretrained DINO model. We observe that DINO's global features are naturally robust to small missing regions and can be effectively adapted to guide semantic alignment in large-mask scenarios, ensuring that the inpainted content remains contextually consistent with the surrounding scene. Extensive experiments on standard benchmarks demonstrate that our method achieves competitive performance in both quantitative metrics and perceptual quality, establishing a new direction for applying Gaussian Splatting to 2D image processing.
Abstract (translated)
高斯点绘（GS）是一种最近用于将离散点转换为连续空间表示的技术，在3D场景建模和2D图像超分辨率方面已显示出令人鼓舞的结果。在这篇论文中，我们探索了它在图像修复中的未开发潜力，该领域需要局部连贯的像素合成以及全局一致的语义恢复。我们提出了首个基于二维高斯点绘的图像修补框架，将不完整的图像编码为2D高斯点绘系数的连续场，并通过可微渲染过程重构最终图像。GS的连续渲染范式内在地促进了修复结果中的像素级连贯性。 为了提高效率和扩展性，我们引入了一种基于块的渲染策略，减少了内存占用并加速了推断速度。对于全局语义一致性，我们结合了一个预训练的DINO模型的功能。我们观察到，DINO的全局特征对小缺失区域具有自然鲁棒性，并且可以有效地适应以指导大掩码场景中的语义对齐，确保修复内容与周围场景保持上下文一致。 在标准基准上的广泛实验表明，我们的方法在定量指标和感知质量方面均实现了竞争性的性能，为应用高斯点绘技术于二维图像处理开辟了新的方向。
URL
https://arxiv.org/abs/2509.01964
PDF
-
Neural Scene Designer: Self-Styled Semantic Image Manipulation
2025-09-01 11:59:03Jianman Lin, Tianshui Chen, Chunmei Qing, Zhijing Yang, Shuangping Huang, Yuheng Ren, Liang LinAbstract
Maintaining stylistic consistency is crucial for the cohesion and aesthetic appeal of images, a fundamental requirement in effective image editing and inpainting. However, existing methods primarily focus on the semantic control of generated content, often neglecting the critical task of preserving this consistency. In this work, we introduce the Neural Scene Designer (NSD), a novel framework that enables photo-realistic manipulation of user-specified scene regions while ensuring both semantic alignment with user intent and stylistic consistency with the surrounding environment. NSD leverages an advanced diffusion model, incorporating two parallel cross-attention mechanisms that separately process text and style information to achieve the dual objectives of semantic control and style consistency. To capture fine-grained style representations, we propose the Progressive Self-style Representational Learning (PSRL) module. This module is predicated on the intuitive premise that different regions within a single image share a consistent style, whereas regions from different images exhibit distinct styles. The PSRL module employs a style contrastive loss that encourages high similarity between representations from the same image while enforcing dissimilarity between those from different images. Furthermore, to address the lack of standardized evaluation protocols for this task, we establish a comprehensive benchmark. This benchmark includes competing algorithms, dedicated style-related metrics, and diverse datasets and settings to facilitate fair comparisons. Extensive experiments conducted on our benchmark demonstrate the effectiveness of the proposed framework.
Abstract (translated)
保持风格一致性对于图像的连贯性和审美吸引力至关重要，这是有效图像编辑和修复的基本要求。然而，现有方法主要集中在生成内容的语义控制上，常常忽略了维持这种一致性的关键任务。在这项工作中，我们引入了神经场景设计器（NSD），这是一种新颖的框架，它能够实现用户指定场景区域的真实感操作，同时确保与用户意图的语义对齐和周围环境的一致风格。NSD利用了一个先进的扩散模型，并采用了两个并行的交叉注意力机制来分别处理文本信息和样式信息，从而实现了语义控制和风格一致性这两大目标。 为了捕捉精细的风格表示，我们提出了渐进式自风格表征学习（PSRL）模块。该模块基于这样一个直观的前提：单个图像内的不同区域具有相同的风格，而来自不同图像的不同区域则具有不同的风格。PSRL模块采用了样式对比损失来鼓励同一图像中的表示相似性，并确保不同图像之间的表示差异。 此外，为了应对这一任务缺乏标准化评估协议的问题，我们建立了一个全面的基准测试。该基准测试包括竞争算法、专门针对样式的相关度量指标以及多样化的数据集和设置，以促进公平比较。我们在我们的基准上进行了广泛的实验，证明了所提出的框架的有效性。
URL
https://arxiv.org/abs/2509.01405
PDF
-
DAOVI: Distortion-Aware Omnidirectional Video Inpainting
2025-08-30 07:27:52Ryosuke Seshimo, Mariko IsogawaAbstract
Omnidirectional videos that capture the entire surroundings are employed in a variety of fields such as VR applications and remote sensing. However, their wide field of view often causes unwanted objects to appear in the videos. This problem can be addressed by video inpainting, which enables the natural removal of such objects while preserving both spatial and temporal consistency. Nevertheless, most existing methods assume processing ordinary videos with a narrow field of view and do not tackle the distortion in equirectangular projection of omnidirectional videos. To address this issue, this paper proposes a novel deep learning model for omnidirectional video inpainting, called Distortion-Aware Omnidirectional Video Inpainting (DAOVI). DAOVI introduces a module that evaluates temporal motion information in the image space considering geodesic distance, as well as a depth-aware feature propagation module in the feature space that is designed to address the geometric distortion inherent to omnidirectional videos. The experimental results demonstrate that our proposed method outperforms existing methods both quantitatively and qualitatively.
Abstract (translated)
全向视频捕获周围环境的整个视场，在虚拟现实应用和遥感等众多领域得到了广泛运用。然而，其广阔的视野往往导致视频中出现不必要的对象。这一问题可以通过视频修复技术（video inpainting）来解决，这种方法可以在保持空间和时间一致性的同时自然地移除这些对象。不过，大多数现有的方法都假定处理的是普通且视场较窄的视频，并未针对全向视频在等距矩形投影中的几何失真进行优化。 为了解决这一问题，本文提出了一种新的深度学习模型，称为“感知畸变的全向视频修复”（Distortion-Aware Omnidirectional Video Inpainting, DAOVI）。DAOVI引入了一个评估图像空间中时间运动信息的模块，并考虑了测地距离。此外，它还设计了一个深度感知特征传播模块，在特征空间内解决全向视频固有的几何失真问题。 实验结果表明，所提出的方法在量化和定性测试上均优于现有方法。
URL
https://arxiv.org/abs/2509.00396
PDF
-
Neural Field Turing Machine: A Differentiable Spatial Computer
Abstract
We introduce the Neural Field Turing Machine (NFTM), a differentiable architecture that unifies symbolic computation, physical simulation, and perceptual inference within continuous spatial fields. NFTM combines a neural controller, continuous memory field, and movable read/write heads that perform local updates. At each timestep, the controller reads local patches, computes updates via learned rules, and writes them back while updating head positions. This design achieves linear O(N) scaling through fixed-radius neighborhoods while maintaining Turing completeness under bounded error. We demonstrate three example instantiations of NFTM: cellular automata simulation (Rule 110), physics-informed PDE solvers (2D heat equation), and iterative image refinement (CIFAR-10 inpainting). These instantiations learn local update rules that compose into global dynamics, exhibit stable long-horizon rollouts, and generalize beyond training horizons. NFTM provides a unified computational substrate bridging discrete algorithms and continuous field dynamics within a single differentiable framework.
Abstract (translated)
我们介绍了一种新的架构——神经场图灵机（Neural Field Turing Machine，NFTM），这是一种可微分的架构，它将符号计算、物理仿真和感知推理统一在连续的空间场中。NFTM结合了神经控制器、连续记忆场以及能够执行局部更新并可在空间上移动的读写头。在每个时间步长内，控制器会读取局部区域的信息，通过学习到的规则进行计算，并将结果写回到记忆场的同时调整读写头的位置。这种设计实现了线性 O(N) 的扩展能力，在保持图灵完备性的前提下使用了固定半径的邻域范围。 我们演示了 NFTM 的三种具体实现：细胞自动机模拟（Rule 110）、物理信息偏微分方程求解器（二维热传导方程）以及迭代图像细化（CIFAR-10 图像修复）。这些实例展示了通过学习局部更新规则，如何组合成全局动态过程，并且在长时间尺度内保持稳定性和泛化能力，超越了训练时间范围。 NFTM 提供了一种统一的计算框架，在单一可微分架构中将离散算法和连续场动力学连接起来。
URL
https://arxiv.org/abs/2509.03370
PDF
-
ROSE: Remove Objects with Side Effects in Videos
2025-08-26 03:18:31Chenxuan Miao, Yutong Feng, Jianshu Zeng, Zixiang Gao, Hantang Liu, Yunfeng Yan, Donglian Qi, Xi Chen, Bin Wang, Hengshuang ZhaoAbstract
Video object removal has achieved advanced performance due to the recent success of video generative models. However, when addressing the side effects of objects, e.g., their shadows and reflections, existing works struggle to eliminate these effects for the scarcity of paired video data as supervision. This paper presents ROSE, termed Remove Objects with Side Effects, a framework that systematically studies the object's effects on environment, which can be categorized into five common cases: shadows, reflections, light, translucency and mirror. Given the challenges of curating paired videos exhibiting the aforementioned effects, we leverage a 3D rendering engine for synthetic data generation. We carefully construct a fully-automatic pipeline for data preparation, which simulates a large-scale paired dataset with diverse scenes, objects, shooting angles, and camera trajectories. ROSE is implemented as an video inpainting model built on diffusion transformer. To localize all object-correlated areas, the entire video is fed into the model for reference-based erasing. Moreover, additional supervision is introduced to explicitly predict the areas affected by side effects, which can be revealed through the differential mask between the paired videos. To fully investigate the model performance on various side effect removal, we presents a new benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five special side effects for comprehensive evaluation. Experimental results demonstrate that ROSE achieves superior performance compared to existing video object erasing models and generalizes well to real-world video scenarios. The project page is this https URL.
Abstract (translated)
视频对象移除技术由于最近视频生成模型的成功而取得了先进的性能。然而，当处理物体的副作用（如阴影和反射）时，现有方法因缺乏成对视频数据作为监督信号而难以消除这些效果。本文介绍了ROSE（Remove Objects with Side Effects），这是一个旨在系统研究环境中物体会影响的框架，并将这些影响分为五种常见情况：阴影、反射、光线变化、透明度以及镜子中的映像。 鉴于收集展示上述效应的成对视频面临的挑战，我们利用3D渲染引擎生成合成数据。我们精心构建了一个全自动的数据准备管道，该管道模拟了涵盖各种场景、物体、拍摄角度和摄像机轨迹的大规模成对数据集。ROSE作为一个基于扩散变换器的视频修复模型来实现，在这种框架下，整个视频被输入到模型中进行参考基线擦除操作，以便定位所有与对象相关的区域。此外，引入了额外的监督机制以明确预测受副作用影响的区域，这些区域可以通过成对视频之间的差异掩码揭示出来。 为了全面评估模型在各种副作用移除方面的性能，我们提出了一项新的基准测试，名为ROSE-Bench，该基准集成了常见场景以及五种特殊副作用进行综合评价。实验结果表明，ROSE与现有的视频对象擦除模型相比，在效果上更胜一筹，并且能很好地应用于现实世界的视频情况中。 项目页面在此链接：https://这个URL应该被替换为实际的网址或者具体描述，因为你提供的信息没有包含具体的网址或项目的公开访问方式。
URL
https://arxiv.org/abs/2508.18633
PDF
-
ObjFiller-3D: Consistent Multi-view 3D Inpainting via Video Diffusion Models
2025-08-25 17:59:40Haitang Feng, Jie Liu, Jie Tang, Gangshan Wu, Beiqi Chen, Jianhuang Lai, Guangcong WangAbstract
3D inpainting often relies on multi-view 2D image inpainting, where the inherent inconsistencies across different inpainted views can result in blurred textures, spatial discontinuities, and distracting visual artifacts. These inconsistencies pose significant challenges when striving for accurate and realistic 3D object completion, particularly in applications that demand high fidelity and structural coherence. To overcome these limitations, we propose ObjFiller-3D, a novel method designed for the completion and editing of high-quality and consistent 3D objects. Instead of employing a conventional 2D image inpainting model, our approach leverages a curated selection of state-of-the-art video editing model to fill in the masked regions of 3D objects. We analyze the representation gap between 3D and videos, and propose an adaptation of a video inpainting model for 3D scene inpainting. In addition, we introduce a reference-based 3D inpainting method to further enhance the quality of reconstruction. Experiments across diverse datasets show that compared to previous methods, ObjFiller-3D produces more faithful and fine-grained reconstructions (PSNR of 26.6 vs. NeRFiller (15.9) and LPIPS of 0.19 vs. Instant3dit (0.25)). Moreover, it demonstrates strong potential for practical deployment in real-world 3D editing applications. Project page: this https URL Code: this https URL .
Abstract (translated)
3D修复（即3D inpainting）通常依赖于多视角的二维图像修补技术，但在不同的修补视图之间可能存在固有的不一致性问题。这些问题会导致模糊纹理、空间不连续性以及视觉干扰等现象。当致力于实现准确且逼真的3D物体完整重建时，特别是在需要高保真度和结构一致性的应用场景中，这种不一致性带来的挑战尤为显著。 为了解决这些局限性，我们提出了一种名为ObjFiller-3D的新方法，旨在完成并编辑高质量且一致的3D对象。与传统2D图像修补模型不同，我们的方法采用了一系列先进的视频编辑模型来填充3D物体中的遮罩区域。通过分析3D和视频之间的表示差距，我们将一种视频修补模型改编用于3D场景修补，并引入了一种基于参考的3D修补方法以进一步提升重建质量。 实验结果显示，在多个数据集上与先前的方法相比，ObjFiller-3D能够生成更为忠实且细节丰富的重构结果（PSNR为26.6对比NeRFiller的15.9和LPIPS为0.19对比Instant3dit的0.25）。此外，它还展示了在现实世界的3D编辑应用中进行实用部署的巨大潜力。 项目页面：[此处插入URL] 代码链接：[此处插入URL]
URL
https://arxiv.org/abs/2508.18271
PDF
-
Diffusion-Based Data Augmentation for Medical Image Segmentation
2025-08-25 09:49:27Maham Nazir, Muhammad Aqeel, Francesco SettiAbstract
Medical image segmentation models struggle with rare abnormalities due to scarce annotated pathological data. We propose DiffAug a novel framework that combines textguided diffusion-based generation with automatic segmentation validation to address this challenge. Our proposed approach uses latent diffusion models conditioned on medical text descriptions and spatial masks to synthesize abnormalities via inpainting on normal images. Generated samples undergo dynamic quality validation through a latentspace segmentation network that ensures accurate localization while enabling single-step inference. The text prompts, derived from medical literature, guide the generation of diverse abnormality types without requiring manual annotation. Our validation mechanism filters synthetic samples based on spatial accuracy, maintaining quality while operating efficiently through direct latent estimation. Evaluated on three medical imaging benchmarks (CVC-ClinicDB, Kvasir-SEG, REFUGE2), our framework achieves state-of-the-art performance with 8-10% Dice improvements over baselines and reduces false negative rates by up to 28% for challenging cases like small polyps and flat lesions critical for early detection in screening applications.
Abstract (translated)
医学图像分割模型在处理罕见异常情况时，由于缺乏标注的病理数据而面临挑战。为此，我们提出了一种名为DiffAug的新框架，该框架结合了基于文本引导的扩散生成技术和自动分割验证技术来应对这一难题。我们的方法利用条件于医疗文本描述和空间掩码的潜在扩散模型，在正常图像上通过修补（inpainting）生成异常情况。生成的数据会经过一个在潜在空间中进行分割网络的动态质量验证过程，确保准确的定位同时支持一步推理操作。从医学文献中提取的文本提示引导生成不同类型且多样化的异常情况，无需手动注释。我们的验证机制根据空间准确性过滤合成样本，在保持高质量的同时通过直接潜在估计实现高效运行。 我们在三个医疗成像基准数据集（CVC-ClinicDB、Kvasir-SEG 和 REFUGE2）上评估了该框架，并取得了与现有方法相比在Dice系数上有8%-10%的性能提升，尤其是在小型息肉和扁平病变等挑战性案例中减少了高达28%的假阴性率。这些类型的异常对于筛查应用中的早期检测至关重要。
URL
https://arxiv.org/abs/2508.17844
PDF
-
PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting
2025-08-22 08:42:46Hohyun Na, Seunghoo Hong, Simon S. WooAbstract
The success of diffusion models has enabled effortless, high-quality image modifications that precisely align with users' intentions, thereby raising concerns about their potential misuse by malicious actors. Previous studies have attempted to mitigate such misuse through adversarial attacks. However, these approaches heavily rely on image-level inconsistencies, which pose fundamental limitations in addressing the influence of textual prompts. In this paper, we propose PromptFlare, a novel adversarial protection method designed to protect images from malicious modifications facilitated by diffusion-based inpainting models. Our approach leverages the cross-attention mechanism to exploit the intrinsic properties of prompt embeddings. Specifically, we identify and target shared token of prompts that is invariant and semantically uninformative, injecting adversarial noise to suppress the sampling process. The injected noise acts as a cross-attention decoy, diverting the model's focus away from meaningful prompt-image alignments and thereby neutralizing the effect of prompt. Extensive experiments on the EditBench dataset demonstrate that our method achieves state-of-the-art performance across various metrics while significantly reducing computational overhead and GPU memory usage. These findings highlight PromptFlare as a robust and efficient protection against unauthorized image manipulations. The code is available at this https URL.
Abstract (translated)
扩散模型的成功使得高质量的图像修改变得轻松且与用户意图高度一致，这引发了对这些技术可能被恶意行为者滥用的关注。以往的研究试图通过对抗攻击来缓解此类问题。然而，这些方法严重依赖于基于图像级别的不一致性，从而在解决文本提示影响方面存在根本性的局限性。为此，在本文中我们提出了 PromptFlare，这是一种新型的对抗防护方法，旨在保护图像免受扩散模型驱动的修补程序带来的恶意修改。 我们的方法利用了跨注意力机制来挖掘提示嵌入的基本属性。具体来说，我们将识别并针对那些不变且语义上无信息的共享提示词注入对抗性噪声，以抑制采样过程。这种注入的噪音作为跨注意力诱饵，使模型的关注点偏离有意义的提示-图像对齐，并从而削弱了提示的作用。 在 EditBench 数据集上的广泛实验表明，我们的方法在各种指标下取得了最先进的性能，同时大幅减少了计算开销和 GPU 内存使用量。这些发现突显了 PromptFlare 是一种针对未经授权的图像操纵的强大且高效的保护措施。代码可在此网址获得：[此URL链接]
URL
https://arxiv.org/abs/2508.16217
PDF
-
GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
2025-08-21 04:31:01Wen-Fan Wang, Ting-Ying Lee, Chien-Ting Lu, Che-Wei Hsu, Nil Ponsa Campany, Yu Chen, Mike Y. Chen, Bing-Yu ChenAbstract
Environment designers in the entertainment industry create imaginative 2D and 3D scenes for games, films, and television, requiring both fine-grained control of specific details and consistent global coherence. Designers have increasingly integrated generative AI into their workflows, often relying on large language models (LLMs) to expand user prompts for text-to-image generation, then iteratively refining those prompts and applying inpainting. However, our formative study with 10 designers surfaced two key challenges: (1) the lengthy LLM-generated prompts make it difficult to understand and isolate the keywords that must be revised for specific visual elements; and (2) while inpainting supports localized edits, it can struggle with global consistency and correctness. Based on these insights, we present GenTune, an approach that enhances human--AI collaboration by clarifying how AI-generated prompts map to image content. Our GenTune system lets designers select any element in a generated image, trace it back to the corresponding prompt labels, and revise those labels to guide precise yet globally consistent image refinement. In a summative study with 20 designers, GenTune significantly improved prompt--image comprehension, refinement quality, and efficiency, and overall satisfaction (all $p < .01$) compared to current practice. A follow-up field study with two studios further demonstrated its effectiveness in real-world settings.
Abstract (translated)
在娱乐行业中，环境设计师负责为游戏、电影和电视节目创造富有想象力的二维和三维场景。这项工作既需要对特定细节进行精细控制，又要求整体一致性。越来越多的设计人员将生成式人工智能（AI）整合到他们的工作流程中，通常利用大型语言模型（LLMs）来扩展用户指令以生成图像，并通过迭代优化这些提示并应用局部重绘技术（inpainting）。然而，在与10位设计师进行初步研究后发现两个主要挑战：一是由LLM生成的长指令使得难以理解及分离出必须修改特定视觉元素的关键字；二是虽然局部重绘支持对特定区域的编辑，但在保持全局一致性和准确性方面存在困难。 基于这些见解，我们提出了GenTune这一方法，旨在通过明确AI生成提示与图像内容之间的关联来增强人机协作。GenTune系统允许设计师选择生成图像中的任何元素，并将其追溯到相应的提示标签，随后修改这些标签以引导精确而全局一致的图像优化过程。在后续对20位设计师进行的总结性研究中，与现行做法相比，使用GenTune显著提高了指令-图像的理解程度、优化质量以及效率和总体满意度（所有p值均小于.01）。此外，在两个工作室进行的跟进实地研究进一步证明了该系统在实际环境中的有效性。
URL
https://arxiv.org/abs/2508.15227
PDF
-
A Real-Time Diminished Reality Approach to Privacy in MR Collaboration
2025-08-21 04:01:56Christian FaneAbstract
Diminished reality (DR) refers to the digital removal of real-world objects by compositing background content in their place. This thesis presents a real-time, inpainting-based DR system designed to enable privacy control in shared-space mixed reality (MR) meetings. The system allows a primary headset user to selectively remove personal or sensitive items from their environment, ensuring that those objects are no longer visible to other participants. Removal is achieved through semantic segmentation and precise object selection, followed by real-time inpainting from the viewpoint of a secondary observer, implemented using a mobile ZED 2i depth camera. The solution is designed to be portable and robust, requiring neither a fixed secondary viewpoint nor prior 3D scanning of the environment. The system utilises YOLOv11 for object detection and a modified Decoupled Spatial-Temporal Transformer (DSTT) model for high-quality video inpainting. At 720p resolution, the pipeline sustains frame rates exceeding 20 fps, demonstrating the feasibility of real-time diminished reality for practical privacy-preserving MR applications.
Abstract (translated)
虚化现实（DR）是指通过合成背景内容来实现从数字空间中移除真实世界物体的技术。本论文介绍了一种基于实时修复的虚化现实系统，旨在为共享空间中的混合现实（MR）会议提供隐私控制功能。该系统允许头戴设备的主要用户选择性地移除个人或敏感物品，确保其他参与者无法看到这些物品。 移除过程通过语义分割和精确的对象选择实现，并且在次要观察者的视角下进行实时修复，使用移动的ZED 2i深度相机来实施。此解决方案具有便携性和鲁棒性的特点，无需固定的次要视点或环境的先验三维扫描数据即可运行。 系统利用YOLOv11模型进行物体检测，并采用修改后的解耦空间-时间变换器（DSTT）模型来进行高质量视频修复。在720p分辨率下，该系统的处理流程能够保持每秒超过20帧的帧率，展示了实时虚化现实技术在实际隐私保护混合现实应用中的可行性。
URL
https://arxiv.org/abs/2509.10466
PDF
-
Tooth-Diffusion: Guided 3D CBCT Synthesis with Fine-Grained Tooth Conditioning
2025-08-19 21:21:35Said Djafar Said, Torkan Gholamalizadeh, Mostafa Mehdipour GhaziAbstract
Despite the growing importance of dental CBCT scans for diagnosis and treatment planning, generating anatomically realistic scans with fine-grained control remains a challenge in medical image synthesis. In this work, we propose a novel conditional diffusion framework for 3D dental volume generation, guided by tooth-level binary attributes that allow precise control over tooth presence and configuration. Our approach integrates wavelet-based denoising diffusion, FiLM conditioning, and masked loss functions to focus learning on relevant anatomical structures. We evaluate the model across diverse tasks, such as tooth addition, removal, and full dentition synthesis, using both paired and distributional similarity metrics. Results show strong fidelity and generalization with low FID scores, robust inpainting performance, and SSIM values above 0.91 even on unseen scans. By enabling realistic, localized modification of dentition without rescanning, this work opens opportunities for surgical planning, patient communication, and targeted data augmentation in dental AI workflows. The codes are available at: this https URL.
Abstract (translated)
尽管牙科CBCT（锥形束计算机断层扫描）在诊断和治疗计划中的重要性日益增加，但生成解剖结构真实且具有精细控制的扫描仍然在医学图像合成中面临挑战。在这项工作中，我们提出了一种新颖的基于条件扩散框架的方法，用于三维牙齿体积生成，并通过牙齿级别的二进制属性进行指导，这些属性允许对牙齿的存在和配置进行精确控制。我们的方法结合了基于小波去噪的扩散、FiLM（特征集成层模块）调节以及掩码损失函数，以专注于相关解剖结构的学习。 我们在包括添加或移除牙齿和全口齿列合成等多样化的任务中评估了模型的表现，并使用配对相似性和分布相似性度量进行评价。结果显示，在FID分数低的情况下表现出强大的真实性和泛化能力、稳健的修补性能以及在未见过的扫描上的SSIM值超过0.91。 通过允许在不重新扫描的情况下实现牙齿结构的真实和局部修改，这项工作为手术规划、患者沟通及牙科AI工作流程中的定向数据增强提供了新的机会。代码可在此网址获得：this https URL。
URL
https://arxiv.org/abs/2508.14276
PDF
-
OmniTry: Virtual Try-On Anything without Masks
2025-08-19 08:47:31Yutong Feng, Linlin Zhang, Hengyuan Cao, Yiming Chen, Xiaoduan Feng, Jian Cao, Yuxiong Wu, Bin WangAbstract
Virtual Try-ON (VTON) is a practical and widely-applied task, for which most of existing works focus on clothes. This paper presents OmniTry, a unified framework that extends VTON beyond garment to encompass any wearable objects, e.g., jewelries and accessories, with mask-free setting for more practical application. When extending to various types of objects, data curation is challenging for obtaining paired images, i.e., the object image and the corresponding try-on result. To tackle this problem, we propose a two-staged pipeline: For the first stage, we leverage large-scale unpaired images, i.e., portraits with any wearable items, to train the model for mask-free localization. Specifically, we repurpose the inpainting model to automatically draw objects in suitable positions given an empty mask. For the second stage, the model is further fine-tuned with paired images to transfer the consistency of object appearance. We observed that the model after the first stage shows quick convergence even with few paired samples. OmniTry is evaluated on a comprehensive benchmark consisting of 12 common classes of wearable objects, with both in-shop and in-the-wild images. Experimental results suggest that OmniTry shows better performance on both object localization and ID-preservation compared with existing methods. The code, model weights, and evaluation benchmark of OmniTry will be made publicly available at this https URL.
Abstract (translated)
虚拟试穿（Virtual Try-ON，VTON）是一项实用且广泛应用的任务，现有大多数研究主要集中在衣物上。本文介绍了OmniTry框架，这是一个统一的框架，将VTON从服饰扩展到任何可穿戴物品，如珠宝和配饰，并采用无掩码设置以实现更广泛的应用场景。在向各种类型的物体扩展时，数据整理变得具有挑战性，因为难以获得成对图像（即物品图像及其对应的试穿结果）。为了解决这个问题，我们提出了一个两阶段的流水线：第一阶段利用大规模未配对的数据集进行模型训练，这些数据包括任何可穿戴项目的肖像图片。具体来说，我们将修复模型重新用于在给定空掩码的情况下自动绘制合适位置的对象。第二阶段中，该模型进一步使用成对图像进行微调以转移物体外观的一致性。我们观察到，在第一阶段之后，即使只用少量的配对样本也能快速收敛。 OmniTry在一个全面的基准测试上进行了评估，此基准包含12种常见的可穿戴物品类别，并涵盖商店内和现实生活中的图片。实验结果表明，与现有的方法相比，OmniTry在物体定位和身份保存方面表现更好。OmniTry的代码、模型权重以及评估基准将在[此处](https://this-url.com)公开发布。
URL
https://arxiv.org/abs/2508.13632
PDF
-
IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusion
2025-08-18 17:59:47Wenhao Hu, Zesheng Li, Haonan Zhou, Liu Liu, Xuexiang Wen, Zhizhong Su, Xi Li, Gaoang WangAbstract
Reconstructing complete and interactive 3D scenes remains a fundamental challenge in computer vision and robotics, particularly due to persistent object occlusions and limited sensor coverage. Multiview observations from a single scene scan often fail to capture the full structural details. Existing approaches typically rely on multi stage pipelines, such as segmentation, background completion, and inpainting or require per-object dense scanning, both of which are error-prone, and not easily scalable. We propose IGFuse, a novel framework that reconstructs interactive Gaussian scene by fusing observations from multiple scans, where natural object rearrangement between captures reveal previously occluded regions. Our method constructs segmentation aware Gaussian fields and enforces bi-directional photometric and semantic consistency across scans. To handle spatial misalignments, we introduce a pseudo-intermediate scene state for unified alignment, alongside collaborative co-pruning strategies to refine geometry. IGFuse enables high fidelity rendering and object level scene manipulation without dense observations or complex pipelines. Extensive experiments validate the framework's strong generalization to novel scene configurations, demonstrating its effectiveness for real world 3D reconstruction and real-to-simulation transfer. Our project page is available online.
Abstract (translated)
重建完整且可交互的三维场景仍然是计算机视觉和机器人技术中的一个基本挑战，特别是由于持续的对象遮挡和有限的传感器覆盖范围。来自单次场景扫描的多视角观测往往无法捕捉到完整的结构细节。现有方法通常依赖于分阶段的工作流程，如分割、背景补全和像素级修复，或者需要对每个对象进行密集扫描，这些方法都容易出错且不易扩展。我们提出了IGFuse，一种新的框架，通过融合来自多次扫描的观察结果来重建交互式高斯场景，其中自然物体在捕获之间的重新排列会揭示之前被遮挡的区域。我们的方法构建了具有分割感知能力的高斯场，并强制执行跨扫描的双向光度和语义一致性。为了解决空间对齐问题，我们引入了一个伪中间场景状态以实现统一的对齐，并提出了协作共修剪策略来细化几何结构。IGFuse能够在没有密集观测或复杂管道的情况下提供高质量的渲染和对象级别的场景操作。广泛的实验验证了该框架在新的场景配置中的强大泛化能力，证明其在现实世界的三维重建和从真实到模拟转换方面的有效性。我们的项目页面在线可用。
URL
https://arxiv.org/abs/2508.13153
PDF
-
Geometry-Aware Video Inpainting for Joint Headset Occlusion Removal and Face Reconstruction in Social XR
2025-08-17 11:45:00Fatemeh Ghorbani Lohesara, Karen Eguiazarian, Sebastian KnorrAbstract
Head-mounted displays (HMDs) are essential for experiencing extended reality (XR) environments and observing virtual content. However, they obscure the upper part of the user's face, complicating external video recording and significantly impacting social XR applications such as teleconferencing, where facial expressions and eye gaze details are crucial for creating an immersive experience. This study introduces a geometry-aware learning-based framework to jointly remove HMD occlusions and reconstruct complete 3D facial geometry from RGB frames captured from a single viewpoint. The method integrates a GAN-based video inpainting network, guided by dense facial landmarks and a single occlusion-free reference frame, to restore missing facial regions while preserving identity. Subsequently, a SynergyNet-based module regresses 3D Morphable Model (3DMM) parameters from the inpainted frames, enabling accurate 3D face reconstruction. Dense landmark optimization is incorporated throughout the pipeline to improve both the inpainting quality and the fidelity of the recovered geometry. Experimental results demonstrate that the proposed framework can successfully remove HMDs from RGB facial videos while maintaining facial identity and realism, producing photorealistic 3D face geometry outputs. Ablation studies further show that the framework remains robust across different landmark densities, with only minor quality degradation under sparse landmark configurations.
Abstract (translated)
头戴式显示器（HMD）对于体验扩展现实（XR）环境和观察虚拟内容至关重要。然而，它们遮挡了用户面部的上半部分，这使得外部视频录制变得复杂，并显著影响到诸如远程会议等需要社交互动的XR应用，在这些场景中，面部表情和视线细节对创造沉浸式体验来说是至关重要的因素。 本文介绍了一种基于几何感知的学习框架，该框架能够从单个视角捕获的RGB帧中同时去除HMD遮挡并重建完整的3D面部几何形状。此方法采用了一个由生成对抗网络（GAN）支持的视频修复网络，并通过密集的面部标志和一个无遮挡参考图像来引导缺失区域的恢复过程，以保持身份特征完整。 随后，在修复后的画面基础上，利用基于SynergyNet的方法回归出三维可变形模型（3DMM）参数，从而实现精确的3D人脸重建。在整个流程中融入了密集地标优化步骤，以此提升修复质量和重建几何形状的真实度。 实验结果显示，所提出的框架能够成功地从RGB面部视频中去除HMD遮挡，并保持面部身份和逼真性，产生高度真实的3D脸部几何输出效果。消融研究进一步表明，在不同密度的标记配置下，该框架仍然表现得非常稳健，即使在稀疏地标的情况下也仅有轻微的质量损失。
URL
https://arxiv.org/abs/2508.12336
PDF
-
DualFit: A Two-Stage Virtual Try-On via Warping and Synthesis
2025-08-16 18:50:31Minh Tran, Johnmark Clements, Annie Prasanna, Tri Nguyen, Ngan LeAbstract
Virtual Try-On technology has garnered significant attention for its potential to transform the online fashion retail experience by allowing users to visualize how garments would look on them without physical trials. While recent advances in diffusion-based warping-free methods have improved perceptual quality, they often fail to preserve fine-grained garment details such as logos and printed text elements that are critical for brand integrity and customer trust. In this work, we propose DualFit, a hybrid VTON pipeline that addresses this limitation by two-stage approach. In the first stage, DualFit warps the target garment to align with the person image using a learned flow field, ensuring high-fidelity preservation. In the second stage, a fidelity-preserving try-on module synthesizes the final output by blending the warped garment with preserved human regions. Particularly, to guide this process, we introduce a preserved-region input and an inpainting mask, enabling the model to retain key areas and regenerate only where necessary, particularly around garment seams. Extensive qualitative results show that DualFit achieves visually seamless try-on results while faithfully maintaining high-frequency garment details, striking an effective balance between reconstruction accuracy and perceptual realism.
Abstract (translated)
虚拟试穿技术因其能够通过允许用户在不进行实体试穿的情况下可视化服装的穿着效果，从而革新在线时尚零售体验而备受关注。尽管基于扩散的方法已经改善了感知质量，但这些方法通常无法保持诸如徽标和印刷文字等细微的服装细节，而这些都是品牌完整性和客户信任的关键因素。为此，我们提出了DualFit技术，这是一种混合虚拟试穿(VTON)流水线，通过两阶段的方法解决了这一限制。 在第一阶段中，DualFit使用学习到的流场将目标服装变形以与人物图像对齐，从而确保高保真度的保存。第二阶段则是一个保持保真度的试穿模块，该模块通过将变形后的服装与保留的人体区域融合来生成最终输出。特别地，为指导这一过程，我们引入了一个保留区域输入和一个修复掩码，使模型能够仅在必要时（例如，在服装接缝周围）保留关键区域并重新生成细节。 大量的定性结果显示，DualFit实现了视觉上无缝的试穿效果，并且忠实地保持了高频次的服装细节，有效平衡了重建准确性和感知真实性。
URL
https://arxiv.org/abs/2508.12131
PDF
-
A Segmentation-driven Editing Method for Bolt Defect Augmentation and Detection
2025-08-14 10:24:46Yangjie Xiao, Ke Zhang, Jiacun Wang, Xin Sheng, Yurong Guo, Meijuan Chen, Zehua Ren, Zhaoye Zheng, Zhenbing ZhaoAbstract
Bolt defect detection is critical to ensure the safety of transmission lines. However, the scarcity of defect images and imbalanced data distributions significantly limit detection performance. To address this problem, we propose a segmentationdriven bolt defect editing method (SBDE) to augment the dataset. First, a bolt attribute segmentation model (Bolt-SAM) is proposed, which enhances the segmentation of complex bolt attributes through the CLAHE-FFT Adapter (CFA) and Multipart- Aware Mask Decoder (MAMD), generating high-quality masks for subsequent editing tasks. Second, a mask optimization module (MOD) is designed and integrated with the image inpainting model (LaMa) to construct the bolt defect attribute editing model (MOD-LaMa), which converts normal bolts into defective ones through attribute editing. Finally, an editing recovery augmentation (ERA) strategy is proposed to recover and put the edited defect bolts back into the original inspection scenes and expand the defect detection dataset. We constructed multiple bolt datasets and conducted extensive experiments. Experimental results demonstrate that the bolt defect images generated by SBDE significantly outperform state-of-the-art image editing models, and effectively improve the performance of bolt defect detection, which fully verifies the effectiveness and application potential of the proposed method. The code of the project is available at this https URL.
Abstract (translated)
螺栓缺陷检测对于确保输电线路的安全至关重要。然而，由于缺乏缺陷图像和数据分布不平衡的问题，限制了检测性能的提升。为了解决这个问题，我们提出了一种基于分割驱动的螺栓缺陷编辑方法（SBDE），用于扩充数据集。首先，提出了一个螺栓属性分割模型（Bolt-SAM）。该模型通过CLAHE-FFT适配器（CFA）和多部件感知掩码解码器（MAMD）增强了复杂螺栓属性的分割能力，生成高质量的掩码以供后续编辑任务使用。其次，设计并集成了一种掩码优化模块（MOD），并与图像修复模型（LaMa）结合构建了螺栓缺陷属性编辑模型（MOD-LaMa）。该模型通过属性编辑将正常螺栓转换为有缺陷的螺栓。最后，我们提出了一种编辑恢复增强（ERA）策略，用于恢复并重新放入经过编辑后的缺陷螺栓到原始检查场景中，并扩展缺陷检测的数据集。 我们构建了多个螺栓数据集，并进行了广泛的实验。实验结果表明，SBDE生成的螺栓缺陷图像在性能上显著优于现有的图像编辑模型，有效提升了螺栓缺陷检测的表现力，充分验证了该方法的有效性和应用潜力。该项目的代码可以在提供的链接处获取。
URL
https://arxiv.org/abs/2508.10509
PDF
-
Masquerade: Learning from In-the-wild Human Videos using Data-Editing
2025-08-13 17:43:34Marion Lepert, Jiaying Fang, Jeannette BohgAbstract
Robot manipulation research still suffers from significant data scarcity: even the largest robot datasets are orders of magnitude smaller and less diverse than those that fueled recent breakthroughs in language and vision. We introduce Masquerade, a method that edits in-the-wild egocentric human videos to bridge the visual embodiment gap between humans and robots and then learns a robot policy with these edited videos. Our pipeline turns each human video into robotized demonstrations by (i) estimating 3-D hand poses, (ii) inpainting the human arms, and (iii) overlaying a rendered bimanual robot that tracks the recovered end-effector trajectories. Pre-training a visual encoder to predict future 2-D robot keypoints on 675K frames of these edited clips, and continuing that auxiliary loss while fine-tuning a diffusion policy head on only 50 robot demonstrations per task, yields policies that generalize significantly better than prior work. On three long-horizon, bimanual kitchen tasks evaluated in three unseen scenes each, Masquerade outperforms baselines by 5-6x. Ablations show that both the robot overlay and co-training are indispensable, and performance scales logarithmically with the amount of edited human video. These results demonstrate that explicitly closing the visual embodiment gap unlocks a vast, readily available source of data from human videos that can be used to improve robot policies.
Abstract (translated)
机器人操纵研究仍然受到数据稀缺的严重影响：即使是最大的机器人数据集，也比最近语言和视觉领域突破所使用的数据小得多且多样性不足。我们引入了一种名为Masquerade的方法，该方法通过编辑野外的第一人称人类视频来弥合人类与机器人的视觉表现差距，并利用这些编辑后的视频学习机器人策略。我们的流水线将每个第一人称人类视频转换为机器人演示，具体步骤如下：(i) 估计3D手部姿态；(ii) 给人类手臂补帧；以及(iii) 在视频中叠加一个渲染的双臂机器人以跟踪恢复出来的末端执行器轨迹。通过对675,000帧编辑后的片段进行预训练，使视觉编码器预测未来2D机器人的关键点，并在仅使用每项任务50个机器人演示的情况下继续微调扩散策略头，在辅助损失下进行训练，可以得到比以往研究显著更好的泛化策略。在三个具有长时域和双臂厨房任务的三个未见过场景中，Masquerade的表现优于基线方法多达5-6倍。消融研究表明，机器人的叠加以及协同训练都是必不可少的，并且性能随着编辑后的人类视频数量以对数方式增加。这些结果表明，明确地弥合视觉表现差距可以解锁大量现成可用的数据源，即人类视频中的数据，这可用于改善机器人策略。
URL
https://arxiv.org/abs/2508.09976
PDF
-
Negative Shanshui: Real-time Interactive Ink Painting Synthesis
2025-08-13 09:24:37Aven-Le ZhouAbstract
This paper presents Negative Shanshui, a real-time interactive AI synthesis approach that reinterprets classical Chinese landscape ink painting, i.e., shanshui, to engage with ecological crises in the Anthropocene. Negative Shanshui optimizes a fine-tuned Stable Diffusion model for real-time inferences and integrates it with gaze-driven inpainting, frame interpolation; it enables dynamic morphing animations in response to the viewer's gaze and presents as an interactive virtual reality (VR) experience. The paper describes the complete technical pipeline, covering the system framework, optimization strategies, gaze-based interaction, and multimodal deployment in an art festival. Further analysis of audience feedback collected during its public exhibition highlights how participants variously engaged with the work through empathy, ambivalence, and critical reflection.
Abstract (translated)
本文介绍了Negative Shanshui，这是一种实时互动AI合成方法，它重新诠释了中国古典山水画（即山水），以应对人类世时期的生态危机。Negative Shanshui 优化了一个经过微调的Stable Diffusion模型，使其能够进行实时推断，并将其与注视驱动的图像修复技术和帧插值技术相结合；该方法使得作品可以根据观众的目光动态生成变形动画，并以互动虚拟现实（VR）体验的形式呈现。文章详细描述了整个技术流程，包括系统框架、优化策略、基于注视的交互以及在艺术节上的多模态部署情况。此外，通过对其公开展览期间收集到的观众反馈进行进一步分析，展示了参与者们如何通过同情、矛盾和批判性反思等各种方式与作品互动。
URL
https://arxiv.org/abs/2508.16612
PDF
-
Transient Noise Removal via Diffusion-based Speech Inpainting
2025-08-12 12:25:53Mordehay Moradi, Sharon GannotAbstract
In this paper, we present PGDI, a diffusion-based speech inpainting framework for restoring missing or severely corrupted speech segments. Unlike previous methods that struggle with speaker variability or long gap lengths, PGDI can accurately reconstruct gaps of up to one second in length while preserving speaker identity, prosody, and environmental factors such as reverberation. Central to this approach is classifier guidance, specifically phoneme-level guidance, which substantially improves reconstruction fidelity. PGDI operates in a speaker-independent manner and maintains robustness even when long segments are completely masked by strong transient noise, making it well-suited for real-world applications, such as fireworks, door slams, hammer strikes, and construction noise. Through extensive experiments across diverse speakers and gap lengths, we demonstrate PGDI's superior inpainting performance and its ability to handle challenging acoustic conditions. We consider both scenarios, with and without access to the transcript during inference, showing that while the availability of text further enhances performance, the model remains effective even in its absence. For audio samples, visit: this https URL
Abstract (translated)
在这篇论文中，我们提出了PGDI（基于扩散的语音填补框架），用于恢复缺失或严重损坏的语音片段。与以往的方法不同，这些方法在处理说话人变化或多段长时间间隔时表现不佳，PGDI能够在保持说话者身份、韵律和环境因素（如混响）的同时，准确重建长达一秒的间隙。该方法的核心是分类器指导，特别是音素级别的指导，这极大地提高了重构的准确性。 PGDI以独立于说话人的方式运行，并且即使在长时间段完全被强烈瞬态噪声掩盖的情况下也能保持稳健性，使其非常适合现实世界的应用场景，例如烟火声、关门声、锤击声和施工噪音。通过针对各种说话者及不同长度间隙进行广泛的实验，我们展示了PGDI卓越的填补性能及其处理复杂声学条件的能力。 本研究还探讨了推理过程中是否有访问转录文本的不同情况，结果显示虽然拥有文本可以进一步提升模型性能，但即便在没有文本的情况下，该模型依旧非常有效。音频样本请访问：[这里插入URL链接]
URL
https://arxiv.org/abs/2508.08890
PDF
