Inpainting
-
Inference-Time Search using Side Information for Diffusion-based Image Reconstruction
2025-10-02 20:16:00Mahdi Farahbakhsh, Vishnu Teja Kunde, Dileep Kalathil, Krishna Narayanan, Jean-Francois ChamberlandAbstract
Diffusion models have emerged as powerful priors for solving inverse problems. However, existing approaches typically overlook side information that could significantly improve reconstruction quality, especially in severely ill-posed settings. In this work, we propose a novel inference-time search algorithm that guides the sampling process using the side information in a manner that balances exploration and exploitation. This enables more accurate and reliable reconstructions, providing an alternative to the gradient-based guidance that is prone to reward-hacking artifacts. Our approach can be seamlessly integrated into a wide range of existing diffusion-based image reconstruction pipelines. Through extensive experiments on a number of inverse problems, such as box inpainting, super-resolution, and various deblurring tasks including motion, Gaussian, nonlinear, and blind deblurring, we show that our approach consistently improves the qualitative and quantitative performance of diffusion-based image reconstruction algorithms. We also show the superior performance of our approach with respect to other baselines, including reward gradient-based guidance algorithms. The code is available at \href{this https URL}{this repository}.
Abstract (translated)
扩散模型作为解决逆问题的强大先验方法已经崭露头角。然而，现有的方法通常忽略了能够显著提高重建质量的侧面信息，尤其是在病态严重的情况下。在这项工作中，我们提出了一种新颖的推理时间搜索算法，该算法在采样过程中利用侧面信息以平衡探索与开发的方式进行引导。这使得更准确、可靠的重建成为可能，并提供了一种替代基于梯度指导的方法，后者容易出现奖励操纵伪影。我们的方法可以无缝集成到各种现有的基于扩散的图像重建管道中。通过在多个逆问题上的广泛实验（例如箱式插值、超分辨率以及包括运动模糊、高斯模糊、非线性模糊和盲去模糊在内的多种去模糊任务），我们展示了我们的方法能够持续提高基于扩散的图像重建算法的质量和量化性能。我们也证明了相对于其他基准，包括基于奖励梯度指导的方法而言，我们的方法具有优越的表现。代码可在[\href{this https URL}{此存储库}](https://this https URL)中获取。
URL
https://arxiv.org/abs/2510.03352
PDF
-
Equivariant Splitting: Self-supervised learning from incomplete data
2025-10-01 14:08:17Victor Sechaud, J\'er\'emy Scanvic, Quentin Barth\'elemy, Patrice Abry, Juli\'an TachellaAbstract
Self-supervised learning for inverse problems allows to train a reconstruction network from noise and/or incomplete data alone. These methods have the potential of enabling learning-based solutions when obtaining ground-truth references for training is expensive or even impossible. In this paper, we propose a new self-supervised learning strategy devised for the challenging setting where measurements are observed via a single incomplete observation model. We introduce a new definition of equivariance in the context of reconstruction networks, and show that the combination of self-supervised splitting losses and equivariant reconstruction networks results in unbiased estimates of the supervised loss. Through a series of experiments on image inpainting, accelerated magnetic resonance imaging, and compressive sensing, we demonstrate that the proposed loss achieves state-of-the-art performance in settings with highly rank-deficient forward models.
Abstract (translated)
自监督学习在逆问题中的应用使得仅通过噪声和/或不完整数据来训练重建网络成为可能。这些方法有望实现基于学习的解决方案，特别是在获取用于训练的真实参考数据昂贵甚至不可能的情况下。本文中，我们提出了一种新的针对测量值仅能通过单一不完整的观察模型获得这一挑战性设置下的自监督学习策略。我们在重建网络背景下引入了等变性的新定义，并证明了结合自我监督分裂损失和等变重建网络可以得到无偏的监督损失估计。通过对图像修复、加速磁共振成像以及压缩感知进行一系列实验，我们展示了所提出的损失函数在具有高度秩亏前向模型设置中达到了最先进的性能。
URL
https://arxiv.org/abs/2510.00929
PDF
-
Photorealistic Inpainting for Perturbation-based Explanations in Ecological Monitoring
2025-10-01 01:18:27G\"unel Aghakishiyeva, Jiayi Zhou, Saagar Arya, James David Poling, Holly R. Houliston, Jamie N. Womble, David W. Johnston, Brinnae BentAbstract
Ecological monitoring is increasingly automated by vision models, yet opaque predictions limit trust and field adoption. We present an inpainting-guided, perturbation-based explanation technique that produces photorealistic, mask-localized edits that preserve scene context. Unlike masking or blurring, these edits stay in-distribution and reveal which fine-grained morphological cues drive predictions in tasks such as species recognition and trait attribution. We demonstrate the approach on a YOLOv9 detector fine-tuned for harbor seal detection in Glacier Bay drone imagery, using Segment-Anything-Model-refined masks to support two interventions: (i) object removal/replacement (e.g., replacing seals with plausible ice/water or boats) and (ii) background replacement with original animals composited onto new scenes. Explanations are assessed by re-scoring perturbed images (flip rate, confidence drop) and by expert review for ecological plausibility and interpretability. The resulting explanations localize diagnostic structures, avoid deletion artifacts common to traditional perturbations, and yield domain-relevant insights that support expert validation and more trustworthy deployment of AI in ecology.
Abstract (translated)
生态监测越来越依赖于视觉模型的自动化，然而不透明的预测限制了信任和现场应用。我们提出了一种基于修复引导、扰动分析的解释技术，该技术可以生成保留场景上下文的真实感光编辑，并且这些编辑集中在特定的遮罩区域。与简单的蒙版或模糊处理不同，这种编辑保持在数据分布之内，并揭示了驱动预测的具体细粒度形态特征，在物种识别和属性归因等任务中尤为关键。 我们在冰川湾无人机图像上对YOLOv9检测器进行微调，用于海港 seal 的检测，并利用Segment-Anything-Model优化的掩码支持两种干预措施：（i）对象移除/替换（例如，将seal替换成合理的冰、水或船只），以及（ii）背景替换，即在新的场景中以原始动物为素材进行合成。我们通过重新评估扰动后的图像得分（翻转率和置信度下降）和专家评审的生态合理性及可解释性来评估这种方法的效果。 结果表明，这些解释能够定位诊断结构，避免传统扰动常见的删除伪影，并提供与领域相关的见解以支持专家验证和更值得信赖的人工智能在生态学中的应用部署。
URL
https://arxiv.org/abs/2510.03317
PDF
-
DWGS: Enhancing Sparse-View Gaussian Splatting with Hybrid-Loss Depth Estimation and Bidirectional Warping
2025-09-29 15:03:31Yu Ma, Guoliang Wei, Yue ChengAbstract
Novel View Synthesis (NVS) from sparse views remains a core challenge in 3D reconstruction, typically suffering from overfitting, geometric distortion, and incomplete scene recovery due to limited multi-view constraints. Although 3D Gaussian Splatting (3DGS) enables real-time, high-fidelity rendering, it suffers from floating artifacts and structural inconsistencies under sparse-input settings. To address these issues, we propose DWGS, a novel unified framework that enhances 3DGS for sparse-view synthesis by integrating robust structural cues, virtual view constraints, and occluded region completion. Our approach introduces three principal contributions: a Hybrid-Loss Depth Estimation module that leverages dense matching priors with reprojection, point propagation, and smoothness constraints to enforce multi-view consistency; a Bidirectional Warping Virtual View Synthesis method generates virtual training views to impose stronger geometric and photometric constraints; and an Occlusion-Aware Reconstruction component that utilizes depth-difference mask and a learning-based inpainting model to recover obscured regions. Extensive experiments on standard benchmarks (LLFF, Blender, and DTU) show that DWGS achieves a new state-of-the-art, achieving up to 21.13 dB PSNR and 0.189 LPIPS, while retaining real-time inference capabilities.
Abstract (translated)
从稀疏视角进行小说视图合成（NVS）在三维重建中仍然是一个核心挑战，通常会因多视角约束不足而出现过拟合、几何变形和场景恢复不完整等问题。尽管3D高斯点阵法（3DGS）能够实现实时且高质量的渲染，但在稀疏输入条件下却会出现漂浮伪影和结构一致性问题。为解决这些问题，我们提出了一种新的统一框架——DWGS，通过整合鲁棒的结构线索、虚拟视图约束以及被遮挡区域完成来增强3DGS在稀疏视角合成中的表现。 我们的方法主要贡献包括： 1. **混合损失深度估计模块**：该模块利用密集匹配先验和重投影、点传播和平滑度约束，以加强多视图一致性。 2. **双向卷积虚拟视图合成立法**：此方法生成虚拟训练视角，从而施加更强的几何和光度学限制。 3. **遮挡感知重建组件**：该模块利用深度差异掩码及学习型修复模型来恢复被遮挡区域。 在标准基准（LLFF、Blender 和 DTU）上的广泛实验表明，DWGS达到了新的最先进水平，在PSNR上最高达到21.13 dB，LPIPS值为0.189，同时保持了实时推断能力。
URL
https://arxiv.org/abs/2509.24893
PDF
-
Of-SemWat: High-payload text embedding for semantic watermarking of AI-generated images with arbitrary size
2025-09-29 14:10:15Benedetta Tondi, Andrea Costanzo, Mauro BarniAbstract
We propose a high-payload image watermarking method for textual embedding, where a semantic description of the image - which may also correspond to the input text prompt-, is embedded inside the image. In order to be able to robustly embed high payloads in large-scale images - such as those produced by modern AI generators - the proposed approach builds upon a traditional watermarking scheme that exploits orthogonal and turbo codes for improved robustness, and integrates frequency-domain embedding and perceptual masking techniques to enhance watermark imperceptibility. Experiments show that the proposed method is extremely robust against a wide variety of image processing, and the embedded text can be retrieved also after traditional and AI inpainting, permitting to unveil the semantic modification the image has undergone via image-text mismatch analysis.
Abstract (translated)
我们提出了一种用于文本嵌入的高负载图像水印方法，其中将对图像的语义描述——这可能也对应于输入的文本提示——嵌入到图像中。为了能够在大规模图像（如现代AI生成器产生的图像）中稳健地嵌入大量信息，所提出的方案建立在传统的水印技术基础上，该技术利用正交和Turbo码来提高鲁棒性，并结合频域嵌入和感知掩蔽技术以增强水印的不可见性。实验表明，所提出的方法对各种图像处理具有极强的稳健性，即使经过传统和AI的修复（inpainting）后，也能检索出嵌入的文本，这使得通过图像-文本不匹配分析揭示图像经历的语义修改成为可能。
URL
https://arxiv.org/abs/2509.24823
PDF
-
ExGS: Extreme 3D Gaussian Compression with Diffusion Priors
2025-09-29 13:23:06Jiaqi Chen, Xinhao Ji, Yuanyuan Gao, Hao Li, Yuning Gong, Yifei Liu, Dan Xu, Zhihang Zhong, Dingwen Zhang, Xiao SunAbstract
Neural scene representations, such as 3D Gaussian Splatting (3DGS), have enabled high-quality neural rendering; however, their large storage and transmission costs hinder deployment in resource-constrained environments. Existing compression methods either rely on costly optimization, which is slow and scene-specific, or adopt training-free pruning and quantization, which degrade rendering quality under high compression ratios. In contrast, recent data-driven approaches provide a promising direction to overcome this trade-off, enabling efficient compression while preserving high rendering quality. We introduce \textbf{ExGS}, a novel feed-forward framework that unifies \textbf{Universal Gaussian Compression} (UGC) with \textbf{GaussPainter} for \textbf{Ex}treme 3D\textbf{GS} compression. \textbf{UGC} performs re-optimization-free pruning to aggressively reduce Gaussian primitives while retaining only essential information, whereas \textbf{GaussPainter} leverages powerful diffusion priors with mask-guided refinement to restore high-quality renderings from heavily pruned Gaussian scenes. Unlike conventional inpainting, GaussPainter not only fills in missing regions but also enhances visible pixels, yielding substantial improvements in degraded renderings. To ensure practicality, it adopts a lightweight VAE and a one-step diffusion design, enabling real-time restoration. Our framework can even achieve over $100\times$ compression (reducing a typical 354.77 MB model to about 3.31 MB) while preserving fidelity and significantly improving image quality under challenging conditions. These results highlight the central role of diffusion priors in bridging the gap between extreme compression and high-quality neural rendering. Our code repository will be released at \href{this https URL}{here}.
Abstract (translated)
神经场景表示，如3D高斯点阵（3DGS），已经实现了高质量的神经渲染；然而，它们庞大的存储和传输成本阻碍了在资源受限环境中的部署。现有压缩方法要么依赖于昂贵的优化过程，这既慢且针对特定场景，要么采用无需训练的剪枝和量化技术，在高压缩比下会降低渲染质量。相比之下，近期的数据驱动方法为克服这一权衡提供了有前景的方向，能够在保持高质量渲染的同时实现高效的压缩。我们介绍了**ExGS**，一种新的前馈框架，它将**通用高斯压缩（UGC）**与**GaussPainter**相结合，用于极端的3D**GS**压缩。**UGC**执行无重新优化的剪枝来激进地减少高斯原语数量，同时保留关键信息；而**GaussPainter**则利用强大的扩散先验和掩码引导的细化从被大量裁减的高斯场景中恢复高质量渲染图像。与传统插值不同，GaussPainter不仅填补缺失区域，还增强可见像素的质量，在质量退化的情况下取得了显著改进。为了确保实用性，该框架采用了一个轻量级的VAE（变分自编码器）和一步扩散设计，实现了实时修复。我们的框架甚至可以在超过100倍压缩（将一个典型的354.77MB模型减少到大约3.31MB）的同时保持保真度，并在挑战性条件下显著提高图像质量。这些结果突显了扩散先验在极端压缩与高质量神经渲染之间的桥梁作用。我们的代码库将在[此处](https://this https URL)发布。
URL
https://arxiv.org/abs/2509.24758
PDF
-
Diffusion Bridge or Flow Matching? A Unifying Framework and Comparative Analysis
2025-09-29 09:45:22Kaizhen Zhu, Mokai Pan, Zhechuan Yu, Jingya Wang, Jingyi Yu, Ye ShiAbstract
Diffusion Bridge and Flow Matching have both demonstrated compelling empirical performance in transformation between arbitrary distributions. However, there remains confusion about which approach is generally preferable, and the substantial discrepancies in their modeling assumptions and practical implementations have hindered a unified theoretical account of their relative merits. We have, for the first time, provided a unified theoretical and experimental validation of these two models. We recast their frameworks through the lens of Stochastic Optimal Control and prove that the cost function of the Diffusion Bridge is lower, guiding the system toward more stable and natural trajectories. Simultaneously, from the perspective of Optimal Transport, interpolation coefficients $t$ and $1-t$ of Flow Matching become increasingly ineffective when the training data size is reduced. To corroborate these theoretical claims, we propose a novel, powerful architecture for Diffusion Bridge built on a latent Transformer, and implement a Flow Matching model with the same structure to enable a fair performance comparison in various experiments. Comprehensive experiments are conducted across Image Inpainting, Super-Resolution, Deblurring, Denoising, Translation, and Style Transfer tasks, systematically varying both the distributional discrepancy (different difficulty) and the training data size. Extensive empirical results align perfectly with our theoretical predictions and allow us to delineate the respective advantages and disadvantages of these two models. Our code is available at this https URL.
Abstract (translated)
扩散桥（Diffusion Bridge）和流匹配（Flow Matching）在任意分布之间的转换中都展示了令人信服的实证性能。然而，关于哪一种方法更优的选择仍然存在疑惑，并且这两种模型之间巨大的建模假设差异及实际实现上的不同也阻碍了对它们相对优势进行统一理论阐述的可能性。我们首次提供了这两种模型统合理论和实验验证的方法。我们将这两个框架重新构想为随机最优控制的视角，并证明扩散桥的成本函数较低，这有助于系统趋向更稳定且自然的轨迹变化。同时，从最优传输的角度来看，在训练数据量减少的情况下，流匹配中的插值系数 $t$ 和 $1-t$ 逐渐变得无效。 为了验证这些理论主张，我们提出了一种新颖而强大的基于潜在Transformer架构的扩散桥模型，并实现了一个结构相同的流匹配模型以在各种实验中进行公平性能比较。我们在图像修复、超分辨率、去模糊化、去噪、转换以及风格迁移等任务上进行了全面实验，系统地改变分布差异（不同难度）和训练数据量。广泛的实证结果与我们的理论预测完全一致，并使我们能够明确指出这两种模型各自的优缺点。 有关该工作的代码可以在提供的链接中获取。
URL
https://arxiv.org/abs/2509.24531
PDF
-
Token Painter: Training-Free Text-Guided Image Inpainting via Mask Autoregressive Models
2025-09-28 14:48:52Longtao Jiang, Mingfei Han, Lei Chen, Yongqiang Yu, Feng Zhao, Xiaojun Chang, Zhihui LiAbstract
Text-guided image inpainting aims to inpaint masked image regions based on a textual prompt while preserving the background. Although diffusion-based methods have become dominant, their property of modeling the entire image in latent space makes it challenging for the results to align well with prompt details and maintain a consistent background. To address these issues, we explore Mask AutoRegressive (MAR) models for this task. MAR naturally supports image inpainting by generating latent tokens corresponding to mask regions, enabling better local controllability without altering the background. However, directly applying MAR to this task makes the inpainting content either ignore the prompts or be disharmonious with the background context. Through analysis of the attention maps from the inpainting images, we identify the impact of background tokens on text tokens during the MAR generation, and leverage this to design \textbf{Token Painter}, a training-free text-guided image inpainting method based on MAR. Our approach introduces two key components: (1) Dual-Stream Encoder Information Fusion (DEIF), which fuses the semantic and context information from text and background in frequency domain to produce novel guidance tokens, allowing MAR to generate text-faithful inpainting content while keeping harmonious with background context. (2) Adaptive Decoder Attention Score Enhancing (ADAE), which adaptively enhances attention scores on guidance tokens and inpainting tokens to further enhance the alignment of prompt details and the content visual quality. Extensive experiments demonstrate that our training-free method outperforms prior state-of-the-art methods across almost all metrics and delivers superior visual results. Codes will be released.
Abstract (translated)
文本引导的图像修复（Text-guided image inpainting）旨在根据给定的文字提示来修补被遮罩的部分，同时保持背景不变。尽管基于扩散的方法已经变得非常流行，但其在潜在空间中模拟整个图像的特点使得结果难以与文字描述细节很好地对齐，并且也很难维持背景的一致性。为了解决这些问题，我们探索了使用Mask AutoRegressive (MAR) 模型来进行这项任务的可能性。MAR模型通过生成对应于掩码区域的潜在令牌来自然地支持图像修复工作，这使得在不改变背景的情况下实现更好的局部可控性成为可能。然而，直接将MAR应用于该任务会导致修复内容要么忽视提示信息，要么与背景环境不协调。 通过对修复后的图像中注意力图的分析，我们发现背景令牌对文本令牌的影响，并利用这一点设计了\textbf{Token Painter}——一种基于MAR的、无需训练即可实现文本引导图像修补的方法。我们的方法引入了两个关键组件： 1. **双流编码器信息融合（Dual-Stream Encoder Information Fusion, DEIF）**：该机制在频域中融合来自文字和背景的信息，生成新的指导令牌，使MAR能够产生与提示一致的修复内容，并且保持与背景环境的一致性。 2. **自适应解码器注意力评分增强（Adaptive Decoder Attention Score Enhancing, ADAE）**：通过自适应地提高指导令牌和修补令牌上的注意分数来进一步加强提示细节与内容视觉质量之间的对齐。 广泛的实验表明，我们提出的无需训练的方法在几乎所有指标上都优于现有的最先进的方法，并且提供了更优秀的视觉结果。代码将公开发布。
URL
https://arxiv.org/abs/2509.23919
PDF
-
Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations
2025-09-27 15:16:23Sihao Ding, Santosh Vasa, Aditi RamadwarAbstract
Vision-Language Models (VLMs) often produce fluent Natural Language Explanations (NLEs) that sound convincing but may not reflect the causal factors driving predictions. This mismatch of plausibility and faithfulness poses technical and governance risks. We introduce Explanation-Driven Counterfactual Testing (EDCT), a fully automated verification procedure for a target VLM that treats the model's own explanation as a falsifiable hypothesis. Given an image-question pair, EDCT: (1) obtains the model's answer and NLE, (2) parses the NLE into testable visual concepts, (3) generates targeted counterfactual edits via generative inpainting, and (4) computes a Counterfactual Consistency Score (CCS) using LLM-assisted analysis of changes in both answers and explanations. Across 120 curated OK-VQA examples and multiple VLMs, EDCT uncovers substantial faithfulness gaps and provides regulator-aligned audit artifacts indicating when cited concepts fail causal tests.
Abstract (translated)
视觉-语言模型（VLM）常常会产生流利的自然语言解释（NLE），这些解释听起来非常可信，但实际上可能并不反映预测背后的原因。这种合理性和忠实性之间的不匹配带来了技术和治理上的风险。我们介绍了一种称为基于解释的反事实测试（EDCT）的完全自动化验证程序，该程序以目标VLM生成的自身解释作为可证伪假设进行处理。对于给定的一组图像和问题： 1. EDCT 首先获取模型的回答和 NLE； 2. 接着将NLE解析成可以测试的视觉概念； 3. 通过生成式修复技术（generative inpainting）来创建有针对性的反事实修改； 4. 最后，使用LLM辅助分析答案和解释的变化来计算反事实一致性得分（CCS）。 通过对120个经过挑选的OK-VQA示例以及多个VLM进行测试，EDCT揭示了显著的忠实性差距，并提供了符合监管要求的审计结果，指示哪些引用的概念未能通过因果关系检验。
URL
https://arxiv.org/abs/2510.00047
PDF
-
Follow-Your-Preference: Towards Preference-Aligned Image Inpainting
2025-09-27 03:32:30Yutao Shen, Junkun Yuan, Toru Aonishi, Hideki Nakayama, Yue MaAbstract
This paper investigates image inpainting with preference alignment. Instead of introducing a novel method, we go back to basics and revisit fundamental problems in achieving such alignment. We leverage the prominent direct preference optimization approach for alignment training and employ public reward models to construct preference training datasets. Experiments are conducted across nine reward models, two benchmarks, and two baseline models with varying structures and generative algorithms. Our key findings are as follows: (1) Most reward models deliver valid reward scores for constructing preference data, even if some of them are not reliable evaluators. (2) Preference data demonstrates robust trends in both candidate scaling and sample scaling across models and benchmarks. (3) Observable biases in reward models, particularly in brightness, composition, and color scheme, render them susceptible to cause reward hacking. (4) A simple ensemble of these models yields robust and generalizable results by mitigating such biases. Built upon these observations, our alignment models significantly outperform prior models across standard metrics, GPT-4 assessments, and human evaluations, without any changes to model structures or the use of new datasets. We hope our work can set a simple yet solid baseline, pushing this promising frontier. Our code is open-sourced at: this https URL.
Abstract (translated)
本文探讨了基于偏好对齐的图像修复技术。不同于提出一种新的方法，我们回归基础，重新审视实现这种对齐的基本问题。我们利用突出的直接偏好优化方法进行对齐训练，并使用公开的奖励模型来构建偏好数据集。实验在九种不同的奖励模型、两个基准测试以及两种结构和生成算法各不相同的基线模型上进行。我们的主要发现如下：（1）大多数奖励模型可以为构造偏好数据提供有效的评分，即使其中一些并不是可靠的评估工具。（2）偏好数据在不同候选方案及样本数量上的趋势对所有模型和基准测试来说都非常稳定。（3）奖励模型中存在的可观察偏差，尤其是亮度、构图和色彩方案方面的偏差，使得它们容易受到奖励作弊行为的影响。（4）这些模型的简单集成可以缓解这种偏差，并产生稳健且通用的结果。基于以上发现，我们的对齐模型在标准度量、GPT-4评估以及人工评价方面均显著超越了先前的模型，而无需更改任何模型结构或引入新的数据集。我们希望本项工作能够为该领域设立一个简单而坚实的基准，推动这一充满前景的研究前沿的发展。我们的代码可在以下网址开放获取：[此 HTTPS URL]。
URL
https://arxiv.org/abs/2509.23082
PDF
-
UniMIC: Token-Based Multimodal Interactive Coding for Human-AI Collaboration
2025-09-26 16:46:12Qi Mao, Tinghan Yang, Jiahao Li, Bin Li, Libiao Jin, Yan LuAbstract
The rapid progress of Large Multimodal Models (LMMs) and cloud-based AI agents is transforming human-AI collaboration into bidirectional, multimodal interaction. However, existing codecs remain optimized for unimodal, one-way communication, resulting in repeated degradation under conventional compress-transmit-reconstruct pipelines. To address this limitation, we propose UniMIC, a Unified token-based Multimodal Interactive Coding framework that bridges edge devices and cloud AI agents. Instead of transmitting raw pixels or plain text, UniMIC employs compact tokenized representations as the communication medium, enabling efficient low-bitrate transmission while maintaining compatibility with LMMs. To further enhance compression, lightweight Transformer-based entropy models with scenario-specific designs-generic, masked, and text-conditioned-effectively minimize inter-token redundancy. Extensive experiments on text-to-image generation, text-guided inpainting, outpainting, and visual question answering show that UniMIC achieves substantial bitrate savings and remains robust even at ultra-low bitrates (<0.05bpp), without compromising downstream task performance. These results establish UniMIC as a practical and forward-looking paradigm for next-generation multimodal interactive communication.
Abstract (translated)
大型多模态模型（LMMs）和基于云的AI代理的快速进步正在将人机协作转变为双向、多模式互动。然而，现有的编解码器仍然针对单模式的一向通信进行优化，在传统的压缩-传输-重建管道中导致重复降质。为了克服这一限制，我们提出了UniMIC，这是一种统一的令牌基多模态交互编码框架，用于连接边缘设备和云端AI代理。与传输原始像素或纯文本不同，UniMIC采用紧凑的标记化表示作为通信媒介，能够在保持与LMMs兼容性的同时实现高效的低比特率传输。为了进一步提高压缩效果，我们设计了基于轻量级Transformer的熵模型，包括通用、掩码和文本条件三种特定场景的设计，有效减少了令牌间的冗余。 在文本到图像生成、文本引导的修补、扩展以及视觉问答等广泛的实验中显示，UniMIC实现了显著的比特率节省，并且即使在极低比特率（<0.05bpp）的情况下也保持了强大的性能稳定性，同时没有影响下游任务的表现。这些结果表明，UniMIC为下一代多模态交互通信建立了一种实用和前瞻性的范式。
URL
https://arxiv.org/abs/2509.22570
PDF
-
SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks
2025-09-26 06:14:55Jialiang Li, Wenzheng Wu, Gaojing Zhang, Yifan Han, Wenzhao LianAbstract
Successfully solving long-horizon manipulation tasks remains a fundamental challenge. These tasks involve extended action sequences and complex object interactions, presenting a critical gap between high-level symbolic planning and low-level continuous control. To bridge this gap, two essential capabilities are required: robust long-horizon task planning and effective goal-conditioned manipulation. Existing task planning methods, including traditional and LLM-based approaches, often exhibit limited generalization or sparse semantic reasoning. Meanwhile, image-conditioned control methods struggle to adapt to unseen tasks. To tackle these problems, we propose SAGE, a novel framework for Scene Graph-Aware Guidance and Execution in Long-Horizon Manipulation Tasks. SAGE utilizes semantic scene graphs as a structural representation for scene states. A structural scene graph enables bridging task-level semantic reasoning and pixel-level visuo-motor control. This also facilitates the controllable synthesis of accurate, novel sub-goal images. SAGE consists of two key components: (1) a scene graph-based task planner that uses VLMs and LLMs to parse the environment and reason about physically-grounded scene state transition sequences, and (2) a decoupled structural image editing pipeline that controllably converts each target sub-goal graph into a corresponding image through image inpainting and composition. Extensive experiments have demonstrated that SAGE achieves state-of-the-art performance on distinct long-horizon tasks.
Abstract (translated)
成功解决长期操作任务仍然是一个基本挑战。这些任务涉及扩展的动作序列和复杂的对象交互，这在高层次的符号规划与低层次的连续控制之间形成了关键差距。为了弥合这一差距，需要两种核心能力：稳健的长时序任务规划和有效的目标导向操控。现有的任务规划方法（包括传统方法及基于LLM的方法）通常表现出有限的泛化能力和稀疏的语义推理。同时，以图像为基础的控制方法难以适应未见过的任务。为了应对这些问题，我们提出了SAGE框架——面向长期操作任务中场景图感知引导与执行的新颖架构。SAGE利用语义场景图作为场景状态的结构性表示。这种结构化的场景图有助于在任务级别的语义推理和像素级视觉-运动控制之间架桥，并促进精确且新颖子目标图像的可控合成。SAGE包含两个关键组件：（1）一个基于场景图的任务规划器，它使用视觉语言模型(VLM) 和大型语言模型(LLM) 来解析环境并思考物理基础的场景状态转变序列；（2）一条分离的结构化图像编辑流水线，该流水线通过图像修复与组合将每个目标子图可控地转换为对应图像。大量的实验表明，SAGE在不同的长期操作任务上达到了最先进的性能水平。
URL
https://arxiv.org/abs/2509.21928
PDF
-
StableDub: Taming Diffusion Prior for Generalized and Efficient Visual Dubbing
2025-09-26 05:23:31Liyang Chen, Tianze Zhou, Xu He, Boshi Tang, Zhiyong Wu, Yang Huang, Yang Wu, Zhongqian Sun, Wei Yang, Helen MengAbstract
The visual dubbing task aims to generate mouth movements synchronized with the driving audio, which has seen significant progress in recent years. However, two critical deficiencies hinder their wide application: (1) Audio-only driving paradigms inadequately capture speaker-specific lip habits, which fail to generate lip movements similar to the target avatar; (2) Conventional blind-inpainting approaches frequently produce visual artifacts when handling obstructions (e.g., microphones, hands), limiting practical deployment. In this paper, we propose StableDub, a novel and concise framework integrating lip-habit-aware modeling with occlusion-robust synthesis. Specifically, building upon the Stable-Diffusion backbone, we develop a lip-habit-modulated mechanism that jointly models phonemic audio-visual synchronization and speaker-specific orofacial dynamics. To achieve plausible lip geometries and object appearances under occlusion, we introduce the occlusion-aware training strategy by explicitly exposing the occlusion objects to the inpainting process. By incorporating the proposed designs, the model eliminates the necessity for cost-intensive priors in previous methods, thereby exhibiting superior training efficiency on the computationally intensive diffusion-based backbone. To further optimize training efficiency from the perspective of model architecture, we introduce a hybrid Mamba-Transformer architecture, which demonstrates the enhanced applicability in low-resource research scenarios. Extensive experimental results demonstrate that StableDub achieves superior performance in lip habit resemblance and occlusion robustness. Our method also surpasses other methods in audio-lip sync, video quality, and resolution consistency. We expand the applicability of visual dubbing methods from comprehensive aspects, and demo videos can be found at this https URL.
Abstract (translated)
视觉配音任务的目标是生成与驱动音频同步的口型动画，近年来取得了显著进展。然而，两个关键缺陷阻碍了其广泛应用：(1) 仅基于音频驱动的方法不足以捕捉特定说话人的唇部习惯，因而无法产生类似于目标角色的口型动作；(2) 常规的盲填充方法在处理遮挡物（如麦克风、手）时常常会产生视觉伪影，限制了实际应用。为此，在本文中我们提出了StableDub，这是一种新颖且简洁的框架，集成了唇部习惯感知建模与遮挡鲁棒合成技术。具体而言，在Stable-Diffusion骨干网络的基础上，我们开发了一种受唇部习惯调制的机制，该机制联合模型音素级音频-视觉同步和特定说话人的口面部动态。为了在存在遮挡的情况下实现逼真的唇形几何结构和对象外观，我们引入了感知遮挡训练策略，并通过显式地将遮挡物暴露给填充过程来提高鲁棒性。通过采用所提出的这些设计，模型消除了对以往方法中成本高昂的先验知识的需求，在计算密集型扩散基础架构上表现出色的培训效率。为从模型架构角度进一步优化培训效率，我们引入了混合Mamba-Transformer架构，这在资源有限的研究场景中显示出增强的应用性。 大量的实验结果表明，StableDub 在唇部习惯相似度和遮挡鲁棒性方面表现卓越，并且在音频与嘴唇同步、视频质量和分辨率一致性等方面也超越了其他方法。我们的方法从多个方面扩展了视觉配音技术的适用性，演示视频可在以下网址找到：[https URL]。
URL
https://arxiv.org/abs/2509.21887
PDF
-
Noise-to-Notes: Diffusion-based Generation and Refinement for Automatic Drum Transcription
2025-09-26 01:12:43Michael Yeung, Keisuke Toyama, Toya Teramoto, Shusuke Takahashi, Tamaki KojimaAbstract
Automatic drum transcription (ADT) is traditionally formulated as a discriminative task to predict drum events from audio spectrograms. In this work, we redefine ADT as a conditional generative task and introduce Noise-to-Notes (N2N), a framework leveraging diffusion modeling to transform audio-conditioned Gaussian noise into drum events with associated velocities. This generative diffusion approach offers distinct advantages, including a flexible speed-accuracy trade-off and strong inpainting capabilities. However, the generation of binary onset and continuous velocity values presents a challenge for diffusion models, and to overcome this, we introduce an Annealed Pseudo-Huber loss to facilitate effective joint optimization. Finally, to augment low-level spectrogram features, we propose incorporating features extracted from music foundation models (MFMs), which capture high-level semantic information and enhance robustness to out-of-domain drum audio. Experimental results demonstrate that including MFM features significantly improves robustness and N2N establishes a new state-of-the-art performance across multiple ADT benchmarks.
Abstract (translated)
自动鼓点转录（ADT）传统上被定义为一个判别性任务，即从音频频谱图中预测鼓点事件。在这项工作中，我们将ADT重新定义为一个条件生成性任务，并引入了Noise-to-Notes（N2N），这是一个框架，利用扩散模型将受音频调节的高斯噪声转化为带有相关速度值的鼓点事件。这种生成式扩散方法提供了灵活的速度与精度权衡和强大的修复能力等独特优势。然而，二进制起始时间和连续力度值的生成对扩散模型来说是一个挑战，为了解决这一问题，我们引入了渐变伪赫比格损失（Annealed Pseudo-Huber loss）以促进有效的联合优化。 最后，为了增强低级频谱图特征，我们建议加入由音乐基础模型（MFMs）提取的特征。这些模型捕捉高层次的语义信息，并增强了对域外鼓声音频的鲁棒性。实验结果表明，包含MFM特征显著提高了鲁棒性，而N2N在多个ADT基准测试中建立了新的性能标杆。
URL
https://arxiv.org/abs/2509.21739
PDF
-
Consistency Models as Plug-and-Play Priors for Inverse Problems
2025-09-25 20:27:56Merve G\"ulle, Junno Yun, Ya\c{s}ar Utku Al\c{c}alar, Mehmet Ak\c{c}akayaAbstract
Diffusion models have found extensive use in solving numerous inverse problems. Such diffusion inverse problem solvers aim to sample from the posterior distribution of data given the measurements, using a combination of the unconditional score function and an approximation of the posterior related to the forward process. Recently, consistency models (CMs) have been proposed to directly predict the final output from any point on the diffusion ODE trajectory, enabling high-quality sampling in just a few NFEs. CMs have also been utilized for inverse problems, but existing CM-based solvers either require additional task-specific training or utilize data fidelity operations with slow convergence, not amenable to large-scale problems. In this work, we reinterpret CMs as proximal operators of a prior, enabling their integration into plug-and-play (PnP) frameworks. We propose a solver based on PnP-ADMM, which enables us to leverage the fast convergence of conjugate gradient method. We further accelerate this with noise injection and momentum, dubbed PnP-CM, and show it maintains the convergence properties of the baseline PnP-ADMM. We evaluate our approach on a variety of inverse problems, including inpainting, super-resolution, Gaussian deblurring, and magnetic resonance imaging (MRI) reconstruction. To the best of our knowledge, this is the first CM trained for MRI datasets. Our results show that PnP-CM achieves high-quality reconstructions in as few as 4 NFEs, and can produce meaningful results in 2 steps, highlighting its effectiveness in real-world inverse problems while outperforming comparable CM-based approaches.
Abstract (translated)
扩散模型在解决各种逆向问题中得到了广泛应用。这些扩散逆向问题求解器旨在利用无条件评分函数和正向过程相关的后验分布的近似值来从给定测量数据的数据后验分布采样。最近，一致性模型（CMs）被提出直接预测扩散ODE轨迹上任一点的最终输出，从而仅通过少量的NFE（网络前向传递次数）就能实现高质量的采样。CMs也被应用于逆向问题求解，但现有的基于CM的方法要么需要额外的任务特定训练，要么使用收敛缓慢的数据保真度操作，不适合大规模的问题。 在本工作中，我们重新解释了CMs作为先验概率的proximal算子，使其能够被集成到即插即用（PnP）框架中。我们提出了一个基于PnP-ADMM的方法，使我们可以利用共轭梯度法快速收敛的优点。进一步地，通过引入噪声注入和动量加速这一方法，并将其命名为PnP-CM，证明其保持了基础的PnP-ADMM收敛特性。 我们在包括图像修复、超分辨率重建、高斯去模糊以及磁共振成像（MRI）重构等不同类型的逆向问题上评估了我们的方法。据我们所知，这是首个专门用于MRI数据集训练的一致性模型。实验结果显示，PnP-CM能够在仅4次NFE的情况下实现高质量的图像重建，并在2步内生成有意义的结果，突出显示其在实际逆向问题中的高效性和优越性，超过了同类CM方法的表现。
URL
https://arxiv.org/abs/2509.22736
PDF
-
KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Models
2025-09-25 11:35:40Sibo Li, Qianyue Hao, Yu Shang, Yong LiAbstract
Robotic world models are a promising paradigm for forecasting future environment states, yet their inference speed and the physical plausibility of generated trajectories remain critical bottlenecks, limiting their real-world applications. This stems from the redundancy of the prevailing frame-to-frame generation approach, where the model conducts costly computation on similar frames, as well as neglecting the semantic importance of key transitions. To address this inefficiency, we propose KeyWorld, a framework that improves text-conditioned robotic world models by concentrating transformers computation on a few semantic key frames while employing a lightweight convolutional model to fill the intermediate frames. Specifically, KeyWorld first identifies significant transitions by iteratively simplifying the robot's motion trajectories, obtaining the ground truth key frames. Then, a DiT model is trained to reason and generate these physically meaningful key frames from textual task descriptions. Finally, a lightweight interpolator efficiently reconstructs the full video by inpainting all intermediate frames. Evaluations on the LIBERO benchmark demonstrate that KeyWorld achieves a 5.68$\times$ acceleration compared to the frame-to-frame generation baseline, and focusing on the motion-aware key frames further contributes to the physical validity of the generated videos, especially on complex tasks. Our approach highlights a practical path toward deploying world models in real-time robotic control and other domains requiring both efficient and effective world models. Code is released at this https URL.
Abstract (translated)
机器人世界的模型是一种有前景的预测未来环境状态的方法，然而它们的推理速度和生成轨迹的物理合理性仍然存在关键瓶颈，限制了其在现实世界中的应用。这源于当前流行的帧对帧生成方法的冗余性，在这种方法中，模型会在相似的帧上进行昂贵的计算，并且忽略了关键过渡过程中的语义重要性。为了应对这种低效率问题，我们提出了KeyWorld框架，通过将Transformer计算集中在少数几个语义关键帧上，并使用轻量级卷积模型填充中间帧来改进基于文本条件的机器人世界模型。 具体来说，KeyWorld首先通过迭代简化机器人的运动轨迹来识别重要的过渡过程，获得真正的关键帧。然后训练一个DiT模型，从任务描述的文字中推理并生成这些具有物理意义的关键帧。最后，使用轻量级插值器高效地重建完整视频，通过填充所有中间帧进行重构。 在LIBERO基准测试上的评估显示，与帧对帧的生成基线相比，KeyWorld实现了5.68倍的速度提升，并且专注于运动感知关键帧进一步提升了所生成视频的物理合理性，尤其是在复杂任务中表现尤为突出。我们的方法强调了一条实用路径，将世界模型应用于实时机器人控制及其他需要高效和有效世界模型的领域。 代码可以在该链接处获取（原文中的具体链接未给出）。
URL
https://arxiv.org/abs/2509.21027
PDF
-
FHRFormer: A Self-supervised Transformer Approach for Fetal Heart Rate Inpainting and Forecasting
2025-09-25 07:40:21Kjersti Engan, Neel Kanwal, Anita Yeconia, Ladislaus Blacy, Yuda Munyaw, Estomih Mduma, Hege ErsdalAbstract
Approximately 10\% of newborns require assistance to initiate breathing at birth, and around 5\% need ventilation support. Fetal heart rate (FHR) monitoring plays a crucial role in assessing fetal well-being during prenatal care, enabling the detection of abnormal patterns and supporting timely obstetric interventions to mitigate fetal risks during labor. Applying artificial intelligence (AI) methods to analyze large datasets of continuous FHR monitoring episodes with diverse outcomes may offer novel insights into predicting the risk of needing breathing assistance or interventions. Recent advances in wearable FHR monitors have enabled continuous fetal monitoring without compromising maternal mobility. However, sensor displacement during maternal movement, as well as changes in fetal or maternal position, often lead to signal dropouts, resulting in gaps in the recorded FHR data. Such missing data limits the extraction of meaningful insights and complicates automated (AI-based) analysis. Traditional approaches to handle missing data, such as simple interpolation techniques, often fail to preserve the spectral characteristics of the signals. In this paper, we propose a masked transformer-based autoencoder approach to reconstruct missing FHR signals by capturing both spatial and frequency components of the data. The proposed method demonstrates robustness across varying durations of missing data and can be used for signal inpainting and forecasting. The proposed approach can be applied retrospectively to research datasets to support the development of AI-based risk algorithms. In the future, the proposed method could be integrated into wearable FHR monitoring devices to achieve earlier and more robust risk detection.
Abstract (translated)
大约有10%的新生儿在出生时需要帮助才能开始呼吸，而约5%的婴儿则需要机械通气支持。胎儿心率（FHR）监测在产前护理中对于评估胎儿健康状况起着至关重要的作用，能够检测出异常模式并支持及时进行助产干预以降低分娩期间对胎儿的风险。利用人工智能（AI）方法分析大量连续FHR监测数据集，并结合不同的结果，可能为预测需要呼吸辅助或干预的新生儿风险提供新的见解。最近可穿戴式FHR监视器的进步使在不牺牲孕妇活动自由度的情况下实现持续性胎儿监控成为可能。然而，在母亲移动时传感器偏移以及胎位和母体位置变化常常会导致信号丢失，产生记录中的心率数据空白。这种缺失的数据限制了提取有意义的见解，并增加了自动化（基于AI）分析的复杂性。传统的处理缺失数据的方法，例如简单的插值技术，往往无法保留信号的频谱特性。 本文提出了一种基于掩码变换器的自动编码器方法来重建缺失的心率信号，该方法能够捕捉到数据的空间和频率成分。所提出的方法在面对不同长度的数据丢失时表现出稳健性，并可用于信号修补（inpainting）和预测。这一新方法可以应用于研究数据集以支持开发基于AI的风险算法的研究工作。在未来，这种方法可能被整合进可穿戴FHR监测设备中，从而实现更早且更为可靠的危险预警。
URL
https://arxiv.org/abs/2509.20852
PDF
-
InstructVTON: Optimal Auto-Masking and Natural-Language-Guided Interactive Style Control for Inpainting-Based Virtual Try-On
2025-09-24 19:52:40Julien Han, Shuwen Qiu, Qi Li, Xingzi Xu, Mehmet Saygin Seyfioglu, Kavosh Asadi, Karim BouyarmaneAbstract
We present InstructVTON, an instruction-following interactive virtual try-on system that allows fine-grained and complex styling control of the resulting generation, guided by natural language, on single or multiple garments. A computationally efficient and scalable formulation of virtual try-on formulates the problem as an image-guided or image-conditioned inpainting task. These inpainting-based virtual try-on models commonly use a binary mask to control the generation layout. Producing a mask that yields desirable result is difficult, requires background knowledge, might be model dependent, and in some cases impossible with the masking-based approach (e.g. trying on a long-sleeve shirt with "sleeves rolled up" styling on a person wearing long-sleeve shirt with sleeves down, where the mask will necessarily cover the entire sleeve). InstructVTON leverages Vision Language Models (VLMs) and image segmentation models for automated binary mask generation. These masks are generated based on user-provided images and free-text style instructions. InstructVTON simplifies the end-user experience by removing the necessity of a precisely drawn mask, and by automating execution of multiple rounds of image generation for try-on scenarios that cannot be achieved with masking-based virtual try-on models alone. We show that InstructVTON is interoperable with existing virtual try-on models to achieve state-of-the-art results with styling control.
Abstract (translated)
我们介绍了InstructVTON，这是一种遵循指令的交互式虚拟试穿系统，它允许通过自然语言引导，在单件或多件服装上实现细粒度和复杂的风格控制。该系统的虚拟试穿问题被表述为一个图像指导或条件下的修复任务（inpainting task），这是一个计算效率高且可扩展的方法。 基于修复技术的虚拟试穿模型通常使用二值掩码来控制生成布局。创建能够产生期望效果的掩模非常困难，需要特定知识，并可能依赖于具体模型；在某些情况下，仅靠掩码方法无法实现目标（例如，对于一个穿着长袖衬衫且袖子未卷起的人，想尝试长袖上衣并带有“袖子卷起来”的风格）。为了应对这些挑战，InstructVTON利用视觉语言模型(VLMs)和图像分割模型来自动生成二值掩模。这些掩模基于用户提供的图片和自由文本样式指令自动生成。 InstructVTON通过去除对精确绘制的掩码的需求以及自动化执行多轮图像生成的过程简化了最终用户的体验，尤其是在使用基于掩码的方法无法实现虚拟试穿的情况中更为明显。我们展示了InstructVTON可以与现有的虚拟试穿模型兼容，并在保持风格控制的同时达到最佳性能水平。 简而言之，InstructVTON通过结合视觉语言理解和自动化技术，显著提升了虚拟试穿系统的灵活性和用户友好性，同时实现了最先进的结果。
URL
https://arxiv.org/abs/2509.20524
PDF
-
Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving
2025-09-24 13:35:15Pengxiang Li, Yinan Zheng, Yue Wang, Huimin Wang, Hang Zhao, Jingjing Liu, Xianyuan Zhan, Kun Zhan, Xianpeng LangAbstract
End-to-End (E2E) solutions have emerged as a mainstream approach for autonomous driving systems, with Vision-Language-Action (VLA) models representing a new paradigm that leverages pre-trained multimodal knowledge from Vision-Language Models (VLMs) to interpret and interact with complex real-world environments. However, these methods remain constrained by the limitations of imitation learning, which struggles to inherently encode physical rules during training. Existing approaches often rely on complex rule-based post-refinement, employ reinforcement learning that remains largely limited to simulation, or utilize diffusion guidance that requires computationally expensive gradient calculations. To address these challenges, we introduce ReflectDrive, a novel learning-based framework that integrates a reflection mechanism for safe trajectory generation via discrete diffusion. We first discretize the two-dimensional driving space to construct an action codebook, enabling the use of pre-trained Diffusion Language Models for planning tasks through fine-tuning. Central to our approach is a safety-aware reflection mechanism that performs iterative self-correction without gradient computation. Our method begins with goal-conditioned trajectory generation to model multi-modal driving behaviors. Based on this, we apply local search methods to identify unsafe tokens and determine feasible solutions, which then serve as safe anchors for inpainting-based regeneration. Evaluated on the NAVSIM benchmark, ReflectDrive demonstrates significant advantages in safety-critical trajectory generation, offering a scalable and reliable solution for autonomous driving systems.
Abstract (translated)
端到端（E2E）解决方案已成为自主驾驶系统中的主流方法，其中视觉-语言-行动（VLA）模型代表了一种新的范式，该范式利用了从视觉-语言模型（VLMs）中预训练的多模态知识来解释和与复杂的现实世界环境互动。然而，这些方法仍然受限于模仿学习的局限性，在培训过程中难以内在地编码物理规则。现有方法往往依赖于复杂基于规则的事后优化、主要限于模拟中的强化学习或需要计算成本高昂的梯度计算的扩散引导。为了解决这些问题，我们引入了ReflectDrive，这是一种新的学习框架，通过离散扩散整合了一个反射机制以生成安全轨迹。首先，我们将二维驾驶空间离散化来构建一个动作代码本，从而能够通过微调预训练的扩散语言模型来进行规划任务。我们的方法的核心是一个具有安全意识的反思机制，该机制可以在没有梯度计算的情况下进行迭代自我修正。我们的方法从基于目标条件的轨迹生成开始建模多模式驾驶行为，并在此基础上应用局部搜索方法来识别不安全令牌并确定可行方案，这些方案随后作为再生中的安全锚点使用。在NAVSIM基准上评估后，ReflectDrive展示了在关键安全性轨迹生成方面的显著优势，为自主驾驶系统提供了可扩展且可靠的解决方案。
URL
https://arxiv.org/abs/2509.20109
PDF
-
GS-RoadPatching: Inpainting Gaussians via 3D Searching and Placing for Driving Scenes
2025-09-24 09:44:37Guo Chen, Jiarun Liu, Sicong Du, Chenming Wu, Deqi Li, Shi-Sheng Huang, Guofeng Zhang, Sheng YangAbstract
This paper presents GS-RoadPatching, an inpainting method for driving scene completion by referring to completely reconstructed regions, which are represented by 3D Gaussian Splatting (3DGS). Unlike existing 3DGS inpainting methods that perform generative completion relying on 2D perspective-view-based diffusion or GAN models to predict limited appearance or depth cues for missing regions, our approach enables substitutional scene inpainting and editing directly through the 3DGS modality, extricating it from requiring spatial-temporal consistency of 2D cross-modals and eliminating the need for time-intensive retraining of Gaussians. Our key insight is that the highly repetitive patterns in driving scenes often share multi-modal similarities within the implicit 3DGS feature space and are particularly suitable for structural matching to enable effective 3DGS-based substitutional inpainting. Practically, we construct feature-embedded 3DGS scenes to incorporate a patch measurement method for abstracting local context at different scales and, subsequently, propose a structural search method to find candidate patches in 3D space effectively. Finally, we propose a simple yet effective substitution-and-fusion optimization for better visual harmony. We conduct extensive experiments on multiple publicly available datasets to demonstrate the effectiveness and efficiency of our proposed method in driving scenes, and the results validate that our method achieves state-of-the-art performance compared to the baseline methods in terms of both quality and interoperability. Additional experiments in general scenes also demonstrate the applicability of the proposed 3D inpainting strategy. The project page and code are available at: this https URL
Abstract (translated)
这篇论文介绍了GS-RoadPatching，这是一种用于驾驶场景完成的图像修复方法，通过参考完全重建区域来进行操作，这些完全重建区域由三维高斯点（3D Gaussian Splatting, 3DGS）表示。与现有的基于2D透视视角扩散或GAN模型预测缺失区域的有限外观或深度线索的生成式补全方法不同，我们的方法直接在3DGS模态下实现了替代场景修复和编辑，从而摆脱了对2D跨模式时空一致性的需求，并消除了重新训练高斯分布所需的大量时间。我们的关键见解在于驾驶场景中的高度重复性图案在其隐式的3DGS特征空间中往往共享多模态相似性，这特别适合于结构匹配以实现有效的基于3DGS的替代修复。 在实践中，我们构建了嵌入特征的3DGS场景，并引入了一种用于不同尺度下抽象局部上下文的补丁测量方法。随后，我们提出了一种结构搜索方法，在3D空间中有效找到候选补丁。最后，我们提出了一种简单而有效的替换与融合优化方法，以实现更好的视觉和谐性。 我们在多个公开数据集上进行了广泛的实验，展示了所提方法在驾驶场景中的有效性和效率，并且结果验证了我们的方法相比基线方法，在质量和互操作性方面均达到了业界领先水平。此外，一般场景中的额外实验也证明了所提出的3D修复策略的应用可行性。项目页面和代码可在以下网址获取：this https URL
URL
https://arxiv.org/abs/2509.19937
PDF
