Inpainting
-
From Artefact to Insight: Efficient Low-Rank Adaptation of BrushNet for Scanning Probe Microscopy Image Restoration
2026-03-16 05:48:58Ziwei Wei, Yao Shen, Wanheng Lu, Ghim Wei Ho, Kaiyang ZengAbstract
Scanning Probe Microscopy or SPM offers nanoscale resolution but is frequently marred by structured artefacts such as line scan dropout, gain induced noise, tip convolution, and phase hops. While most available methods treat SPM artefact removal as isolated denoising or interpolation tasks, the generative inpainting perspective remains largely unexplored. In this work, we introduce a diffusion based inpainting framework tailored to scientific grayscale imagery. By fine tuning less than 0.2 percent of BrushNet weights with rank constrained low rank adaptation (LoRA), we adapt a pretrained diffusion model using only 7390 artefact, clean pairs distilled from 739 experimental scans. On our forthcoming public SPM InpBench benchmark, the LoRA enhanced model lifts the Peak Signal to Noise Ratio or PSNR by 6.61 dB and halves the Learned Perceptual Image Patch Similarity or LPIPS relative to zero-shot inference, while matching or slightly surpassing the accuracy of full retraining, trainable on a single GPU instead of four high-memory cards. The approach generalizes across various SPM image channels including height, amplitude and phase, faithfully restores subtle structural details, and suppresses hallucination artefacts inherited from natural image priors. This lightweight framework enables efficient, scalable recovery of irreplaceable SPM images and paves the way for a broader diffusion model adoption in nanoscopic imaging analysis.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14850
PDF
-
PHAC: Promptable Human Amodal Completion
2026-03-16 02:29:33Seung Young Noh, Ju Yong ChangAbstract
Conditional image generation methods are increasingly used in human-centric applications, yet existing human amodal completion (HAC) models offer users limited control over the completed content. Given an occluded person image, they hallucinate invisible regions while preserving visible ones, but cannot reliably incorporate user-specified constraints such as a desired pose or spatial extent. As a result, users often resort to repeatedly sampling the model until they obtain a satisfactory output. Pose-guided person image synthesis (PGPIS) methods allow explicit pose conditioning, but frequently fail to preserve the instance-specific visible appearance and tend to be biased toward the training distribution, even when built on strong diffusion model priors. To address these limitations, we introduce promptable human amodal completion (PHAC), a new task that completes occluded human images while satisfying both visible appearance constraints and multiple user prompts. Users provide simple point-based prompts, such as additional joints for the target pose or bounding boxes for desired regions; these prompts are encoded using ControlNet modules specialized for each prompt type. These modules inject the prompt signals into a pre-trained diffusion model, and we fine-tune only the cross-attention blocks to obtain strong prompt alignment without degrading the underlying generative prior. To further preserve visible content, we propose an inpainting-based refinement module that starts from a slightly noised coarse completion, faithfully preserves the visible regions, and ensures seamless blending at occlusion boundaries. Extensive experiments on the HAC and PGPIS benchmarks show that our approach yields more physically plausible and higher-quality completions, while significantly improving prompt alignment compared with existing amodal completion and pose-guided synthesis methods.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14741
PDF
-
A$^2$-Edit: Precise Reference-Guided Image Editing of Arbitrary Objects and Ambiguous Masks
2026-03-11 11:54:13Huayu Zheng, Guangzhao Li, Baixuan Zhao, Siqi Luo, Hantao Jiang, Guangtao Zhai, Xiaohong LiuAbstract
We propose \textbf{A$^2$-Edit}, a unified inpainting framework for arbitrary object categories, which allows users to replace any target region with a reference object using only a coarse mask. To address the issues of severe homogenization and limited category coverage in existing datasets, we construct a large-scale, multi-category dataset \textbf{UniEdit-500K}, which includes 8 major categories, 209 fine-grained subcategories, and a total of 500,104 image pairs. Such rich category diversity poses new challenges for the model, requiring it to automatically learn semantic relationships and distinctions across categories. To this end, we introduce the \textbf{Mixture of Transformer} module, which performs differentiated modeling of various object categories through dynamic expert selection, and further enhances cross-category semantic transfer and generalization through collaboration among experts. In addition, we propose a \textbf{Mask Annealing Training Strategy} (MATS) that progressively relaxes mask precision during training, reducing the model's reliance on accurate masks and improving robustness across diverse editing tasks. Extensive experiments on benchmarks such as VITON-HD and AnyInsertion demonstrate that A$^2$-Edit consistently outperforms existing approaches across all metrics, providing a new and efficient solution for arbitrary object editing.
Abstract (translated)
我们提出了一种名为\textbf{A$^2$-Edit}的统一修补框架，该框架针对任意对象类别设计，允许用户仅通过一个粗略掩码就能将目标区域替换为参考对象。为了应对现有数据集中严重同质化和有限类别覆盖的问题，我们构建了一个大规模、多类别的数据集\textbf{UniEdit-500K}，它包括8个主要类别，209个细粒度子类别以及总共500,104对图像。这种丰富的类别多样性给模型带来了新的挑战，需要模型自动学习不同类别间的语义关系和区分性。为此，我们引入了\textbf{Transformer混合}模块，该模块通过动态专家选择来进行各种对象类别的差异化建模，并通过专家之间的协作进一步增强跨类别的语义传递和泛化能力。此外，我们还提出了一种名为\textbf{掩码退火训练策略}(MATS)的方法，在训练过程中逐步放松对掩码精度的要求，减少模型对准确掩码的依赖并提高其在各种编辑任务中的鲁棒性。在VITON-HD和AnyInsertion等基准测试上的广泛实验表明，A$^2$-Edit在所有指标上都始终优于现有方法，为任意对象编辑提供了一种新的高效解决方案。
URL
https://arxiv.org/abs/2603.10685
PDF
-
Overcoming Visual Clutter in Vision Language Action Models via Concept-Gated Visual Distillation
2026-03-11 02:21:02Sangmim Song, Sarath Kodagoda, Marc Carmichael, Karthick ThiyagarajanAbstract
Vision-Language-Action (VLA) models demonstrate impressive zero-shot generalization but frequently suffer from a "Precision-Reasoning Gap" in cluttered environments. This failure is driven by background-induced feature dilution, where high-frequency semantic noise corrupts the geometric grounding required for precise manipulation. To bridge this gap, we propose Concept-Gated Visual Distillation (CGVD), a training-free, model-agnostic inference framework that stabilizes VLA policies. CGVD operates by parsing instructions into safe and distractor sets, utilizing a two-layer target refinement process--combining cross-validation and spatial disambiguation--to explicitly penalize false positives and isolate genuine manipulation targets. We then process the scene via Fourier-based inpainting, generating a clean observation that actively suppresses semantic distractors while preserving critical spatial geometry and visual proprioception. Extensive evaluations in highly cluttered manipulation tasks demonstrate that CGVD prevents performance collapse. In environments with dense semantic distractors, our method significantly outperforms state-of-the-art baselines, achieving a 77.5% success rate compared to the baseline's 43.0%. By enforcing strict attribute adherence, CGVD establishes inference-time visual distillation as a critical prerequisite for robust robotic manipulation in the clutter.
Abstract (translated)
视觉-语言-行动（VLA）模型在零样本泛化方面表现出色，但在杂乱环境中往往会出现“精度推理差距”。这种失败主要是由于背景诱导的特征稀释所导致，其中高频语义噪声会破坏精确操作所需的几何基础。为了弥合这一差距，我们提出了概念门控视觉蒸馏（Concept-Gated Visual Distillation, CGVD），这是一种无需训练、模型无关的推理框架，用于稳定VLA策略。 CGVD通过将指令解析为安全集和干扰集，并利用两层目标细化过程——结合交叉验证与空间消歧——来明确惩罚假阳性并隔离真实的操作目标。然后，我们采用基于傅里叶的方法进行场景处理，在抑制语义干扰的同时保持关键的空间几何结构和视觉自我感知。在高度杂乱的操作任务中的广泛评估表明，CGVD可以防止性能崩溃。 在密集的语义干扰环境中，我们的方法显著优于最先进的基线模型，成功率达到了77.5%，而基线的成功率仅为43.0%。通过强制执行严格的属性遵守，CGVD建立了推理时间视觉蒸馏作为机器人杂乱环境操作稳健性的关键前提条件。
URL
https://arxiv.org/abs/2603.10340
PDF
-
From Ideal to Real: Stable Video Object Removal under Imperfect Conditions
2026-03-10 07:07:27Jiagao Hu, Yuxuan Chen, Fuhao Li, Zepeng Wang, Fei Wang, Daiguo Zhou, Jian LuanAbstract
Removing objects from videos remains difficult in the presence of real-world imperfections such as shadows, abrupt motion, and defective masks. Existing diffusion-based video inpainting models often struggle to maintain temporal stability and visual consistency under these challenges. We propose Stable Video Object Removal (SVOR), a robust framework that achieves shadow-free, flicker-free, and mask-defect-tolerant removal through three key designs: (1) Mask Union for Stable Erasure (MUSE), a windowed union strategy applied during temporal mask downsampling to preserve all target regions observed within each window, effectively handling abrupt motion and reducing missed removals; (2) Denoising-Aware Segmentation (DA-Seg), a lightweight segmentation head on a decoupled side branch equipped with Denoising-Aware AdaLN and trained with mask degradation to provide an internal diffusion-aware localization prior without affecting content generation; and (3) Curriculum Two-Stage Training: where Stage I performs self-supervised pretraining on unpaired real-background videos with online random masks to learn realistic background and temporal priors, and Stage II refines on synthetic pairs using mask degradation and side-effect-weighted losses, jointly removing objects and their associated shadows/reflections while improving cross-domain robustness. Extensive experiments show that SVOR attains new state-of-the-art results across multiple datasets and degraded-mask benchmarks, advancing video object removal from ideal settings toward real-world applications.
Abstract (translated)
从视频中移除对象在面对现实世界中的不完美情况（如阴影、突然移动和缺陷遮罩）时仍然颇具挑战。现有的基于扩散的视频修复模型通常难以在这种情况下保持时间稳定性和视觉一致性。为此，我们提出了SVOR（Stable Video Object Removal），这是一种稳健框架，通过三个关键设计实现无影、无闪烁且能够容忍遮罩缺陷的移除： 1. **Mask Union for Stable Erasure (MUSE)**：这是一种在时序掩码下采样过程中应用的窗口化联合策略，用于保存每个窗口内观察到的所有目标区域。这种方法有效地处理了突然移动并减少了遗漏的移除情况。 2. **Denoising-Aware Segmentation (DA-Seg)**：这是轻量级分割头，连接在一个解耦的侧支上，并装备有Denoising-Aware AdaLN，在使用遮罩退化进行训练的情况下提供内部扩散感知定位先验，同时不影响内容生成。 3. **Curriculum Two-Stage Training**：第一阶段（Stage I）执行未配对的真实背景视频上的自我监督预训练，并在线随机应用掩码以学习真实的背景和时间先验。第二阶段（Stage II）使用遮罩退化及副作用加权损失，在合成对上进行精炼，同时去除对象及其相关的阴影/反射，并提高跨域鲁棒性。 广泛的实验表明，SVOR在多个数据集和遮罩降级基准测试中达到了新的最先进水平，从而将视频对象移除从理想设置推进到了现实世界的应用场景。
URL
https://arxiv.org/abs/2603.09283
PDF
-
DogWeave: High-Fidelity 3D Canine Reconstruction from a Single Image via Normal Fusion and Conditional Inpainting
2026-03-08 03:35:49Shufan Sun, Chenchen Wang, Zongfu YuAbstract
Monocular 3D animal reconstruction is challenging due to complex articulation, self-occlusion, and fine-scale details such as fur. Existing methods often produce distorted geometry and inconsistent textures due to the lack of articulated 3D supervision and limited availability of back-view images in 2D datasets, which makes reconstructing unobserved regions particularly difficult. To address these limitations, we propose DogWeave, a model-based framework for reconstructing high-fidelity 3D canine models from a single RGB image. DogWeave improves geometry by refining a coarsely-initiated parametric mesh into a detailed SDF representation through multi-view normal field optimization using diffusion-enhanced normals. It then generates view-consistent textures through conditional partial inpainting guided by structure and style cues, enabling realistic reconstruction of unobserved regions. Using only about 7,000 dog images processed via our 2D pipeline for training, DogWeave produces complete, realistic 3D models and outperforms state-of-the-art single image to 3d reconstruction methods in both shape accuracy and texture realism for canines.
Abstract (translated)
单目三维动物重建由于复杂的关节运动、自遮挡以及如毛发这样的细小特征，极具挑战性。现有方法通常会因缺乏关节运动的三维监督和二维数据集中背面视角图像的稀缺而产生几何失真和纹理不一致的问题，这使得难以重建未观察到的区域。为解决这些问题，我们提出了DogWeave——一种基于模型的框架，用于从单张RGB图像中重构高质量的3D犬类模型。 DogWeave通过多视图法线场优化（利用扩散增强的法线）来细化初始参数化网格，从而改进几何形状，并将其转化为详细的SDF表示。随后，它根据结构和风格提示生成一致的纹理，通过条件部分修复完成未观察区域的真实重建工作。仅使用我们处理过的约7,000张犬类图像（经过2D管道训练），DogWeave能够产生完整且真实的3D模型，并在犬类形状准确度和纹理真实感方面超越了最先进的单图到三维重建方法。
URL
https://arxiv.org/abs/2603.07441
PDF
-
Latent Generative Models with Tunable Complexity for Compressed Sensing and other Inverse Problems
2026-03-07 21:41:46Sean Gunn, Jorio Cocola, Oliver De Candido, Vaggos Chatziafratis, Paul HandAbstract
Generative models have emerged as powerful priors for solving inverse problems. These models typically represent a class of natural signals using a single fixed complexity or dimensionality. This can be limiting: depending on the problem, a fixed complexity may result in high representation error if too small, or overfitting to noise if too large. We develop tunable-complexity priors for diffusion models, normalizing flows, and variational autoencoders, leveraging nested dropout. Across tasks including compressed sensing, inpainting, denoising, and phase retrieval, we show empirically that tunable priors consistently achieve lower reconstruction errors than fixed-complexity baselines. In the linear denoising setting, we provide a theoretical analysis that explicitly characterizes how the optimal tuning parameter depends on noise and model structure. This work demonstrates the potential of tunable-complexity generative priors and motivates both the development of supporting theory and their application across a wide range of inverse problems.
Abstract (translated)
生成模型已经作为解决逆向问题的强大先验条件而出现。这些模型通常使用单一固定的复杂度或维度来表示一类自然信号。然而，这可能具有局限性：根据具体的问题，固定不变的复杂度可能会导致在复杂度过低时产生高代表误差，或者在复杂度过高时过度拟合噪声。我们开发了可调复杂度先验条件用于扩散模型、归一化流和变分自编码器，并利用嵌套 dropout 技术实现。在包括压缩感知、图像修复、去噪以及相位检索在内的多项任务中，实验结果表明，可调的先验条件始终比固定复杂度的基础模型获得更低的重建误差。 在线性去噪设定下，我们提供了理论分析，明确说明了最优调整参数如何依赖于噪声水平和模型结构。这项工作展示了可调复杂度生成先验的巨大潜力，并激发了开发支持理论以及将其应用于广泛逆向问题的研究动力。
URL
https://arxiv.org/abs/2603.07357
PDF
-
3D CBCT Artefact Removal Using Perpendicular Score-Based Diffusion Models
2026-03-06 14:04:04Susanne Schaub, Florentin Bieder, Matheus L. Oliveira, Yulan Wang, Dorothea Dagassan-Berndt, Michael M. Bornstein, Philippe C. CattinAbstract
Cone-beam computed tomography (CBCT) is a widely used 3D imaging technique in dentistry, offering high-resolution images while minimising radiation exposure for patients. However, CBCT is highly susceptible to artefacts arising from high-density objects such as dental implants, which can compromise image quality and diagnostic accuracy. To reduce artefacts, implant inpainting in the sequence of projections plays a crucial role in many artefact reduction approaches. Recently, diffusion models have achieved state-of-the-art results in image generation and have widely been applied to image inpainting tasks. However, to our knowledge, existing diffusion-based methods for implant inpainting operate on independent 2D projections. This approach neglects the correlations among individual projections, resulting in inconsistencies in the reconstructed images. To address this, we propose a 3D dental implant inpainting approach based on perpendicular score-based diffusion models, each trained in two different planes and operating in the projection domain. The 3D distribution of the projection series is modelled by combining the two 2D score-based diffusion models in the sampling scheme. Our results demonstrate the method's effectiveness in producing high-quality, artefact-reduced 3D CBCT images, making it a promising solution for improving clinical imaging.
Abstract (translated)
锥束计算机断层扫描（CBCT）是牙科中广泛使用的三维成像技术，它能提供高分辨率的图像同时将患者的辐射暴露降至最低。然而，CBCT极易受到诸如牙齿植入物等高密度物体引起的伪影影响，这会降低图像质量和诊断准确性。为了减少这些伪影，序列投影中的植入体修复在许多伪影减少方法中扮演着关键角色。最近，扩散模型在图像生成方面取得了最先进的成果，并被广泛应用于图像修复任务中。然而，据我们所知，现有的基于扩散的方法对于植入体修复仅操作于独立的二维投影上。这种方法忽视了各个投影之间的关联性，在重建图像时会产生不一致性。为了解决这一问题，我们提出了一种基于垂直评分扩散模型的三维牙科植入体修复方法，该方法在两个不同的平面上进行训练，并且是在投影域内操作的。通过结合这两个二维评分扩散模型来构建投影序列的三维分布，在采样方案中实现了这一点。我们的结果显示，这种方法能够有效生成高质量、伪影减少的3D CBCT图像，有望成为改善临床成像的一个有前景的解决方案。
URL
https://arxiv.org/abs/2603.06300
PDF
-
Longitudinal Lesion Inpainting in Brain MRI via 3D Region Aware Diffusion
2026-03-05 21:34:47Zahra Karimaghaloo, Dumitru Fetco, Haz-Edine Assemlal, Hassan Rivaz, Douglas L. ArnoldAbstract
Accurate longitudinal analysis of brain MRI is often hindered by evolving lesions, which bias automated neuroimaging pipelines. While deep generative models have shown promise in inpainting these lesions, most existing methods operate cross-sectionally or lack 3D anatomical continuity. We present a novel pseudo-3D longitudinal inpainting framework based on Denoising Diffusion Probabilistic Models (DDPM). Our approach utilizes multi-channel conditioning to incorporate longitudinal context from distinct visits (t_1, t_2) and extends Region-Aware Diffusion (RAD) to the medical domain, focusing the generative process on pathological regions without altering surrounding healthy tissue. We evaluated our model against state-of-the-art baselines on longitudinal brain MRI from 93 patients. Our model significantly outperforms the leading baseline (FastSurfer-LIT) in terms of perceptual fidelity, reducing the Learned Perceptual Image Patch Similarity (LPIPS) distance from 0.07 to 0.03 while effectively eliminating inter-slice discontinuities. Furthermore, our model demonstrates high longitudinal stability with a Temporal Fidelity Index of 1.024, closely approaching the ideal value of 1.0 and substantially narrowing the gap compared to LIT's TFI of 1.22. Notably, the RAD mechanism provides a substantial gain in efficiency; our framework achieves an average processing time of 2.53 min per volume, representing approximately 10x speedup over the 24.30 min required by LIT. By leveraging longitudinal priors and region-specific denoising, our framework provides a highly reliable and efficient preprocessing step for the study of progressive neurodegenerative diseases. A derivative dataset consisting of 93 pre-processed scans used for testing will be available upon request after acceptance. Code will be released upon acceptance.
Abstract (translated)
准确的脑部MRI纵向分析常常因病变随时间演变而受阻，这会偏移自动化神经影像管线。虽然深度生成模型在修复这些病灶方面显示出潜力，但大多数现有方法要么是横断面操作，要么缺乏3D解剖连续性。我们提出了一种基于去噪扩散概率模型（DDPM）的新型伪三维纵向修复框架。我们的方法利用多通道条件，整合来自不同时间点(t_1, t_2)的纵向信息，并将区域感知扩散(RAD)机制扩展至医学领域，专注于病变区域的生成过程，而不改变周围的健康组织。 我们在93名患者的纵向脑部MRI上评估了该模型，与最先进的基准方法相比，我们的模型在感知保真度方面显著优于FastSurfer-LIT基线模型，将学习到的感知图像块相似性(LPIPS)距离从0.07降至0.03，并有效消除了层间不连续性。此外，我们的模型展示了高度的时间稳定性，时间保真指数为1.024，接近理想值1.0，并显著缩小了与LIT TFI(1.22)的差距。 值得注意的是，RAD机制提供了巨大的效率提升；我们的框架实现了每个体积平均处理时间为2.53分钟，相比LIT所需的24.30分钟大约快了十倍。通过利用纵向先验和区域特异性去噪，我们提供的预处理步骤对于进行渐进性神经退行性疾病的研究是高度可靠且高效的。 在被接受后，我们将提供一个包含93个预处理扫描的衍生数据集供测试使用，并将在获得许可后发布代码。
URL
https://arxiv.org/abs/2603.05693
PDF
-
Transformer-Based Inpainting for Real-Time 3D Streaming in Sparse Multi-Camera Setups
2026-03-05 18:59:59Leif Van Holland, Domenic Zingsheim, Mana Takhsha, Hannah Dr\"oge, Patrick Stotko, Markus Plack, Reinhard KleinAbstract
High-quality 3D streaming from multiple cameras is crucial for immersive experiences in many AR/VR applications. The limited number of views - often due to real-time constraints - leads to missing information and incomplete surfaces in the rendered images. Existing approaches typically rely on simple heuristics for the hole filling, which can result in inconsistencies or visual artifacts. We propose to complete the missing textures using a novel, application-targeted inpainting method independent of the underlying representation as an image-based post-processing step after the novel view rendering. The method is designed as a standalone module compatible with any calibrated multi-camera system. For this we introduce a multi-view aware, transformer-based network architecture using spatio-temporal embeddings to ensure consistency across frames while preserving fine details. Additionally, our resolution-independent design allows adaptation to different camera setups, while an adaptive patch selection strategy balances inference speed and quality, allowing real-time performance. We evaluate our approach against state-of-the-art inpainting techniques under the same real-time constraints and demonstrate that our model achieves the best trade-off between quality and speed, outperforming competitors in both image and video-based metrics.
Abstract (translated)
高质量的3D流媒体对于许多AR/VR应用中的沉浸式体验至关重要。由于实时限制，来自多个摄像头的有限视图会导致渲染图像中缺失信息和不完整的表面。现有方法通常依赖简单的启发式填充策略来填补这些空白，这可能会导致视觉上的不一致或伪影。我们提出了一种新颖的应用特定的修补方法，用于完成缺失纹理，并将其作为基于图像的后期处理步骤，在新视角渲染后独立于底层表示进行操作。该方法设计为一个与任何校准多摄像头系统兼容的独立模块。 为此，我们引入了一个多视图感知、基于变压器的网络架构，使用时空嵌入以确保帧间的一致性同时保留细节点。此外，我们的分辨率无关设计允许适应不同的摄像机设置，而自适应补丁选择策略则平衡了推断速度和质量，在保证实时性能的同时提供出色的表现。 我们在与最新技术相同的实时限制下评估了我们的方法，并展示了模型在质量和速度之间的最佳权衡，无论是在基于图像还是视频的度量标准上均优于竞争对手。
URL
https://arxiv.org/abs/2603.05507
PDF
-
Mask-aware inference with State-Space Models
2026-03-04 19:55:48Ignasi Mas, Ramon Morros, Javier-Ruiz Hidalgo, Ivan HuertaAbstract
Many real-world computer vision tasks, such as depth completion, must handle inputs with arbitrarily shaped regions of missing or invalid data. For Convolutional Neural Networks (CNNs), Partial Convolutions solved this by a mask-aware re-normalization conditioned only on valid pixels. Recently, State Space Models (SSMs) like Mamba have emerged, offering high performance with linear complexity. However, these architectures lack an inherent mechanism for handling such arbitrarily shaped invalid data at inference time. To bridge this gap, we introduce Partial Vision Mamba (PVM), a novel architectural component that ports the principles of partial operations to the Mamba backbone. We also define a series of rules to design architectures using PVM. We show the efficacy and generalizability of our approach in the tasks of depth completion, image inpainting, and classification with invalid data.
Abstract (translated)
许多现实世界的计算机视觉任务，如深度完成（depth completion），需要处理具有任意形状的缺失或无效数据区域的输入。对于卷积神经网络（CNN）而言，部分卷积通过使用仅基于有效像素进行掩码感知重归一化解决了这一问题。最近，像Mamba这样的状态空间模型(SSMs)已经出现，提供了高效率和线性复杂度的性能表现。然而，这些架构在推理时缺乏处理此类任意形状无效数据的内在机制。 为弥补这一不足，我们引入了部分视觉曼巴（Partial Vision Mamba, PVM），这是一个将部分操作原则移植到Mamba骨干网络上的新型架构组件。同时，我们也定义了一系列规则来使用PVM设计架构。我们在深度完成、图像修复以及带有无效数据的分类任务中展示了我们的方法的有效性和泛化能力。
URL
https://arxiv.org/abs/2603.04568
PDF
-
HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Images
2026-03-02 18:59:36Yichen Liu, Donghao Zhou, Jie Wang, Xin Gao, Guisheng Liu, Jiatong Li, Quanwei Zhang, Qiang Lyu, Lanqing Guo, Shilei Wen, Weiqiang Wang, Pheng-Ann HengAbstract
Human-product images, which showcase the integration of humans and products, play a vital role in advertising, e-commerce, and digital marketing. The essential challenge of generating such images lies in ensuring the high-fidelity preservation of product details. Among existing paradigms, reference-based inpainting offers a targeted solution by leveraging product reference images to guide the inpainting process. However, limitations remain in three key aspects: the lack of diverse large-scale training data, the struggle of current models to focus on product detail preservation, and the inability of coarse supervision for achieving precise guidance. To address these issues, we propose HiFi-Inpaint, a novel high-fidelity reference-based inpainting framework tailored for generating human-product images. HiFi-Inpaint introduces Shared Enhancement Attention (SEA) to refine fine-grained product features and Detail-Aware Loss (DAL) to enforce precise pixel-level supervision using high-frequency maps. Additionally, we construct a new dataset, HP-Image-40K, with samples curated from self-synthesis data and processed with automatic filtering. Experimental results show that HiFi-Inpaint achieves state-of-the-art performance, delivering detail-preserving human-product images.
Abstract (translated)
人与产品图像展示了人类和产品的结合，在广告、电子商务和数字营销中扮演着重要角色。生成这类图片的关键挑战在于确保高保真地保留产品细节。现有方法中，基于参考的修复（inpainting）提供了一种通过利用产品参考图来引导修复过程的目标解决方案。然而，这一方法在三个方面仍存在局限：缺乏多样化的大规模训练数据、现有模型难以专注于保持产品细节以及粗略监督无法实现精确指导。 为了解决这些问题，我们提出了HiFi-Inpaint，这是一个新型的高保真度基于参考的修补框架，专用于生成人与产品图像。HiFi-Inpaint 引入了共享增强注意力（SEA），以精炼细粒度的产品特征，并引入细节感知损失（DAL）来使用高频图强制执行精确的像素级监督。 此外，我们构建了一个新的数据集HP-Image-40K，该数据集包含从自我合成的数据中挑选出来的样本并经过自动过滤处理。实验结果表明，HiFi-Inpaint 达到了最先进的性能水平，并能够生成保持细节的人与产品图像。
URL
https://arxiv.org/abs/2603.02210
PDF
-
State-Action Inpainting Diffuser for Continuous Control with Delay
2026-03-02 07:28:27Dongqi Han, Wei Wang, Enze Zhang, Dongsheng LiAbstract
Signal delay poses a fundamental challenge in continuous control and reinforcement learning (RL) by introducing a temporal gap between interaction and perception. Current solutions have largely evolved along two distinct paradigms: model-free approaches which utilize state augmentation to preserve Markovian properties, and model-based methods which focus on inferring latent beliefs via dynamics modeling. In this paper, we bridge these perspectives by introducing State-Action Inpainting Diffuser (SAID), a framework that integrates the inductive bias of dynamics learning with the direct decision-making capability of policy optimization. By formulating the problem as a joint sequence inpainting task, SAID implicitly captures environmental dynamics while directly generating consistent plans, effectively operating at the intersection of model-based and model-free paradigms. Crucially, this generative formulation allows SAID to be seamlessly applied to both online and offline RL. Extensive experiments on delayed continuous control benchmarks demonstrate that SAID achieves state-of-the-art and robust performance. Our study suggests a new methodology to advance the field of RL with delay.
Abstract (translated)
信号延迟在连续控制和强化学习（RL）中构成了一个根本性的挑战，因为它会在交互与感知之间引入时间间隔。目前的解决方案主要沿着两个不同的范式发展：无模型方法通过状态扩充来保持马尔可夫性质；基于模型的方法则侧重于通过动态建模推断潜在信念。在本文中，我们提出了一种新框架——状态-动作填充扩散器（State-Action Inpainting Diffuser, SAID），该框架将动力学学习的归纳偏差与策略优化的直接决策能力相结合，从而弥合了这两种视角之间的差距。通过将其问题表述为联合序列填充任务，SAID可以隐式捕捉环境动态并直接生成一致性的计划，在基于模型和无模型范式的交界处运作。尤为重要的是，这种生成性方法使得SAID能够无缝应用于在线和离线RL中。 在延迟连续控制基准测试中的广泛实验表明，SAID实现了最先进的性能且表现稳定。我们的研究提出了一种新的方法论来推进有延迟情况下的强化学习领域的发展。
URL
https://arxiv.org/abs/2603.01553
PDF
-
Reparameterized Tensor Ring Functional Decomposition for Multi-Dimensional Data Recovery
2026-03-01 10:19:03Yangyang Xu, Junbo Ke, You-Wei Wen, Chao WangAbstract
Tensor Ring (TR) decomposition is a powerful tool for high-order data modeling, but is inherently restricted to discrete forms defined on fixed meshgrids. In this work, we propose a TR functional decomposition for both meshgrid and non-meshgrid data, where factors are parameterized by Implicit Neural Representations (INRs). However, optimizing this continuous framework to capture fine-scale details is intrinsically difficult. Through a frequency-domain analysis, we demonstrate that the spectral structure of TR factors determines the frequency composition of the reconstructed tensor and limits the high-frequency modeling capacity. To mitigate this, we propose a reparameterized TR functional decomposition, in which each TR factor is a structured combination of a learnable latent tensor and a fixed basis. This reparameterization is theoretically shown to improve the training dynamics of TR factor learning. We further derive a principled initialization scheme for the fixed basis and prove the Lipschitz continuity of our proposed model. Extensive experiments on image inpainting, denoising, super-resolution, and point cloud recovery demonstrate that our method achieves consistently superior performance over existing approaches. Code is available at this https URL.
Abstract (translated)
张量环（TR）分解是高阶数据分析的强大工具，但本质上局限于在固定网格上的离散形式。在这项工作中，我们提出了一种适用于网格数据和非网格数据的TR函数分解方法，其中因子由隐式神经表示(INR)参数化。然而，优化这种连续框架以捕捉细粒度细节在本质上是困难的。通过频域分析，我们展示了TR因子的谱结构决定了重构张量的频率构成，并限制了高频建模的能力。为了解决这个问题，我们提出了一个重新参数化的TR函数分解方法，在这种方法中每个TR因子是由可学习的潜在张量和固定基底有结构地组合而成。理论上证明，这种重参数化可以改善TR因子的学习动态过程。此外，我们还推导出了一种用于固定基底的原则性初始化方案，并证明了所提出模型的Lipschitz连续性。在图像修复、去噪、超分辨率和点云恢复等广泛的实验中表明，我们的方法比现有方法达到了一致且优越的表现水平。代码可在提供的URL处获取。
URL
https://arxiv.org/abs/2603.01034
PDF
-
CamDirector: Towards Long-Term Coherent Video Trajectory Editing
2026-02-27 19:23:18Zhihao Shi, Kejia Yin, Weilin Wan, Yuhongze Zhou, Yuanhao Yu, Xinxin Zuo, Qiang Sun, Juwei LuAbstract
Video (camera) trajectory editing aims to synthesize new videos that follow user-defined camera paths while preserving scene content and plausibly inpainting previously unseen regions, upgrading amateur footage into professionally styled videos. Existing VTE methods struggle with precise camera control and long-range consistency because they either inject target poses through a limited-capacity embedding or rely on single-frame warping with only implicit cross-frame aggregation in video diffusion models. To address these issues, we introduce a new VTE framework that 1) explicitly aggregates information across the entire source video via a hybrid warping scheme. Specifically, static regions are progressively fused into a world cache then rendered to target camera poses, while dynamic regions are directly warped; their fusion yields globally consistent coarse frames that guide refinement. 2) processes video segments jointly with their history via a history-guided autoregressive diffusion model, while the world cache is incrementally updated to reinforce already inpainted content, enabling long-term temporal coherence. Finally, we present iPhone-PTZ, a new VTE benchmark with diverse camera motions and large trajectory variations, and achieve state-of-the-art performance with fewer parameters.
Abstract (translated)
视频（相机）轨迹编辑的目标是根据用户定义的摄像机路径合成新的视频，同时保持场景内容并合理地填补之前未见的区域，从而将业余拍摄的素材升级为专业风格的视频。现有的VTE方法在精确控制摄像机和长跨度一致性方面存在困难：它们要么通过有限容量的嵌入方式注入目标姿态，要么依赖于仅具有隐式跨帧聚合功能的单帧扭曲技术，在视频扩散模型中使用这些方法时效果不佳。 为了应对这些问题，我们引入了一种新的VTE框架： 1) 该框架明确地在整个源视频上进行信息聚合，采用了混合扭曲方案。具体来说，静态区域逐步融合到一个世界缓存区中，并根据目标摄像机姿态进行渲染；动态区域则直接被扭曲处理；两者融合后生成全局一致的粗略帧，这些帧引导进一步细化。 2) 该框架通过历史导向的自回归扩散模型同时处理视频片段及其历史记录。与此同时，世界缓存会逐步更新，以强化已经填充的内容，从而实现长时间的一致性。 最后，我们介绍了iPhone-PTZ，这是一个新的VTE基准测试平台，它包括多样化的相机运动和大的轨迹变化，并且通过更少的参数实现了最先进的性能。
URL
https://arxiv.org/abs/2603.02256
PDF
-
ProjFlow: Projection Sampling with Flow Matching for Zero-Shot Exact Spatial Motion Control
2026-02-26 08:29:25Akihisa Watanabe, Qing Yu, Edgar Simo-Serra, Kent FujiwaraAbstract
Generating human motion with precise spatial control is a challenging problem. Existing approaches often require task-specific training or slow optimization, and enforcing hard constraints frequently disrupts motion naturalness. Building on the observation that many animation tasks can be formulated as a linear inverse problem, we introduce ProjFlow, a training-free sampler that achieves zero-shot, exact satisfaction of linear spatial constraints while preserving motion realism. Our key advance is a novel kinematics-aware metric that encodes skeletal topology. This metric allows the sampler to enforce hard constraints by distributing corrections coherently across the entire skeleton, avoiding the unnatural artifacts of naive projection. Furthermore, for sparse inputs, such as filling in long gaps between a few keyframes, we introduce a time-varying formulation using pseudo-observations that fade during sampling. Extensive experiments on representative applications, motion inpainting, and 2D-to-3D lifting, demonstrate that ProjFlow achieves exact constraint satisfaction and matches or improves realism over zero-shot baselines, while remaining competitive with training-based controllers.
Abstract (translated)
生成具有精确空间控制的人体运动是一个挑战性问题。现有方法通常需要特定任务的训练或缓慢优化，而强加硬约束常常会破坏动作的自然流畅性。基于观察到许多动画任务可以被表述为线性逆向问题，我们引入了ProjFlow，这是一种无需训练的采样器，能够在不牺牲运动真实感的情况下实现对线性空间约束的零样本、精确满足。我们的关键进展是一种新颖的动力学感知度量方法，该方法编码骨骼拓扑结构。这种度量使得采样器能够通过在整个骨架上合理地分布修正来强制执行硬约束，从而避免了简单投影所产生的不自然效果。此外，在处理稀疏输入（如填充几个关键帧之间的长时间间隙）时，我们提出了一种使用伪观察的随时间变化的形式化方法，该方法在采样过程中逐渐淡化。 通过具代表性的应用场景、运动插补和2D到3D提升进行的广泛实验表明，ProjFlow能够精确满足约束条件，并且在不牺牲自然流畅性的情况下，在零样本基准上保持或提高了真实感。此外，ProjFlow仍然与基于训练的控制器具有竞争力。
URL
https://arxiv.org/abs/2602.22742
PDF
-
Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context
2026-02-25 14:09:03JiaKui Hu, Jialun Liu, Liying Yang, Xinliang Zhang, Kaiwen Li, Shuang Zeng, Yuanwei Li, Haibin Huang, Chi Zhang, Yanye LuAbstract
Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control.
Abstract (translated)
场景一致的视频生成旨在基于相机轨迹探索三维场景并创建相应的视频。以往的方法依赖于使用外部内存来保持一致性或迭代地进行3D重建和图像修复，但这些方法在推理过程中会因中间输出错误、不可微分过程以及模型分离而积累误差。为克服这些限制，我们引入了“几何作为上下文”（geometry-as-context）的概念。该方法利用自回归相机控制的视频生成模型迭代地完成以下步骤：(1) 估计当前视图所需的三维重建几何形状；(2) 模拟并恢复由3D场景渲染的新视角图像。 在此多任务框架下，我们开发了相机门控注意模块（camera gated attention module），以增强模型有效利用相机姿态的能力。在训练阶段，使用文本上下文来确定生成的输出是几何信息还是RGB图像。为了确保模型在推断过程中可以仅生成RGB输出，我们在交错的文本-图像-几何训练序列中随机丢弃几何上下文。 该方法已在单向和往返轨迹的场景视频生成上进行了测试，并且结果表明其在保持场景一致性和相机控制方面优于先前的方法。
URL
https://arxiv.org/abs/2602.21929
PDF
-
SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model
2026-02-25 11:47:00Guibin Chen, Dixuan Lin, Jiangping Yang, Youqiang Zhang, Zhengcong Fei, Debang Li, Sheng Chen, Chaofeng Ao, Nuo Pang, Yiming Wang, Yikun Dou, Zheng Chen, Mingyuan Fan, Tuanhui Li, Mingshan Chang, Hao Zhang, Xiaopeng Sun, Jingtao Xu, Yuqiang Xie, Jiahua Wang, Zhiheng Xu, Weiming Xiong, Yuzhe Jin, Baoxuan Gu, Binjie Mao, Yunjie Yu, Jujie He, Yuhao Feng, Shiwen Tu, Chaojie Wang, Rui Yan, Wei Shen, Jingchen Wu, Peng Zhao, Xuanyue Zhong, Zhuangzhuang Liu, Kaifei Wang, Fuxiang Zhang, Weikai Xu, Wenyan Liu, Binglu Zhang, Yu Shen, Tianhui Xiong, Bin Peng, Liang Zeng, Xuchen Song, Haoxiang Guo, Peiyu Wang, Yahui ZhouAbstract
SkyReels V4 is a unified multi modal video foundation model for joint video audio generation, inpainting, and editing. The model adopts a dual stream Multimodal Diffusion Transformer (MMDiT) architecture, where one branch synthesizes video and the other generates temporally aligned audio, while sharing a powerful text encoder based on the Multimodal Large Language Models (MMLM). SkyReels V4 accepts rich multi modal instructions, including text, images, video clips, masks, and audio references. By combining the MMLMs multi modal instruction following capability with in context learning in the video branch MMDiT, the model can inject fine grained visual guidance under complex conditioning, while the audio branch MMDiT simultaneously leverages audio references to guide sound generation. On the video side, we adopt a channel concatenation formulation that unifies a wide range of inpainting style tasks, such as image to video, video extension, and video editing under a single interface, and naturally extends to vision referenced inpainting and editing via multi modal prompts. SkyReels V4 supports up to 1080p resolution, 32 FPS, and 15 second duration, enabling high fidelity, multi shot, cinema level video generation with synchronized audio. To make such high resolution, long-duration generation computationally feasible, we introduce an efficiency strategy: Joint generation of low resolution full sequences and high-resolution keyframes, followed by dedicated super-resolution and frame interpolation models. To our knowledge, SkyReels V4 is the first video foundation model that simultaneously supports multi-modal input, joint video audio generation, and a unified treatment of generation, inpainting, and editing, while maintaining strong efficiency and quality at cinematic resolutions and durations.
Abstract (translated)
SkyReels V4 是一个统一的多模态视频基础模型，用于联合生成视频和音频、修复和编辑。该模型采用双流多模态扩散变换器（MMDiT）架构，其中一条分支合成视频，另一条分支生成时间对齐的音频，并共享基于多模态大型语言模型（MMLM）的强大文本编码器。SkyReels V4 接受丰富的多模态指令，包括文本、图像、视频片段、遮罩和音频参考。通过结合 MMLM 的多模态指令跟随能力和视频分支 MMDiT 中的上下文学习能力，该模型可以在复杂的条件下注入精细的视觉引导，而音频分支 MMDiT 同时利用音频参考来指导声音生成。在视频方面，我们采用通道级联的形式，统一了包括图像到视频、视频扩展和视频编辑在内的广泛修复风格任务，并通过多模态提示自然地扩展到了基于视觉参考的修复和编辑。SkyReels V4 支持高达 1080p 分辨率、32 FPS 和 15 秒时长，能够生成高保真度、多镜头、电影级别的视频并配有同步音频。为了使这种高分辨率、长时间生成在计算上可行，我们引入了一个效率策略：先联合生成低分辨率完整序列和高清关键帧，然后使用专门的超分辨率模型和帧插值模型进行后处理。据我们所知，SkyReels V4 是第一个同时支持多模态输入、联合视频音频生成，并统一处理生成、修复和编辑的视频基础模型，且在电影级分辨率和时长下保持了强大的效率和质量。
URL
https://arxiv.org/abs/2602.21818
PDF
-
BoxSplitGen: A Generative Model for 3D Part Bounding Boxes in Varying Granularity
2026-02-24 08:15:25Juil Koo, Wei-Tung Lin, Chanho Park, Chanhyeok Park, Minhyuk SungAbstract
Human creativity follows a perceptual process, moving from abstract ideas to finer details during creation. While 3D generative models have advanced dramatically, models specifically designed to assist human imagination in 3D creation -- particularly for detailing abstractions from coarse to fine -- have not been explored. We propose a framework that enables intuitive and interactive 3D shape generation by iteratively splitting bounding boxes to refine the set of bounding boxes. The main technical components of our framework are two generative models: the box-splitting generative model and the box-to-shape generative model. The first model, named BoxSplitGen, generates a collection of 3D part bounding boxes with varying granularity by iteratively splitting coarse bounding boxes. It utilizes part bounding boxes created through agglomerative merging and learns the reverse of the merging process -- the splitting sequences. The model consists of two main components: the first learns the categorical distribution of the box to be split, and the second learns the distribution of the two new boxes, given the set of boxes and the indication of which box to split. The second model, the box-to-shape generative model, is trained by leveraging the 3D shape priors learned by an existing 3D diffusion model while adapting the model to incorporate bounding box conditioning. In our experiments, we demonstrate that the box-splitting generative model outperforms token prediction models and the inpainting approach with an unconditional diffusion model. Also, we show that our box-to-shape model, based on a state-of-the-art 3D diffusion model, provides superior results compared to a previous model.
Abstract (translated)
人类的创造力遵循一种感知过程，从抽象概念逐步细化到具体细节。虽然三维生成模型取得了显著进步，但专门用于协助人类在三维创作中将粗糙的概念转化为精细细节的设计尚未得到充分探索。我们提出了一种框架，通过迭代地划分边界框来生成直观且交互式的三维形状。该框架的主要技术组件是两个生成式模型：一个是边界框分割生成模型（BoxSplitGen），另一个是从边界框到形状的生成模型。 第一个模型，名为BoxSplitGen，通过迭代地分割粗糙的边界框产生一系列不同粒度的3D部件边界框集合。它利用聚合合并产生的部分边界框，并学习与其相对的过程——即分割序列。该模型包含两个主要组成部分：第一部分负责从一组给定的边界框中预测要被分裂的具体边界框，而第二部分则基于指示哪些边界框需要被分割的信息来生成新的子边界框。 第二个模型是从现有的三维扩散模型所学得的形状先验出发训练而成的，同时该模型也适应性地融入了边界框条件。通过这种方式，它可以更有效地将边界框转换为详细的三维形状。 在我们的实验中，我们展示了边界框分割生成模型优于基于令牌预测和无条件扩散模型的方法。此外，我们也证明了基于先进的3D扩散模型构建的边界框到形状模型相比之前的方法提供了更好的结果。
URL
https://arxiv.org/abs/2602.20666
PDF
-
AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents
2026-02-24 05:37:35Jiaqi Wu, Yuchen Zhou, Muduo Xu, Zisheng Liang, Simiao Ren, Jiayu Xue, Meige Yang, Siying Chen, Jingheng HuanAbstract
We present AIForge-Doc, the first dedicated benchmark targeting exclusively diffusion-model-based inpainting in financial and form documents with pixel-level annotation. Existing document forgery datasets rely on traditional digital editing tools (e.g., Adobe Photoshop, GIMP), creating a critical gap: state-of-the-art detectors are blind to the rapidly growing threat of AI-forged document fraud. AIForge-Doc addresses this gap by systematically forging numeric fields in real-world receipt and form images using two AI inpainting APIs -- Gemini 2.5 Flash Image and Ideogram v2 Edit -- yielding 4,061 forged images from four public document datasets (CORD, WildReceipt, SROIE, XFUND) across nine languages, annotated with pixel-precise tampered-region masks in DocTamper-compatible format. We benchmark three representative detectors -- TruFor, DocTamper, and a zero-shot GPT-4o judge -- and find that all existing methods degrade substantially: TruFor achieves AUC=0.751 (zero-shot, out-of-distribution) vs. AUC=0.96 on NIST16; DocTamper achieves AUC=0.563 vs. AUC=0.98 in-distribution, with pixel-level IoU=0.020; GPT-4o achieves only 0.509 -- essentially at chance -- confirming that AI-forged values are indistinguishable to automated detectors and VLMs. These results demonstrate that AIForge-Doc represents a qualitatively new and unsolved challenge for document forensics.
Abstract (translated)
我们介绍了AIForge-Doc，这是第一个专注于使用扩散模型进行图像修复的文档伪造基准测试，该基准包含针对财务和表格文档的像素级注释。现有的文档伪造数据集依赖于传统的数字编辑工具（例如Adobe Photoshop、GIMP），这造成了一个关键差距：最先进的检测器对于日益增长的人工智能伪造文档欺诈威胁视而不见。AIForge-Doc通过使用两个AI修复API——Gemini 2.5 Flash Image和Ideogram v2 Edit，在真实世界的收据和表格图像中系统性地伪造数字字段，从而填补了这一空白。这产生了4,061张伪造的图片，这些图片来自四个公开的数据集（CORD、WildReceipt、SROIE、XFUND），涵盖九种语言，并以DocTamper兼容格式进行了像素级篡改区域标注。 我们评估了三种代表性的检测器——TruFor、DocTamper以及零样本GPT-4o法官，并发现所有现有方法在效果上都有显著下降：TruFor的AUC值为0.751（零样本，分布外）对比NIST16上的0.96；DocTamper的AUC值为0.563（分布内），与之相比其像素级IoU仅为0.020；GPT-4o仅达到0.509——基本等同于随机猜测。这些结果证实了AI伪造的价值对于自动化检测器和视觉语言模型而言是无法区分的。 这一成果表明，AIForge-Doc代表了一个文档取证领域中的全新且未解决的重大挑战。
URL
https://arxiv.org/abs/2602.20569
PDF
