Inpainting
-
OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transfer
2025-12-09 18:56:30Jessica Yin, Haozhi Qi, Youngsun Wi, Sayantan Kundu, Mike Lambeta, William Yang, Changhao Wang, Tingfan Wu, Jitendra Malik, Tess HellebrekersAbstract
Human video demonstrations provide abundant training data for learning robot policies, but video alone cannot capture the rich contact signals critical for mastering manipulation. We introduce OSMO, an open-source wearable tactile glove designed for human-to-robot skill transfer. The glove features 12 three-axis tactile sensors across the fingertips and palm and is designed to be compatible with state-of-the-art hand-tracking methods for in-the-wild data collection. We demonstrate that a robot policy trained exclusively on human demonstrations collected with OSMO, without any real robot data, is capable of executing a challenging contact-rich manipulation task. By equipping both the human and the robot with the same glove, OSMO minimizes the visual and tactile embodiment gap, enabling the transfer of continuous shear and normal force feedback while avoiding the need for image inpainting or other vision-based force inference. On a real-world wiping task requiring sustained contact pressure, our tactile-aware policy achieves a 72% success rate, outperforming vision-only baselines by eliminating contact-related failure modes. We release complete hardware designs, firmware, and assembly instructions to support community adoption.
Abstract (translated)
人类视频演示为学习机器人策略提供了丰富的训练数据，但仅靠视频无法捕捉到掌握操作技能所需的丰富接触信号。为此，我们引入了OSMO——一款开源可穿戴触觉手套，旨在促进从人到机器人的技能转移。这款手套在指尖和手掌上配备了12个三轴触觉传感器，并设计为与最先进的手部追踪方法兼容，以便于野外数据采集。 实验表明，在仅使用配备OSMO的手套收集的人类演示数据训练机器人策略的情况下，该机器人能够执行一项具有挑战性的接触密集型操作任务。通过让人类和机器人都佩戴相同的触觉手套，OSMO减少了视觉和触觉感知差距，使连续切向力和法向力反馈的传输成为可能，并且避免了图像修复或其他基于视觉的方法来推断力的需求。 在需要持续接触压力的真实世界擦拭任务中，我们的具备触觉感知能力的策略达到了72%的成功率，优于仅依赖视觉的基础模型。通过消除与接触相关的问题模式，该系统显著提高了性能。 我们发布了完整的硬件设计、固件以及组装说明，以支持社区采用这一技术。
URL
https://arxiv.org/abs/2512.08920
PDF
-
Refining Visual Artifacts in Diffusion Models via Explainable AI-based Flaw Activation Maps
2025-12-09 16:30:31Seoyeon Lee, Gwangyeol Yu, Chaewon Kim, Jonghyuk ParkAbstract
Diffusion models have achieved remarkable success in image synthesis. However, addressing artifacts and unrealistic regions remains a critical challenge. We propose self-refining diffusion, a novel framework that enhances image generation quality by detecting these flaws. The framework employs an explainable artificial intelligence (XAI)-based flaw highlighter to produce flaw activation maps (FAMs) that identify artifacts and unrealistic regions. These FAMs improve reconstruction quality by amplifying noise in flawed regions during the forward process and by focusing on these regions during the reverse process. The proposed approach achieves up to a 27.3% improvement in Fréchet inception distance across various diffusion-based models, demonstrating consistently strong performance on diverse datasets. It also shows robust effectiveness across different tasks, including image generation, text-to-image generation, and inpainting. These results demonstrate that explainable AI techniques can extend beyond interpretability to actively contribute to image refinement. The proposed framework offers a versatile and effective approach applicable to various diffusion models and tasks, significantly advancing the field of image synthesis.
Abstract (translated)
扩散模型在图像生成方面取得了显著的成功，但解决伪影和不真实区域的问题仍然是一个关键挑战。我们提出了一种名为“自我完善扩散”的新型框架，该框架通过检测这些缺陷来提升图像生成的质量。此框架使用基于可解释人工智能（XAI）的瑕疵高亮器来生成瑕疵激活图（FAMs），以识别伪影和不真实区域。在正向过程中，这些FAMs通过放大有缺陷区域中的噪声来改进重建质量；而在反向过程中，则通过聚焦于这些特定区域来进行优化。 所提出的方法在各种基于扩散的模型上实现了高达27.3%的Fréchet inception距离（FID）改善，并且在不同的数据集上均表现出强大的性能。此外，这种方法还显示出对不同任务的强大效果，包括图像生成、文本到图像生成以及修复画笔工具应用等。 这些结果表明，可解释的人工智能技术不仅可以用于提高模型的透明度和理解性，还可以主动参与图像精炼过程。所提出的框架提供了一种适用于各种扩散模型和任务的灵活有效的方法，从而在图像合成领域取得了显著进展。
URL
https://arxiv.org/abs/2512.08774
PDF
-
PaintFlow: A Unified Framework for Interactive Oil Paintings Editing and Generation
2025-12-09 12:31:00Zhangli Hu, Ye Chen, Jiajun Yao, Bingbing NiAbstract
Oil painting, as a high-level medium that blends human abstract thinking with artistic expression, poses substantial challenges for digital generation and editing due to its intricate brushstroke dynamics and stylized characteristics. Existing generation and editing techniques are often constrained by the distribution of training data and primarily focus on modifying real photographs. In this work, we introduce a unified multimodal framework for oil painting generation and editing. The proposed system allows users to incorporate reference images for precise semantic control, hand-drawn sketches for spatial structure alignment, and natural language prompts for high-level semantic guidance, while consistently maintaining a unified painting style across all outputs. Our method achieves interactive oil painting creation through three crucial technical advancements. First, we enhance the training stage with spatial alignment and semantic enhancement conditioning strategy, which map masks and sketches into spatial constraints, and encode contextual embedding from reference images and text into feature constraints, enabling object-level semantic alignment. Second, to overcome data scarcity, we propose a self-supervised style transfer pipeline based on Stroke-Based Rendering (SBR), which simulates the inpainting dynamics of oil painting restoration, converting real images into stylized oil paintings with preserved brushstroke textures to construct a large-scale paired training dataset. Finally, during inference, we integrate features using the AdaIN operator to ensure stylistic consistency. Extensive experiments demonstrate that our interactive system enables fine-grained editing while preserving the artistic qualities of oil paintings, achieving an unprecedented level of imagination realization in stylized oil paintings generation and editing.
Abstract (translated)
油画作为一种将人类抽象思维与艺术表达相结合的高级媒介，由于其复杂的笔触动态和风格化特征，在数字生成和编辑方面面临着重大挑战。现有的生成和编辑技术通常受到训练数据分布的限制，并且主要侧重于修改真实照片。在这项工作中，我们引入了一个统一的多模态框架，用于油画的生成和编辑。该系统允许用户通过参考图像进行精确语义控制、手绘草图进行空间结构对齐以及自然语言提示进行高层次语义指导的同时，在所有输出中保持一致的绘画风格。我们的方法通过三项关键技术进步实现了互动式油画创作。 首先，我们加强了训练阶段的空间对齐和语义增强条件策略，将蒙版和草图映射为空间约束，并从参考图像和文本编码上下文嵌入以形成特征约束，从而实现对象级别的语义对齐。其次，为了克服数据稀缺性的问题，我们提出了一种基于笔触基渲染（SBR）的自我监督风格迁移管道，该方法模拟了油画修复中的填色动态，将真实图像转换为具有保存下来的笔触纹理的风格化油画，以构建大规模成对训练数据集。最后，在推理阶段，我们使用AdaIN操作符整合特征，确保风格一致性。 广泛的实验表明，我们的互动系统能够在保留油画艺术品质的同时进行精细编辑，实现了在风格化油画生成和编辑方面前所未有的想象力实现水平。
URL
https://arxiv.org/abs/2512.08534
PDF
-
OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation
2025-12-09 06:49:33Yexin Liu, Manyuan Zhang, Yueze Wang, Hongyu Li, Dian Zheng, Weiming Zhang, Changsheng Lu, Xunliang Cai, Yan Feng, Peng Pei, Harry YangAbstract
Despite the promising progress in subject-driven image generation, current models often deviate from the reference identities and struggle in complex scenes with multiple subjects. To address this challenge, we introduce OpenSubject, a video-derived large-scale corpus with 2.5M samples and 4.35M images for subject-driven generation and manipulation. The dataset is built with a four-stage pipeline that exploits cross-frame identity priors. (i) Video Curation. We apply resolution and aesthetic filtering to obtain high-quality clips. (ii) Cross-Frame Subject Mining and Pairing. We utilize vision-language model (VLM)-based category consensus, local grounding, and diversity-aware pairing to select image pairs. (iii) Identity-Preserving Reference Image Synthesis. We introduce segmentation map-guided outpainting to synthesize the input images for subject-driven generation and box-guided inpainting to generate input images for subject-driven manipulation, together with geometry-aware augmentations and irregular boundary erosion. (iv) Verification and Captioning. We utilize a VLM to validate synthesized samples, re-synthesize failed samples based on stage (iii), and then construct short and long captions. In addition, we introduce a benchmark covering subject-driven generation and manipulation, and then evaluate identity fidelity, prompt adherence, manipulation consistency, and background consistency with a VLM judge. Extensive experiments show that training with OpenSubject improves generation and manipulation performance, particularly in complex scenes.
Abstract (translated)
尽管主题驱动的图像生成在取得显著进展的同时，当前模型仍然常常偏离参考身份，并且在处理包含多个主体的复杂场景时存在困难。为了解决这一挑战，我们引入了OpenSubject，这是一个基于视频的大规模语料库，包含250万样本和435万张图片，旨在用于主题驱动生成与操作。该数据集通过一个四阶段管道构建，并利用跨帧身份先验进行优化。 (i) 视频筛选：应用分辨率和美学过滤以获取高质量的片段。 (ii) 跨帧主体挖掘与配对：采用基于视觉-语言模型（VLM）的类别共识、局部定位以及多样性感知匹配来选择图像对。 (iii) 保持身份一致性参考图像合成：引入分割图引导的扩展技术来合成用于主题驱动生成的输入图像，并使用边界框指导的填充技术为操作任务生成输入图片。此外，还结合了几何增强和不规则边界的侵蚀处理。 (iv) 验证与描述：利用VLM验证合成样本，针对阶段(iii)中失败的样本进行重新合成，并构建短长双语描述。 此外，我们还提出了一项涵盖主题驱动生成与操作的新基准测试，并使用基于视觉-语言模型（VLM）的评估标准来评估身份保真度、指令遵循性、操作一致性以及背景一致性。广泛的实验表明，使用OpenSubject进行训练可以提高在复杂场景中的图像生成和编辑性能。
URL
https://arxiv.org/abs/2512.08294
PDF
-
UnCageNet: Tracking and Pose Estimation of Caged Animal
2025-12-08 17:00:06Sayak Dutta, Harish Katti, Shashikant Verma, Shanmuganathan RamanAbstract
Animal tracking and pose estimation systems, such as STEP (Simultaneous Tracking and Pose Estimation) and ViTPose, experience substantial performance drops when processing images and videos with cage structures and systematic occlusions. We present a three-stage preprocessing pipeline that addresses this limitation through: (1) cage segmentation using a Gabor-enhanced ResNet-UNet architecture with tunable orientation filters, (2) cage inpainting using CRFill for content-aware reconstruction of occluded regions, and (3) evaluation of pose estimation and tracking on the uncaged frames. Our Gabor-enhanced segmentation model leverages orientation-aware features with 72 directional kernels to accurately identify and segment cage structures that severely impair the performance of existing methods. Experimental validation demonstrates that removing cage occlusions through our pipeline enables pose estimation and tracking performance comparable to that in environments without occlusions. We also observe significant improvements in keypoint detection accuracy and trajectory consistency.
Abstract (translated)
动物跟踪和姿态估计系统（如STEP（同时跟踪与姿态估计）和ViTPose）在处理含有笼子结构和系统性遮挡的图像和视频时，性能会显著下降。我们提出了一种三阶段预处理流水线来解决这一问题： 1. 使用改进了Gabor滤波器的ResNet-UNet架构进行笼子分割，该模型具有可调方向滤波器； 2. 采用CRFill技术对遮挡区域进行内容感知重建（即“填充”）以恢复被笼子遮挡的部分； 3. 在去除了笼子后的图像帧上评估姿态估计和跟踪性能。 我们的Gabor增强型分割模型利用了具有72个方向内核的方向感知特征，能够准确地识别并分割严重妨碍现有方法性能的笼子结构。实验验证表明，通过我们的流水线去除遮挡后，姿态估计和跟踪性能可与无遮挡环境中的表现相媲美，并且在关键点检测精度和轨迹一致性方面取得了显著改进。
URL
https://arxiv.org/abs/2512.07712
PDF
-
Task adaptation of Vision-Language-Action model: 1st Place Solution for the 2025 BEHAVIOR Challenge
2025-12-07 18:08:45Ilia Larchenko, Gleb Zarin, Akash KarnatakAbstract
We present a vision-action policy that won 1st place in the 2025 BEHAVIOR Challenge - a large-scale benchmark featuring 50 diverse long-horizon household tasks in photo-realistic simulation, requiring bimanual manipulation, navigation, and context-aware decision making. Building on the Pi0.5 architecture, we introduce several innovations. Our primary contribution is correlated noise for flow matching, which improves training efficiency and enables correlation-aware inpainting for smooth action sequences. We also apply learnable mixed-layer attention and System 2 stage tracking for ambiguity resolution. Training employs multi-sample flow matching to reduce variance, while inference uses action compression and challenge-specific correction rules. Our approach achieves 26% q-score across all 50 tasks on both public and private leaderboards.
Abstract (translated)
我们提出了一种视觉动作策略，该策略在2025年BEHAVIOR挑战赛中荣获第一名。这项大型基准测试包括了50个多样化的长期家庭任务，在高度逼真的模拟环境中进行，需要双手操作、导航以及情境感知决策能力。基于Pi0.5架构，我们引入了几项创新： 我们的主要贡献是相关噪声的流量匹配技术，这提高了训练效率，并且使情境感知插值成为可能，从而产生了流畅的动作序列。此外，我们还应用了可学习混合层注意力和System 2阶段跟踪来解决模棱两可的情况。 在训练过程中，我们采用多样本流匹配以减少方差，而在推理时则使用动作压缩以及特定挑战的校正规则。我们的方法在公开和私有排行榜的所有50个任务中都获得了26%的q-score分数。
URL
https://arxiv.org/abs/2512.06951
PDF
-
Training-Time Action Conditioning for Efficient Real-Time Chunking
2025-12-05 18:57:28Kevin Black, Allen Z. Ren, Michael Equi, Sergey LevineAbstract
Real-time chunking (RTC) enables vision-language-action models (VLAs) to generate smooth, reactive robot trajectories by asynchronously predicting action chunks and conditioning on previously committed actions via inference-time inpainting. However, this inpainting method introduces computational overhead that increases inference latency. In this work, we propose a simple alternative: simulating inference delay at training time and conditioning on action prefixes directly, eliminating any inference-time overhead. Our method requires no modifications to the model architecture or robot runtime, and can be implemented with only a few additional lines of code. In simulated experiments, we find that training-time RTC outperforms inference-time RTC at higher inference delays. In real-world experiments on box building and espresso making tasks with the $\pi_{0.6}$ VLA, we demonstrate that training-time RTC maintains both task performance and speed parity with inference-time RTC while being computationally cheaper. Our results suggest that training-time action conditioning is a practical drop-in replacement for inference-time inpainting in real-time robot control.
Abstract (translated)
实时分块（RTC）技术使视觉-语言-行动模型（VLAs）能够通过异步预测动作片段并利用推理时的补帧方法参考之前已确定的动作，生成流畅且响应迅速的机器人轨迹。然而，这种补帧方式引入了计算开销，增加了推断延迟。在这项工作中，我们提出了一种简单的替代方案：在训练阶段模拟推理延迟，并直接基于行动前缀进行条件设置，从而消除了所有推断时的额外开销。我们的方法不需要对模型架构或机器人运行时间进行任何修改，仅需添加少量代码即可实现。在仿真实验中，我们发现训练时的RTC优于推理时的RTC，在更高的推断延迟下表现更佳。在现实世界中的盒子构建和意式咖啡制作任务上使用$\pi_{0.6}$ VLA模型的真实实验表明，训练时的RTC不仅保持了与推断时RTC相同的任务性能和速度，并且计算成本更低。我们的结果表明，基于训练阶段的动作条件设置可以作为实时机器人控制中推理时补帧的有效替代方案。
URL
https://arxiv.org/abs/2512.05964
PDF
-
InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem
2025-12-05 12:31:09Yeobin Hong, Suhyeon Lee, Hyungjin Chung, Jong Chul YeAbstract
Recent approaches to controllable 4D video generation often rely on fine-tuning pre-trained Video Diffusion Models (VDMs). This dominant paradigm is computationally expensive, requiring large-scale datasets and architectural modifications, and frequently suffers from catastrophic forgetting of the model's original generative priors. Here, we propose InverseCrafter, an efficient inpainting inverse solver that reformulates the 4D generation task as an inpainting problem solved in the latent space. The core of our method is a principled mechanism to encode the pixel space degradation operator into a continuous, multi-channel latent mask, thereby bypassing the costly bottleneck of repeated VAE operations and backpropagation. InverseCrafter not only achieves comparable novel view generation and superior measurement consistency in camera control tasks with near-zero computational overhead, but also excels at general-purpose video inpainting with editing. Code is available at this https URL.
Abstract (translated)
最近的可控4D视频生成方法通常依赖于微调预训练的视频扩散模型（VDMs）。这一主流范式在计算上成本高昂，需要大规模的数据集和架构修改，并且常常会导致模型原始生成先验知识的灾难性遗忘。为此，我们提出了InverseCrafter，这是一种高效的修复逆向求解器，它将4D生成任务重新表述为一种在潜在空间中解决的修补问题。我们的方法核心是一个有原则的机制，用于将像素空间退化算子编码成一个连续的、多通道的潜在掩模，从而绕过了重复VAE操作和反向传播的成本高昂瓶颈。 InverseCrafter不仅实现了与现有方法相当的新视角生成能力，并且在相机控制任务中表现出更优的测量一致性，同时几乎不增加计算开销。此外，在通用视频修补和编辑方面也表现优异。代码可在提供的链接地址获取。
URL
https://arxiv.org/abs/2512.05672
PDF
-
Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models
2025-12-04 19:07:42Rowan Bradbury, Dazhi ZhongAbstract
Latent inpainting in diffusion models still relies almost universally on linearly interpolating VAE latents under a downsampled mask. We propose a key principle for compositing image latents: Pixel-Equivalent Latent Compositing (PELC). An equivalent latent compositor should be the same as compositing in pixel space. This principle enables full-resolution mask control and true soft-edge alpha compositing, even though VAEs compress images 8x spatially. Modern VAEs capture global context beyond patch-aligned local structure, so linear latent blending cannot be pixel-equivalent: it produces large artifacts at mask seams and global degradation and color shifts. We introduce DecFormer, a 7.7M-parameter transformer that predicts per-channel blend weights and an off-manifold residual correction to realize mask-consistent latent fusion. DecFormer is trained so that decoding after fusion matches pixel-space alpha compositing, is plug-compatible with existing diffusion pipelines, requires no backbone finetuning and adds only 0.07% of FLUX.1-Dev's parameters and 3.5% FLOP overhead. On the FLUX.1 family, DecFormer restores global color consistency, soft-mask support, sharp boundaries, and high-fidelity masking, reducing error metrics around edges by up to 53% over standard mask interpolation. Used as an inpainting prior, a lightweight LoRA on FLUX.1-Dev with DecFormer achieves fidelity comparable to FLUX.1-Fill, a fully finetuned inpainting model. While we focus on inpainting, PELC is a general recipe for pixel-equivalent latent editing, as we demonstrate on a complex color-correction task.
Abstract (translated)
扩散模型中的潜在域图像修补技术几乎普遍依赖于在下采样掩码下的VAE潜在变量的线性插值。我们提出了一项用于合成图像潜在量的关键原则：像素等效潜在合成（Pixel-Equivalent Latent Compositing，PELC）。一个等效的潜在合成器应当与像素空间中的合成过程一致。这一原则使得全分辨率掩码控制和真正的软边缘alpha合成成为可能，即使VAE将图像的空间压缩了8倍。现代VAE捕捉到了超越局部结构对齐的全局上下文信息，因此线性潜在融合不能达到像素等效：在掩码接缝处会产生显著瑕疵，并且全局退化和颜色偏移现象严重。我们引入了一个名为DecFormer的7.7M参数Transformer模型，它能够预测每通道的混合权重以及离流形残差校正来实现一致性的潜在融合。DecFormer经过训练以确保在合并后解码的结果与像素空间中的alpha合成相匹配，并且可以无缝兼容现有的扩散流程，无需对骨干网络进行微调，仅增加了FLUX.1-Dev参数的0.07%和3.5%的FLOP开销。在FLUX.1系列中，DecFormer能够恢复全局颜色一致性、支持软掩码、保持清晰边界以及高保真度的遮罩处理，在边缘处降低了误差指标高达53%，相比标准掩码插值有显著改进。当作为修补前缀使用时，结合轻量级LoRA和FLUX.1-Dev的DecFormer模型可以实现与完全微调后的FLUX.1-Fill修补模型相当的保真度。尽管我们专注于图像修补，PELC作为一个通用的方法用于像素等效潜在编辑，我们在一个复杂的颜色校正任务中也展示了其应用效果。
URL
https://arxiv.org/abs/2512.05198
PDF
-
Semantic-Guided Two-Stage GAN for Face Inpainting with Hybrid Perceptual Encoding
2025-12-04 17:56:08Abhigyan Bhattacharya, Hiranmoy RoyAbstract
Facial Image inpainting aim is to restore the missing or corrupted regions in face images while preserving identity, structural consistency and photorealistic image quality, a task specifically created for photo restoration. Though there are recent lot of advances in deep generative models, existing methods face problems with large irregular masks, often producing blurry textures on the edges of the masked region, semantic inconsistencies, or unconvincing facial structures due to direct pixel level synthesis approach and limited exploitation of facial priors. In this paper we propose a novel architecture, which address these above challenges through semantic-guided hierarchical synthesis. Our approach starts with a method that organizes and synthesizes information based on meaning, followed by refining the texture. This process gives clear insights into the facial structure before we move on to creating detailed images. In the first stage, we blend two techniques: one that focuses on local features with CNNs and global features with Vision Transformers. This helped us create clear and detailed semantic layouts. In the second stage, we use a Multi-Modal Texture Generator to refine these layouts by pulling in information from different scales, ensuring everything looks cohesive and consistent. The architecture naturally handles arbitrary mask configurations through dynamic attention without maskspecific training. Experiment on two datasets CelebA-HQ and FFHQ shows that our model outperforms other state-of-the-art methods, showing improvements in metrics like LPIPS, PSNR, and SSIM. It produces visually striking results with better semantic preservation, in challenging large-area inpainting situations.
Abstract (translated)
面部图像修复（Facial Image Inpainting）的目标是恢复脸部图像中丢失或损坏的区域，同时保持身份识别、结构一致性和逼真的照片质量。这一任务专门用于图片修复领域。尽管近年来深度生成模型取得了显著进展，现有方法在处理大面积不规则遮罩时仍面临挑战，通常会导致边缘模糊、语义不一致性或面部结构不合理等问题，这是因为直接采用像素级合成的方法和有限地利用面部先验知识所致。 本文提出了一种新架构，通过语义引导的层次化综合解决了上述问题。我们的方法首先基于意义组织并合成了信息，然后细化纹理，从而在创造详细图像之前提供了对面部结构清晰的见解。第一阶段中，我们融合了两种技术：一种专注于使用CNN进行局部特征分析，另一种则利用Vision Transformers对全局特征进行处理。这有助于创建清晰且详细的语义布局。 第二阶段中，我们采用多模态纹理生成器来细化这些布局，并从不同尺度获取信息，确保一切看起来连贯一致。该架构通过动态注意力机制自然地处理了任意遮罩配置，无需针对特定遮罩的训练。在CelebA-HQ和FFHQ两个数据集上的实验表明，我们的模型优于其他最先进的方法，在LPIPS、PSNR和SSIM等指标上显示出改进，并在具有挑战性的大面积修复情况下产生了视觉效果更佳且语义保存更好的结果。 简而言之，所提出的新型架构通过引入层次化综合和多模态纹理生成器，显著改善了面部图像的修复质量。这种方法不仅提高了视觉质量和结构一致性，而且对于解决复杂的大面积损坏问题也展现了强大的潜力。
URL
https://arxiv.org/abs/2512.05039
PDF
-
Plug-and-Play Image Restoration with Flow Matching: A Continuous Viewpoint
2025-12-03 21:50:36Fan Jia, Yuhao Huang, Shih-Hsin Wang, Cristina Garcia-Cardona, Andrea L. Bertozzi, Bao WangAbstract
Flow matching-based generative models have been integrated into the plug-and-play image restoration framework, and the resulting plug-and-play flow matching (PnP-Flow) model has achieved some remarkable empirical success for image restoration. However, the theoretical understanding of PnP-Flow lags its empirical success. In this paper, we derive a continuous limit for PnP-Flow, resulting in a stochastic differential equation (SDE) surrogate model of PnP-Flow. The SDE model provides two particular insights to improve PnP-Flow for image restoration: (1) It enables us to quantify the error for image restoration, informing us to improve step scheduling and regularize the Lipschitz constant of the neural network-parameterized vector field for error reduction. (2) It informs us to accelerate off-the-shelf PnP-Flow models via extrapolation, resulting in a rescaled version of the proposed SDE model. We validate the efficacy of the SDE-informed improved PnP-Flow using several benchmark tasks, including image denoising, deblurring, super-resolution, and inpainting. Numerical results show that our method significantly outperforms the baseline PnP-Flow and other state-of-the-art approaches, achieving superior performance across evaluation metrics.
Abstract (translated)
基于流匹配的生成模型已被集成到“即插即用”的图像恢复框架中，由此产生的即插即用流匹配（PnP-Flow）模型在图像恢复方面取得了显著的实际成功。然而，关于PnP-Flow的理论理解落后于其实际表现。在这篇论文中，我们为PnP-Flow推导出了一个连续极限，并得出了一种随机微分方程（SDE）替代模型来表示PnP-Flow。该SDE模型提供了两个特别的见解，以改进用于图像恢复的PnP-Flow：(1) 它使我们可以量化图像恢复中的误差，告知我们如何优化步长调度并调节神经网络参数化向量场的Lipschitz常数，从而减少误差。(2) 它还告诫我们可以通过外推加速现成的PnP-Flow模型，这导致了提议SDE模型的一个重缩放版本。我们使用包括图像去噪、去模糊、超分辨率和插值在内的多个基准任务来验证增强版PnP-Flow（基于SDE）的有效性。数值结果显示，我们的方法在评估指标上显著优于基线的PnP-Flow和其他最先进方法，实现了卓越的整体性能。
URL
https://arxiv.org/abs/2512.04283
PDF
-
Fast & Efficient Normalizing Flows and Applications of Image Generative Models
2025-12-03 18:29:03Sandeep NagarAbstract
This thesis presents novel contributions in two primary areas: advancing the efficiency of generative models, particularly normalizing flows, and applying generative models to solve real-world computer vision challenges. The first part introduce significant improvements to normalizing flow architectures through six key innovations: 1) Development of invertible 3x3 Convolution layers with mathematically proven necessary and sufficient conditions for invertibility, (2) introduction of a more efficient Quad-coupling layer, 3) Design of a fast and efficient parallel inversion algorithm for kxk convolutional layers, 4) Fast & efficient backpropagation algorithm for inverse of convolution, 5) Using inverse of convolution, in Inverse-Flow, for the forward pass and training it using proposed backpropagation algorithm, and 6) Affine-StableSR, a compact and efficient super-resolution model that leverages pre-trained weights and Normalizing Flow layers to reduce parameter count while maintaining performance. The second part: 1) An automated quality assessment system for agricultural produce using Conditional GANs to address class imbalance, data scarcity and annotation challenges, achieving good accuracy in seed purity testing; 2) An unsupervised geological mapping framework utilizing stacked autoencoders for dimensionality reduction, showing improved feature extraction compared to conventional methods; 3) We proposed a privacy preserving method for autonomous driving datasets using on face detection and image inpainting; 4) Utilizing Stable Diffusion based image inpainting for replacing the detected face and license plate to advancing privacy-preserving techniques and ethical considerations in the field.; and 5) An adapted diffusion model for art restoration that effectively handles multiple types of degradation through unified fine-tuning.
Abstract (translated)
这篇论文在两个主要领域提出了创新贡献：一是提升生成模型（尤其是归一化流）的效率，二是应用生成模型解决实际计算机视觉挑战。第一部分通过六项关键创新对归一化流动架构进行了重大改进： 1) 开发了具有数学证明的必要和充分条件以确保可逆性的可逆3x3卷积层。 2) 引入了一种更高效的四耦合层（Quad-coupling layer）。 3) 设计了一个快速且有效的并行反演算法，用于kxk卷积层。 4) 为卷积的逆向传播开发了高效算法。 5) 在前向传递中使用卷积的逆进行Invert-Flow，并通过提出的反向传播算法进行训练。 6) Affine-StableSR，一个紧凑高效的超分辨率模型，利用预训练权重和归一化流动层来减少参数数量同时保持性能。 第二部分包括： 1) 一种使用条件生成对抗网络（Conditional GANs）的自动质量评估系统，用于解决农业产品中的类别不平衡、数据稀缺和标注挑战，并在种子纯度测试中实现了良好的准确性。 2) 一个利用堆叠自编码器进行降维的无监督地质制图框架，在特征提取方面优于传统方法。 3) 提出了一种使用面部检测和图像修复来保护自动驾驶数据集隐私的方法。 4) 利用基于Stable Diffusion的图像修复技术，将检测到的脸部和车牌替换为提高隐私保护技术和伦理考虑的方法。 5) 一种经过改进的扩散模型用于艺术作品恢复，能够通过统一微调处理多种类型的退化。
URL
https://arxiv.org/abs/2512.04039
PDF
-
PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement
2025-12-02 21:35:57Haitian Zheng, Yuan Yao, Yongsheng Yu, Yuqian Zhou, Jiebo Luo, Zhe LinAbstract
Latent Diffusion Models (LDMs) have markedly advanced the quality of image inpainting and local editing. However, the inherent latent compression often introduces pixel-level inconsistencies, such as chromatic shifts, texture mismatches, and visible seams along editing boundaries. Existing remedies, including background-conditioned latent decoding and pixel-space harmonization, usually fail to fully eliminate these artifacts in practice and do not generalize well across different latent representations or tasks. We introduce PixPerfect, a pixel-level refinement framework that delivers seamless, high-fidelity local edits across diverse LDM architectures and tasks. PixPerfect leverages (i) a differentiable discriminative pixel space that amplifies and suppresses subtle color and texture discrepancies, (ii) a comprehensive artifact simulation pipeline that exposes the refiner to realistic local editing artifacts during training, and (iii) a direct pixel-space refinement scheme that ensures broad applicability across diverse latent representations and tasks. Extensive experiments on inpainting, object removal, and insertion benchmarks demonstrate that PixPerfect substantially enhances perceptual fidelity and downstream editing performance, establishing a new standard for robust and high-fidelity localized image editing.
Abstract (translated)
潜扩散模型（LDMs）显著提高了图像修复和局部编辑的质量。然而，其固有的潜在压缩常常引入像素级别的不一致性，例如色彩偏移、纹理不匹配以及编辑边界处的可见接缝等问题。现有的补救措施，包括背景条件下的潜在解码和像素空间调和，在实际应用中往往无法完全消除这些伪影，并且在不同的潜表示或任务之间泛化效果不佳。我们引入了PixPerfect，这是一种像素级别的细化框架，它能够提供无缝、高质量的局部编辑效果，适用于各种LDM架构和任务。 PixPerfect利用了以下三个关键要素： 1. **可微分判别像素空间**：放大并抑制细微的颜色和纹理差异。 2. **全面的伪影仿真管道**：在训练过程中让细化器接触现实中的本地编辑伪影。 3. **直接的像素空间细化方案**：确保广泛的适用性，适用于各种潜在表示和任务。 在图像修复、对象移除和插入基准上的大量实验表明，PixPerfect显著提升了感知保真度和下游编辑性能，为鲁棒且高质量的局部图像编辑树立了新标准。
URL
https://arxiv.org/abs/2512.03247
PDF
-
LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization
2025-12-02 17:01:07Zhihan Xiao, Lin Liu, Yixin Gao, Xiaopeng Zhang, Haoxuan Che, Songping Mai, Qi TianAbstract
Text-guided video editing, particularly for object removal and addition, remains a challenging task due to the need for precise spatial and temporal consistency. Existing methods often rely on auxiliary masks or reference images for editing guidance, which limits their scalability and generalization. To address these issues, we propose LoVoRA, a novel framework for mask-free video object removal and addition using object-aware localization mechanism. Our approach utilizes a unique dataset construction pipeline that integrates image-to-video translation, optical flow-based mask propagation, and video inpainting, enabling temporally consistent edits. The core innovation of LoVoRA is its learnable object-aware localization mechanism, which provides dense spatio-temporal supervision for both object insertion and removal tasks. By leveraging a Diffusion Mask Predictor, LoVoRA achieves end-to-end video editing without requiring external control signals during inference. Extensive experiments and human evaluation demonstrate the effectiveness and high-quality performance of LoVoRA.
Abstract (translated)
文本指导的视频编辑，特别是在对象移除和添加方面，由于需要精确的空间和时间一致性而仍然是一项具有挑战性的任务。现有方法通常依赖于辅助掩码或参考图像来进行编辑引导，这限制了它们的可扩展性和泛化能力。为了解决这些问题，我们提出了LoVoRA（Learnable Object-aware Localization for Video Remasking and Addition），这是一种新的无掩码视频对象移除和添加框架，采用感知物体定位机制。我们的方法利用了一个独特的数据集构建流水线，该流水线结合了图像到视频的转换、基于光流的掩码传播以及视频修复技术，从而实现时间一致性编辑。 LoVoRA的核心创新在于其可学习的对象感知定位机制，这种机制为对象插入和移除任务提供了密集的空间-时间监督。通过利用扩散掩码预测器（Diffusion Mask Predictor），LoVoRA能够实现端到端的视频编辑，并且在推理过程中不需要外部控制信号。大量的实验以及人类评估证明了LoVoRA的有效性和高质量性能。 该技术不仅解决了现有方法中依赖辅助信息的问题，还提高了视频编辑任务中的空间和时间一致性的准确性，为更广泛的视频编辑应用场景提供了可能。
URL
https://arxiv.org/abs/2512.02933
PDF
-
ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data
2025-12-02 12:14:19Yuxing Liu, Yong LiuAbstract
Anomaly segmentation seeks to detect and localize unknown or out-of-distribution (OoD) objects that fall outside predefined semantic classes a capability essential for safe autonomous driving. However, the scarcity and limited diversity of anomaly data severely constrain model generalization in open-world environments. Existing approaches mitigate this issue through synthetic data generation, either by copy-pasting external objects into driving scenes or by leveraging text-to-image diffusion models to inpaint anomalous regions. While these methods improve anomaly diversity, they often lack contextual coherence and physical realism, resulting in domain gaps between synthetic and real data. In this paper, we present ClimaDrive, a semantics-guided image-to-image framework for synthesizing semantically coherent, weather-diverse, and physically plausible OoD driving data. ClimaDrive unifies structure-guided multi-weather generation with prompt-driven anomaly inpainting, enabling the creation of visually realistic training data. Based on this framework, we construct ClimaOoD, a large-scale benchmark spanning six representative driving scenarios under both clear and adverse weather conditions. Extensive experiments on four state-of-the-art methods show that training with ClimaOoD leads to robust improvements in anomaly segmentation. Across all methods, AUROC, AP, and FPR95 show notable gains, with FPR95 dropping from 3.97 to 3.52 for RbA on Fishyscapes LAF. These results demonstrate that ClimaOoD enhances model robustness, offering valuable training data for better generalization in open-world anomaly detection.
Abstract (translated)
异常分割旨在检测和定位未知或分布外（OoD）的物体，这些物体超出了预定义的语义类别范畴，这一能力对于安全自主驾驶至关重要。然而，缺乏多样性以及稀少性的异常数据严重制约了模型在开放环境中的泛化能力。现有方法通过生成合成数据来缓解此问题，例如将外部对象复制粘贴到驾驶场景中或将文本转换为图像扩散模型用于修补异常区域。尽管这些方法提高了异常多样性和训练的便利性，但它们通常缺乏上下文连贯性和物理现实感，导致合成数据与真实数据之间的领域差距。 在本文中，我们介绍了ClimaDrive，这是一种基于语义指导的图像到图像框架，专门用于生成语义上一致、天气多样化且物理上合理的OoD驾驶数据。ClimaDrive结合了结构引导下的多天气生成和提示驱动的异常修补，从而能够创建视觉上真实的训练数据。在此基础上，我们构建了ClimaOoD，这是一个大型基准测试库，在六种具有代表性的驾驶场景下进行测试，涵盖了晴朗及恶劣天气条件。 对四种最先进的方法进行了广泛的实验后显示，使用ClimaOoD进行训练可以显著提高异常分割的鲁棒性。所有方法在AUROC、AP和FPR95指标上都取得了显著的进步，在Fishyscapes LAF数据集上RbA模型的FPR95从3.97降低到3.52。 这些结果表明，ClimaOoD可以增强模型的鲁棒性，并提供有价值的训练数据以提高开放世界异常检测中的泛化能力。
URL
https://arxiv.org/abs/2512.02686
PDF
-
Evaluating the Clinical Impact of Generative Inpainting on Bone Age Estimation
2025-11-28 10:48:50Felipe Akio Matsuoka, Eduardo Moreno J. M. Farina, Augusto Sarquis Serpa, Soraya Monteiro, Rodrigo Ragazzini, Nitamar Abdala, Marcelo Straus Takahashi, Felipe Campos KitamuraAbstract
Generative foundation models can remove visual artifacts through realistic image inpainting, but their impact on medical AI performance remains uncertain. Pediatric hand radiographs often contain non-anatomical markers, and it is unclear whether inpainting these regions preserves features needed for bone age and gender prediction. To evaluate the clinical reliability of generative model-based inpainting for artifact removal, we used the RSNA Bone Age Challenge dataset, selecting 200 original radiographs and generating 600 inpainted versions with gpt-image-1 using natural language prompts to target non-anatomical artifacts. Downstream performance was assessed with deep learning ensembles for bone age estimation and gender classification, using mean absolute error (MAE) and area under the ROC curve (AUC) as metrics, and pixel intensity distributions to detect structural alterations. Inpainting markedly degraded model performance: bone age MAE increased from 6.26 to 30.11 months, and gender classification AUC decreased from 0.955 to 0.704. Inpainted images displayed pixel-intensity shifts and inconsistencies, indicating structural modifications not corrected by simple calibration. These findings show that, although visually realistic, foundation model-based inpainting can obscure subtle but clinically relevant features and introduce latent bias even when edits are confined to non-diagnostic regions, underscoring the need for rigorous, task-specific validation before integrating such generative tools into clinical AI workflows.
Abstract (translated)
URL
https://arxiv.org/abs/2511.23066
PDF
-
Improving Robotic Manipulation Robustness via NICE Scene Surgery
2025-11-27 22:02:02Sajjad Pakdamansavoji, Mozhgan Pourkeshavarz, Adam Sigal, Zhiyuan Li, Rui Heng Yang, Amir RasouliAbstract
Learning robust visuomotor policies for robotic manipulation remains a challenge in real-world settings, where visual distractors can significantly degrade performance and safety. In this work, we propose an effective and scalable framework, Naturalistic Inpainting for Context Enhancement (NICE). Our method minimizes out-of-distribution (OOD) gap in imitation learning by increasing visual diversity through construction of new experiences using existing demonstrations. By utilizing image generative frameworks and large language models, NICE performs three editing operations, object replacement, restyling, and removal of distracting (non-target) objects. These changes preserve spatial relationships without obstructing target objects and maintain action-label consistency. Unlike previous approaches, NICE requires no additional robot data collection, simulator access, or custom model training, making it readily applicable to existing robotic datasets. Using real-world scenes, we showcase the capability of our framework in producing photo-realistic scene enhancement. For downstream tasks, we use NICE data to finetune a vision-language model (VLM) for spatial affordance prediction and a vision-language-action (VLA) policy for object manipulation. Our evaluations show that NICE successfully minimizes OOD gaps, resulting in over 20% improvement in accuracy for affordance prediction in highly cluttered scenes. For manipulation tasks, success rate increases on average by 11% when testing in environments populated with distractors in different quantities. Furthermore, we show that our method improves visual robustness, lowering target confusion by 6%, and enhances safety by reducing collision rate by 7%.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22777
PDF
-
What Shape Is Optimal for Masks in Text Removal?
2025-11-27 14:34:35Hyakka Nakada, Marika KubotaAbstract
The advent of generative models has dramatically improved the accuracy of image inpainting. In particular, by removing specific text from document images, reconstructing original images is extremely important for industrial applications. However, most existing methods of text removal focus on deleting simple scene text which appears in images captured by a camera in an outdoor environment. There is little research dedicated to complex and practical images with dense text. Therefore, we created benchmark data for text removal from images including a large amount of text. From the data, we found that text-removal performance becomes vulnerable against mask profile perturbation. Thus, for practical text-removal tasks, precise tuning of the mask shape is essential. This study developed a method to model highly flexible mask profiles and learn their parameters using Bayesian optimization. The resulting profiles were found to be character-wise masks. It was also found that the minimum cover of a text region is not optimal. Our research is expected to pave the way for a user-friendly guideline for manual masking.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22499
PDF
-
AmodalGen3D: Generative Amodal 3D Object Reconstruction from Sparse Unposed Views
2025-11-26 22:11:56Junwei Zhou, Yu-Wing TaiAbstract
Reconstructing 3D objects from a few unposed and partially occluded views is a common yet challenging problem in real-world scenarios, where many object surfaces are never directly observed. Traditional multi-view or inpainting-based approaches struggle under such conditions, often yielding incomplete or geometrically inconsistent reconstructions. We introduce AmodalGen3D, a generative framework for amodal 3D object reconstruction that infers complete, occlusion-free geometry and appearance from arbitrary sparse inputs. The model integrates 2D amodal completion priors with multi-view stereo geometry conditioning, supported by a View-Wise Cross Attention mechanism for sparse-view feature fusion and a Stereo-Conditioned Cross Attention module for unobserved structure inference. By jointly modeling visible and hidden regions, AmodalGen3D faithfully reconstructs 3D objects that are consistent with sparse-view constraints while plausibly hallucinating unseen parts. Experiments on both synthetic and real-world datasets demonstrate that AmodalGen3D achieves superior fidelity and completeness under occlusion-heavy sparse-view settings, addressing a pressing need for object-level 3D scene reconstruction in robotics, AR/VR, and embodied AI applications.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21945
PDF
-
From Diffusion to One-Step Generation: A Comparative Study of Flow-Based Models with Application to Image Inpainting
2025-11-26 09:44:51Umang Agarwal, Rudraksh Sangore, Sumit LaddhaAbstract
We present a comprehensive comparative study of three generative modeling paradigms: Denoising Diffusion Probabilistic Models (DDPM), Conditional Flow Matching (CFM), and MeanFlow. While DDPM and CFM require iterative sampling, MeanFlow enables direct one-step generation by modeling the average velocity over time intervals. We implement all three methods using a unified TinyUNet architecture (<1.5M parameters) on CIFAR-10, demonstrating that CFM achieves an FID of 24.15 with 50 steps, significantly outperforming DDPM (FID 402.98). MeanFlow achieves FID 29.15 with single-step sampling -- a 50X reduction in inference time. We further extend CFM to image inpainting, implementing mask-guided sampling with four mask types (center, random bbox, irregular, half). Our fine-tuned inpainting model achieves substantial improvements: PSNR increases from 4.95 to 8.57 dB on center masks (+73%), and SSIM improves from 0.289 to 0.418 (+45%), demonstrating the effectiveness of inpainting-aware training.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21215
PDF
