Inpainting
-
Self-Supervised Masked Autoencoders with Dense-Unet for Coronary Calcium Removal in limited CT Data
2025-12-25 12:36:37Mo ChenAbstract
Coronary calcification creates blooming artifacts in Computed Tomography Angiography (CTA), severely hampering the diagnosis of lumen stenosis. While Deep Convolutional Neural Networks (DCNNs) like Dense-Unet have shown promise in removing these artifacts via inpainting, they often require large labeled datasets which are scarce in the medical domain. Inspired by recent advancements in Masked Autoencoders (MAE) for 3D point clouds, we propose \textbf{Dense-MAE}, a novel self-supervised learning framework for volumetric medical data. We introduce a pre-training strategy that randomly masks 3D patches of the vessel lumen and trains the Dense-Unet to reconstruct the missing geometry. This forces the encoder to learn high-level latent features of arterial topology without human annotation. Experimental results on clinical CTA datasets demonstrate that initializing the Calcium Removal network with our MAE-based weights significantly improves inpainting accuracy and stenosis estimation compared to training from scratch, specifically in few-shot scenarios.
Abstract (translated)
冠状钙化在计算机断层血管成像（CTA）中会产生“开花”伪影，严重阻碍了管腔狭窄的诊断。尽管深度卷积神经网络（DCNN），如Dense-Unet，在通过图像修复去除这些伪影方面显示出了一定潜力，但它们通常需要大量标注的数据集，而这类数据在医学领域往往较为稀缺。受最近在3D点云中应用遮罩自编码器（MAE）进展的启发，我们提出了一种新的针对体积型医疗数据的自监督学习框架——**Dense-MAE**。我们的方法引入了预训练策略：随机屏蔽血管管腔中的3D补丁，并通过训练Dense-Unet来重构缺失的几何结构。这迫使编码器在没有人工标注的情况下，学习动脉拓扑的高级潜在特征。 实验结果表明，在临床CTA数据集上使用基于我们MAE权重初始化钙去除网络的方法，在几例样本（few-shot）场景下，显著提高了修复精度和狭窄程度估计的准确性，相比从头开始训练有明显改进。
URL
https://arxiv.org/abs/2601.02392
PDF
-
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
2025-12-25 10:28:43Zhiyuan Liu, Daocheng Fu, Pinlong Cai, Lening Wang, Ying Liu, Yilong Ren, Botian Shi, Jianqiang WangAbstract
High-fidelity and controllable 3D simulation is essential for addressing the long-tail data scarcity in Autonomous Driving (AD), yet existing methods struggle to simultaneously achieve photorealistic rendering and interactive traffic editing. Current approaches often falter in large-angle novel view synthesis and suffer from geometric or lighting artifacts during asset manipulation. To address these challenges, we propose SymDrive, a unified diffusion-based framework capable of joint high-quality rendering and scene editing. We introduce a Symmetric Auto-regressive Online Restoration paradigm, which constructs paired symmetric views to recover fine-grained details via a ground-truth-guided dual-view formulation and utilizes an auto-regressive strategy for consistent lateral view generation. Furthermore, we leverage this restoration capability to enable a training-free harmonization mechanism, treating vehicle insertion as context-aware inpainting to ensure seamless lighting and shadow consistency. Extensive experiments demonstrate that SymDrive achieves state-of-the-art performance in both novel-view enhancement and realistic 3D vehicle insertion.
Abstract (translated)
高保真且可控的三维仿真对于解决自主驾驶（AD）中的长尾数据稀缺问题至关重要，但现有方法难以同时实现逼真的渲染和互动式交通编辑。目前的方法在处理大角度新颖视角合成时常常表现出色不足，并在资产操作过程中出现几何或光照瑕疵。为了解决这些挑战，我们提出了SymDrive，这是一种统一的基于扩散框架的方法，能够联合高质量的渲染与场景编辑。我们引入了对称自回归在线恢复范式，该范式通过构造配对的对称视图并采用一个以真实标签引导的双视角方案来恢复细微细节，并利用一种自回归策略生成一致侧向视图。此外，我们借助这种恢复能力实现了一种无训练调和机制，在处理车辆插入时将其视为上下文感知的修复过程，确保光照和阴影的一致性。广泛的实验表明，SymDrive在新颖视角增强及逼真的三维车辆插入方面均达到了当前的最佳性能水平。
URL
https://arxiv.org/abs/2512.21618
PDF
-
EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal
2025-12-25 07:34:38Sanghyun Jo, Donghwan Lee, Eunji Jung, Seong Je Oh, Kyungsu KimAbstract
Object removal differs from common inpainting, since it must prevent the masked target from reappearing and reconstruct the occluded background with structural and contextual fidelity, rather than merely filling a hole plausibly. Recent dataset-free approaches that redirect self-attention inside the mask fail in two ways: non-target foregrounds are often misinterpreted as background, which regenerates unwanted objects, and direct attention manipulation disrupts fine details and hinders coherent integration of background cues. We propose EraseLoRA, a novel dataset-free framework that replaces attention surgery with background-aware reasoning and test-time adaptation. First, Background-aware Foreground Exclusion (BFE), uses a multimodal large-language models to separate target foreground, non-target foregrounds, and clean background from a single image-mask pair without paired supervision, producing reliable background cues while excluding distractors. Second, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as complementary pieces and enforces their consistent integration through reconstruction and alignment objectives, preserving local detail and global structure without explicit attention intervention. We validate EraseLoRA as a plug-in to pretrained diffusion models and across benchmarks for object removal, demonstrating consistent improvements over dataset-free baselines and competitive results against dataset-driven methods. The code will be made available upon publication.
Abstract (translated)
对象移除与常见的图像修复（inpainting）有所不同，因为它必须防止被遮罩的目标重新出现，并且需要以结构和上下文的一致性来重建被遮挡的背景，而不仅仅是合理地填充一个空洞。最近一些不依赖数据集的方法通过重定向掩码内的自注意力机制来处理对象移除问题，在两个方面存在不足：非目标前景经常被误认为是背景，从而生成不需要的对象；直接操作注意机制会干扰细节，并阻碍背景线索的连贯整合。 我们提出了一种新的无数据集框架——EraseLoRA，它用基于背景感知推理和测试时适应的方法替代了注意力手术。首先，通过“基于背景感知的前景排除”（BFE），利用多模态大型语言模型从单一图像-掩码对中分离出目标前景、非目标前景以及干净背景，无需配对监督即可产生可靠的背景线索并排除干扰项。其次，“基于背景感知的子类型聚合重建”（BRSA）执行测试时优化，将推断出来的背景子类视为互补部分，并通过重构和对齐目标强制它们的一致整合，在没有显式注意力干预的情况下保持局部细节和全局结构。 我们在预训练扩散模型中验证了EraseLoRA作为插件的有效性，并在对象移除的基准数据集上进行了测试，结果表明相比于无数据集的基础方法它有持续改进的表现，同时也与基于数据驱动的方法具有竞争力。代码将在发表时提供。
URL
https://arxiv.org/abs/2512.21545
PDF
-
FreeInpaint: Tuning-free Prompt Alignment and Visual Rationality Enhancement in Image Inpainting
2025-12-24 11:06:26Chao Gong, Dong Li, Yingwei Pan, Jingjing Chen, Ting Yao, Tao MeiAbstract
Text-guided image inpainting endeavors to generate new content within specified regions of images using textual prompts from users. The primary challenge is to accurately align the inpainted areas with the user-provided prompts while maintaining a high degree of visual fidelity. While existing inpainting methods have produced visually convincing results by leveraging the pre-trained text-to-image diffusion models, they still struggle to uphold both prompt alignment and visual rationality simultaneously. In this work, we introduce FreeInpaint, a plug-and-play tuning-free approach that directly optimizes the diffusion latents on the fly during inference to improve the faithfulness of the generated images. Technically, we introduce a prior-guided noise optimization method that steers model attention towards valid inpainting regions by optimizing the initial noise. Furthermore, we meticulously design a composite guidance objective tailored specifically for the inpainting task. This objective efficiently directs the denoising process, enhancing prompt alignment and visual rationality by optimizing intermediate latents at each step. Through extensive experiments involving various inpainting diffusion models and evaluation metrics, we demonstrate the effectiveness and robustness of our proposed FreeInpaint.
Abstract (translated)
文本引导的图像修复技术旨在通过用户的文本提示，在指定区域内生成新的内容。主要挑战在于确保修复区域与用户提供的提示准确对齐，同时保持高度的视觉真实性。尽管现有的图像修复方法已经利用预训练的文本到图像扩散模型取得了令人信服的视觉效果，但它们在同时维持提示对齐和视觉合理性方面仍面临困难。 在此项工作中，我们引入了FreeInpaint，这是一种即插即用且无需调整的方法，它通过在推理过程中直接优化扩散潜变量来提高生成图像的真实性。技术上，我们提出了一种引导先验的噪声优化方法，该方法通过对初始噪声进行优化，使模型注意力集中在有效的修复区域上。此外，我们精心设计了一个复合引导目标，专门针对图像修复任务。这一目标可以有效指导去噪过程，在每一步通过优化中间潜变量来增强提示对齐和视觉合理性。 通过涉及各种图像修复扩散模型及评估指标的广泛实验，我们展示了所提出的FreeInpaint的有效性和鲁棒性。
URL
https://arxiv.org/abs/2512.21104
PDF
-
Matrix Completion Via Reweighted Logarithmic Norm Minimization
2025-12-24 08:31:44Zhijie Wang, Liangtian He, Qinghua Zhang, Jifei Miao, Liang-Jian Deng, Jun LiuAbstract
Low-rank matrix completion (LRMC) has demonstrated remarkable success in a wide range of applications. To address the NP-hard nature of the rank minimization problem, the nuclear norm is commonly used as a convex and computationally tractable surrogate for the rank function. However, this approach often yields suboptimal solutions due to the excessive shrinkage of singular values. In this letter, we propose a novel reweighted logarithmic norm as a more effective nonconvex surrogate, which provides a closer approximation than many existing alternatives. We efficiently solve the resulting optimization problem by employing the alternating direction method of multipliers (ADMM). Experimental results on image inpainting demonstrate that the proposed method achieves superior performance compared to state-of-the-art LRMC approaches, both in terms of visual quality and quantitative metrics.
Abstract (translated)
低秩矩阵补全（LRMC）在广泛的应用中表现出显著的成功。为了应对秩最小化问题的NP难性质，通常使用核范数作为秩函数的有效且计算可行的凸替代品。然而，这种方法往往由于奇异值过度收缩而产生次优解。在这封信中，我们提出了一种新颖的加权对数范数作为一种更有效的非凸替代方法，它比许多现有方案提供了一个更为接近的近似值。通过采用交替方向乘子法（ADMM），我们高效地解决了由此产生的优化问题。在图像插值实验中，所提方法在视觉质量和定量指标方面均优于最先进的LRMC方法。
URL
https://arxiv.org/abs/2512.21050
PDF
-
X-ray Insights Unleashed: Pioneering the Enhancement of Multi-Label Long-Tail Data
2025-12-24 06:14:55Xinquan Yang, Jinheng Xie, Yawen Huang, Yuexiang Li, Huimin Huang, Hao Zheng, Xian Wu, Yefeng Zheng, Linlin ShenAbstract
Long-tailed pulmonary anomalies in chest radiography present formidable diagnostic challenges. Despite the recent strides in diffusion-based methods for enhancing the representation of tailed lesions, the paucity of rare lesion exemplars curtails the generative capabilities of these approaches, thereby leaving the diagnostic precision less than optimal. In this paper, we propose a novel data synthesis pipeline designed to augment tail lesions utilizing a copious supply of conventional normal X-rays. Specifically, a sufficient quantity of normal samples is amassed to train a diffusion model capable of generating normal X-ray images. This pre-trained diffusion model is subsequently utilized to inpaint the head lesions present in the diseased X-rays, thereby preserving the tail classes as augmented training data. Additionally, we propose the integration of a Large Language Model Knowledge Guidance (LKG) module alongside a Progressive Incremental Learning (PIL) strategy to stabilize the inpainting fine-tuning process. Comprehensive evaluations conducted on the public lung datasets MIMIC and CheXpert demonstrate that the proposed method sets a new benchmark in performance.
Abstract (translated)
胸部放射影像中的长尾肺部异常呈现出了巨大的诊断挑战。尽管最近基于扩散的方法在增强尾部病灶表示方面取得了进展，但罕见病变样本的稀缺限制了这些方法的生成能力，从而导致诊断精度不尽如人意。在这篇论文中，我们提出了一种新颖的数据合成流程，旨在利用大量常规正常X光片来扩充尾部病灶。具体而言，通过收集足够的正常样本训练一个可以生成正常X光图像的扩散模型。之后，该预训练的扩散模型被用来修复患病X光中的头部病灶，从而保留尾部类别作为增强的训练数据。此外，我们还提出了一种集成大型语言模型知识指导（LKG）模块和渐进式增量学习（PIL）策略的方法，以稳定图像修补微调过程。在公开的肺部数据集MIMIC和CheXpert上进行的全面评估表明，所提出的这种方法建立了新的性能基准。
URL
https://arxiv.org/abs/2512.20980
PDF
-
Diffusion Forcing for Multi-Agent Interaction Sequence Modeling
2025-12-19 18:59:02Vongani H. Maluleke, Kie Horiuchi, Lea Wilken, Evonne Ng, Jitendra Malik, Angjoo KanazawaAbstract
Understanding and generating multi-person interactions is a fundamental challenge with broad implications for robotics and social computing. While humans naturally coordinate in groups, modeling such interactions remains difficult due to long temporal horizons, strong inter-agent dependencies, and variable group sizes. Existing motion generation methods are largely task-specific and do not generalize to flexible multi-agent generation. We introduce MAGNet (Multi-Agent Diffusion Forcing Transformer), a unified autoregressive diffusion framework for multi-agent motion generation that supports a wide range of interaction tasks through flexible conditioning and sampling. MAGNet performs dyadic prediction, partner inpainting, and full multi-agent motion generation within a single model, and can autoregressively generate ultra-long sequences spanning hundreds of v. Building on Diffusion Forcing, we introduce key modifications that explicitly model inter-agent coupling during autoregressive denoising, enabling coherent coordination across agents. As a result, MAGNet captures both tightly synchronized activities (e.g, dancing, boxing) and loosely structured social interactions. Our approach performs on par with specialized methods on dyadic benchmarks while naturally extending to polyadic scenarios involving three or more interacting people, enabled by a scalable architecture that is agnostic to the number of agents. We refer readers to the supplemental video, where the temporal dynamics and spatial coordination of generated interactions are best appreciated. Project page: this https URL
Abstract (translated)
理解并生成多人互动是机器人技术和社交计算中一个基本且具有广泛影响的挑战。虽然人类在群体活动中自然地进行协调，但由于长时间跨度、强烈的代理间依赖关系以及变化不定的群组规模，建模这些互动仍然非常困难。现有的运动生成方法大多针对特定任务，并不能很好地泛化到灵活多代理体的生成上。 我们引入了MAGNet（多代理扩散强迫变压器），这是一种统一的自回归扩散框架，用于多代理体动作生成，支持通过灵活条件和采样进行多种交互任务。MAGNet能够在单个模型中执行二元预测、伙伴补全以及完全多代理体运动生成，并能自回归地生成跨越数百帧的超长序列。基于扩散强迫技术，我们引入了关键修改，这些修改明确地在自回归去噪过程中建模代理间耦合关系，从而使得不同代理间的协调更加连贯。因此，MAGNet能够捕捉到紧密同步的活动（如跳舞、拳击）和松散结构化的社交互动。 我们的方法在二元基准测试中与专用方法表现相当，并且自然地扩展到了涉及三个或更多相互作用的人的多边场景，这是由一个不依赖于代理体数量的可伸缩架构支持的。我们鼓励读者参考项目页面上的补充视频，在那里可以更好地欣赏生成互动中的时间动态和空间协调。 项目页面: [此处应为实际链接，请参见原始文献获取]
URL
https://arxiv.org/abs/2512.17900
PDF
-
UrbanDIFF: A Denoising Diffusion Model for Spatial Gap Filling of Urban Land Surface Temperature Under Dense Cloud Cover
2025-12-19 16:51:29Arya Chavoshi, Hassan Dashtian, Naveen Sudharsan, Dev NiyogiAbstract
Satellite-derived Land Surface Temperature (LST) products are central to surface urban heat island (SUHI) monitoring due to their consistent grid-based coverage over large metropolitan regions. However, cloud contamination frequently obscures LST observations, limiting their usability for continuous SUHI analysis. Most existing LST reconstruction methods rely on multitemporal information or multisensor data fusion, requiring auxiliary observations that may be unavailable or unreliable under persistent cloud cover. Purely spatial gap-filling approaches offer an alternative, but traditional statistical methods degrade under large or spatially contiguous gaps, while many deep learning based spatial models deteriorate rapidly with increasing missingness. Recent advances in denoising diffusion based image inpainting models have demonstrated improved robustness under high missingness, motivating their adoption for spatial LST reconstruction. In this work, we introduce UrbanDIFF, a purely spatial denoising diffusion model for reconstructing cloud contaminated urban LST imagery. The model is conditioned on static urban structure information, including built-up surface data and a digital elevation model, and enforces strict consistency with revealed cloud free pixels through a supervised pixel guided refinement step during inference. UrbanDIFF is trained and evaluated using NASA MODIS Terra LST data from seven major United States metropolitan areas spanning 2002 to 2025. Experiments using synthetic cloud masks with 20 to 85 percent coverage show that UrbanDIFF consistently outperforms an interpolation baseline, particularly under dense cloud occlusion, achieving SSIM of 0.89, RMSE of 1.2 K, and R2 of 0.84 at 85 percent cloud coverage, while exhibiting slower performance degradation as cloud density increases.
Abstract (translated)
基于卫星的陆地表面温度（LST）产品对于城市热岛（UHI）监测至关重要，因为它们在大型都市区域提供了连续一致的网格覆盖。然而，云层污染经常遮蔽了LST观测值，限制了其用于持续性UHI分析的能力。大多数现有的LST重建方法依赖于多时相信息或多传感器数据融合，这些方法需要辅助观察，但在长时间的云层覆盖下可能不可用或不准确。纯空间插值法提供了另一种选择，但传统的统计方法在处理大范围或连续的空间缺口时效果较差；而基于深度学习的空间模型在缺失值增加的情况下性能迅速下降。 最近，在去噪扩散基础上进行图像修复的方法显示出更高的鲁棒性来应对高比例的缺失数据，这激发了这些方法用于空间LST重建的应用。在这项工作中，我们介绍了一种专为城市地区设计的纯空间去噪扩散模型——UrbanDIFF，用于恢复被云污染的城市LST影像。该模型结合静态城市结构信息（包括建筑表面数据和数字高程模型），并通过对未受云层遮蔽像素进行监督式的像素引导细化步骤，在推理过程中严格保证与已知清晰区域的一致性。 UrbanDIFF使用美国宇航局MODIS Terra LST数据在七个主要的美国都市区进行了训练和评估，时间跨度从2002年到2025年。实验采用合成云掩模（覆盖范围为20%至85%）显示，UrbanDIFF在所有测试条件下均优于插值基准方法，特别是在高密度云层遮蔽的情况下表现出色，达到了SSIM为0.89、RMSE为1.2K以及R²为0.84的性能指标（当云覆盖达到85%时），并且随着云量增加而显示出较慢的性能下降趋势。
URL
https://arxiv.org/abs/2512.17782
PDF
-
3One2: One-step Regression Plus One-step Diffusion for One-hot Modulation in Dual-path Video Snapshot Compressive Imaging
2025-12-19 13:44:36Ge Wang, Xing Liu, Xin YuanAbstract
Video snapshot compressive imaging (SCI) captures dynamic scene sequences through a two-dimensional (2D) snapshot, fundamentally relying on optical modulation for hardware compression and the corresponding software reconstruction. While mainstream video SCI using random binary modulation has demonstrated success, it inevitably results in temporal aliasing during compression. One-hot modulation, activating only one sub-frame per pixel, provides a promising solution for achieving perfect temporal decoupling, thereby alleviating issues associated with aliasing. However, no algorithms currently exist to fully exploit this potential. To bridge this gap, we propose an algorithm specifically designed for one-hot masks. First, leveraging the decoupling properties of one-hot modulation, we transform the reconstruction task into a generative video inpainting problem and introduce a stochastic differential equation (SDE) of the forward process that aligns with the hardware compression process. Next, we identify limitations of the pure diffusion method for video SCI and propose a novel framework that combines one-step regression initialization with one-step diffusion refinement. Furthermore, to mitigate the spatial degradation caused by one-hot modulation, we implement a dual optical path at the hardware level, utilizing complementary information from another path to enhance the inpainted video. To our knowledge, this is the first work integrating diffusion into video SCI reconstruction. Experiments conducted on synthetic datasets and real scenes demonstrate the effectiveness of our method.
Abstract (translated)
视频快照压缩成像（SCI）通过二维快照捕获动态场景序列，其基础是依赖光学调制进行硬件压缩，并配合相应的软件重建。虽然主流的使用随机二进制调制的视频SCI已经取得了成功，但在压缩过程中不可避免地会产生时域混叠问题。一种称为“一对一”调制的方法，即每次仅激活每个像素的一个子帧，为实现完美的时间解耦提供了一个有前景的解决方案，从而缓解了与混叠相关的问题。然而，目前还没有算法能够充分利用这一潜力。 为了弥补这一空白，我们提出了一种专门针对一对一掩码设计的新算法。首先，利用一对一调制的时间解耦特性，我们将重建任务转化为生成视频插值问题，并引入了一个与硬件压缩过程相匹配的前向随机微分方程（SDE）。接下来，我们识别了纯扩散方法在视频SCI中的局限性，并提出了一种新的框架，该框架结合了一步回归初始化和一步扩散细化。此外，为了缓解一对一调制造成的空间退化问题，我们在硬件层面实施了一个双光路方案，利用另一条路径的互补信息来增强插值后的视频。 据我们所知，这是首次将扩散方法整合到视频SCI重建中的一项工作。在合成数据集和真实场景上的实验展示了我们的方法的有效性。
URL
https://arxiv.org/abs/2512.17578
PDF
-
RoomEditor++: A Parameter-Sharing Diffusion Architecture for High-Fidelity Furniture Synthesis
2025-12-19 13:39:43Qilong Wang, Xiaofan Ming, Zhenyi Lin, Jinwen Li, Dongwei Ren, Wangmeng Zuo, Qinghua HuAbstract
Virtual furniture synthesis, which seamlessly integrates reference objects into indoor scenes while maintaining geometric coherence and visual realism, holds substantial promise for home design and e-commerce applications. However, this field remains underexplored due to the scarcity of reproducible benchmarks and the limitations of existing image composition methods in achieving high-fidelity furniture synthesis while preserving background integrity. To overcome these challenges, we first present RoomBench++, a comprehensive and publicly available benchmark dataset tailored for this task. It consists of 112,851 training pairs and 1,832 testing pairs drawn from both real-world indoor videos and realistic home design renderings, thereby supporting robust training and evaluation under practical conditions. Then, we propose RoomEditor++, a versatile diffusion-based architecture featuring a parameter-sharing dual diffusion backbone, which is compatible with both U-Net and DiT architectures. This design unifies the feature extraction and inpainting processes for reference and background images. Our in-depth analysis reveals that the parameter-sharing mechanism enforces aligned feature representations, facilitating precise geometric transformations, texture preservation, and seamless integration. Extensive experiments validate that RoomEditor++ is superior over state-of-the-art approaches in terms of quantitative metrics, qualitative assessments, and human preference studies, while highlighting its strong generalization to unseen indoor scenes and general scenes without task-specific fine-tuning. The dataset and source code are available at \url{this https URL}.
Abstract (translated)
虚拟家具合成技术将参考对象无缝集成到室内场景中，同时保持几何一致性和视觉逼真度，在家庭设计和电子商务应用方面展现出巨大的潜力。然而，由于缺乏可重复使用的基准数据集以及现有图像组合方法在实现高保真家具合成的同时难以保留背景完整性的限制，这一领域仍处于探索阶段。 为了克服这些挑战，我们首先提出了RoomBench++，这是一个全面且公开可用的基准数据集，专门针对此类任务设计。该数据集包括112,851对训练样本和1,832对测试样本，分别从真实世界的室内视频和逼真的家居设计渲染中提取，从而支持在实际条件下稳健地进行训练和评估。 接着，我们提出了RoomEditor++，这是一种基于扩散的多功能架构，具有参数共享双扩散主干结构，并且可以兼容U-Net和DiT架构。该设计统一了参考图像与背景图像的特征提取和修复过程。深入分析表明，参数共享机制强制执行一致的特征表示，从而促进了精确的几何变换、纹理保持以及无缝集成。 广泛的实验验证显示，RoomEditor++在定量指标、定性评估和人类偏好研究方面均优于当前最先进的方法，并且表现出强大的泛化能力，在未见过的室内场景及一般场景中无需特定任务微调就能发挥作用。数据集和源代码可在此网址获取：\url{this https URL}。
URL
https://arxiv.org/abs/2512.17573
PDF
-
RePlan: Reasoning-guided Region Planning for Complex Instruction-based Image Editing
2025-12-18 18:34:23Tianyuan Qu, Lei Ke, Xiaohang Zhan, Longxiang Tang, Yuqi Liu, Bohao Peng, Bei Yu, Dong Yu, Jiaya JiaAbstract
Instruction-based image editing enables natural-language control over visual modifications, yet existing models falter under Instruction-Visual Complexity (IV-Complexity), where intricate instructions meet cluttered or ambiguous scenes. We introduce RePlan (Region-aligned Planning), a plan-then-execute framework that couples a vision-language planner with a diffusion editor. The planner decomposes instructions via step-by-step reasoning and explicitly grounds them to target regions; the editor then applies changes using a training-free attention-region injection mechanism, enabling precise, parallel multi-region edits without iterative inpainting. To strengthen planning, we apply GRPO-based reinforcement learning using 1K instruction-only examples, yielding substantial gains in reasoning fidelity and format reliability. We further present IV-Edit, a benchmark focused on fine-grained grounding and knowledge-intensive edits. Across IV-Complex settings, RePlan consistently outperforms strong baselines trained on far larger datasets, improving regional precision and overall fidelity. Our project page: this https URL
Abstract (translated)
指令驱动的图像编辑允许通过自然语言对视觉修改进行控制，但现有的模型在处理复杂指令与混乱或含糊场景相结合的情况（即Instruction-Visual Complexity, IV-Complexity）时会遇到困难。我们提出了RePlan（Region-aligned Planning），这是一种“先规划后执行”的框架，它结合了视觉语言规划器和扩散编辑器。该规划器通过逐步推理分解指令，并明确将其与目标区域对齐；随后，编辑器利用无需训练的注意力-区域注入机制进行更改操作，从而支持精确且并行的多区域修改，而不需要迭代修复过程。 为了增强规划效果，我们应用了基于GRPO（Graph Regularized Policy Optimization）的强化学习方法，并使用仅包含1000个指令示例的数据集进行了训练，这种方法在推理准确性和格式可靠性方面取得了显著提升。此外，我们还介绍了IV-Edit，这是一个专注于细粒度对齐和知识密集型编辑任务的新基准测试。 在复杂的IV设置下，RePlan框架相对于那些基于更大数据集（远超过1000个样本）训练的强大基线模型而言，在区域精度和整体精确性方面表现出持续的优越性能。欲了解更多详情，请访问我们的项目页面：[点击这里](this https URL)。
URL
https://arxiv.org/abs/2512.16864
PDF
-
Detecting Localized Deepfakes: How Well Do Synthetic Image Detectors Handle Inpainting?
2025-12-18 15:54:51Serafino Pandolfini, Lorenzo Pellegrini, Matteo Ferrara, Davide MaltoniAbstract
The rapid progress of generative AI has enabled highly realistic image manipulations, including inpainting and region-level editing. These approaches preserve most of the original visual context and are increasingly exploited in cybersecurity-relevant threat scenarios. While numerous detectors have been proposed for identifying fully synthetic images, their ability to generalize to localized manipulations remains insufficiently characterized. This work presents a systematic evaluation of state-of-the-art detectors, originally trained for the deepfake detection on fully synthetic images, when applied to a distinct challenge: localized inpainting detection. The study leverages multiple datasets spanning diverse generators, mask sizes, and inpainting techniques. Our experiments show that models trained on a large set of generators exhibit partial transferability to inpainting-based edits and can reliably detect medium- and large-area manipulations or regeneration-style inpainting, outperforming many existing ad hoc detection approaches.
Abstract (translated)
生成式人工智能的迅速进步使得高度逼真的图像操纵成为可能，包括修复绘画（inpainting）和区域级别的编辑。这些方法保留了大部分原始视觉上下文，并且在网络安全相关的威胁场景中被越来越多地利用。虽然已经提出了多种用于识别完全合成图像的检测器，但它们对局部修改的泛化能力尚未充分表征。这项工作系统性地评估了最先进的检测器，在其原本用于深度伪造图像检测的基础上，当应用于不同的挑战时的表现：即局部修复绘画（inpainting）的检测。该研究利用了涵盖多种生成模型、遮罩大小和修复技术的多个数据集。我们的实验表明，基于大量生成模型训练出的模型在面对基于修复绘画的编辑时表现出部分可转移性，并且能够可靠地检测到中等面积和大面积的人为改动或再生风格的修复绘画，优于许多现有的临时检测方法。
URL
https://arxiv.org/abs/2512.16688
PDF
-
BLANKET: Anonymizing Faces in Infant Video Recordings
2025-12-17 15:49:56Ditmar Hadera, Jan Cech, Miroslav Purkrabek, Matej HoffmannAbstract
Ensuring the ethical use of video data involving human subjects, particularly infants, requires robust anonymization methods. We propose BLANKET (Baby-face Landmark-preserving ANonymization with Keypoint dEtection consisTency), a novel approach designed to anonymize infant faces in video recordings while preserving essential facial attributes. Our method comprises two stages. First, a new random face, compatible with the original identity, is generated via inpainting using a diffusion model. Second, the new identity is seamlessly incorporated into each video frame through temporally consistent face swapping with authentic expression transfer. The method is evaluated on a dataset of short video recordings of babies and is compared to the popular anonymization method, DeepPrivacy2. Key metrics assessed include the level of de-identification, preservation of facial attributes, impact on human pose estimation (as an example of a downstream task), and presence of artifacts. Both methods alter the identity, and our method outperforms DeepPrivacy2 in all other respects. The code is available as an easy-to-use anonymization demo at this https URL.
Abstract (translated)
确保涉及人类主体（特别是婴儿）的视频数据的伦理使用，需要强大的匿名化方法。我们提出了一种名为BLANKET（Baby-face Landmark-preserving ANonymization with Keypoint dEtection consisTency）的新方法，旨在在保留面部关键特征的前提下对婴儿脸部进行匿名处理。该方法分为两个阶段：首先，通过使用扩散模型进行图像修复，生成一个与原始身份兼容的新的随机面孔；其次，将新身份无缝地融合到每个视频帧中，并且保持时间上的一致性以及真实表情的传递。 我们在一组短的婴儿视频记录数据集上评估了该方法，并将其与流行的匿名化方法DeepPrivacy2进行了比较。评价指标包括去识别水平、面部特征保留情况、对人类姿态估计（作为下游任务的一个示例）的影响，以及是否存在人工痕迹。两种方法都会改变身份，但在其他所有方面，我们的方法均优于DeepPrivacy2。 该代码已作为一个易于使用的匿名化演示工具公开提供，在此链接中可以获取：[请在此处插入实际URL]
URL
https://arxiv.org/abs/2512.15542
PDF
-
EMFusion: Conditional Diffusion Framework for Trustworthy Frequency Selective EMF Forecasting in Wireless Networks
2025-12-17 04:12:52Zijiang Yan, Yixiang Huang, Jianhua Pei, Hina Tabassum, Luca ChiaraviglioAbstract
The rapid growth in wireless infrastructure has increased the need to accurately estimate and forecast electromagnetic field (EMF) levels to ensure ongoing compliance, assess potential health impacts, and support efficient network planning. While existing studies rely on univariate forecasting of wideband aggregate EMF data, frequency-selective multivariate forecasting is needed to capture the inter-operator and inter-frequency variations essential for proactive network planning. To this end, this paper introduces EMFusion, a conditional multivariate diffusion-based probabilistic forecasting framework that integrates diverse contextual factors (e.g., time of day, season, and holidays) while providing explicit uncertainty estimates. The proposed architecture features a residual U-Net backbone enhanced by a cross-attention mechanism that dynamically integrates external conditions to guide the generation process. Furthermore, EMFusion integrates an imputation-based sampling strategy that treats forecasting as a structural inpainting task, ensuring temporal coherence even with irregular measurements. Unlike standard point forecasters, EMFusion generates calibrated probabilistic prediction intervals directly from the learned conditional distribution, providing explicit uncertainty quantification essential for trustworthy decision-making. Numerical experiments conducted on frequency-selective EMF datasets demonstrate that EMFusion with the contextual information of working hours outperforms the baseline models with or without conditions. The EMFusion outperforms the best baseline by 23.85% in continuous ranked probability score (CRPS), 13.93% in normalized root mean square error, and reduces prediction CRPS error by 22.47%.
Abstract (translated)
无线基础设施的快速增长增加了对准确估计和预测电磁场（EMF）水平的需求，以确保持续合规性、评估潜在健康影响和支持高效网络规划。尽管现有研究依赖于宽带聚合EMF数据的单变量预测，但为了捕捉运营商间和频段间的变动情况——这些对于主动网络规划至关重要——需要进行频率选择性的多变量预测。 为此，本文介绍了EMFusion，这是一种基于条件多变量扩散的概率预测框架，它结合了多样化的上下文因素（如一天中的时间、季节和假日），同时提供明确的不确定性估计。所提出的架构以残差U-Net骨干为基础，并通过交叉注意机制增强了外部条件的动态整合，以引导生成过程。此外，EMFusion集成了基于插补的采样策略，将预测视为结构修复任务，即使在不规则测量的情况下也能确保时间一致性。与标准点预测器不同的是，EMFusion直接从学习到的条件分布中生成校准的概率预测区间，提供了进行信任决策所需的具体不确定性量化。 在频率选择性EMF数据集上进行的数值实验表明，在结合工作时间上下文信息后，EMFusion的表现优于有或无条件下的基线模型。与最佳基准相比，EMFusion在连续排名概率评分（CRPS）中提高了23.85%，在归一化均方根误差中提高了13.93%，将预测CRPS误差减少了22.47%。
URL
https://arxiv.org/abs/2512.15067
PDF
-
InpaintDPO: Mitigating Spatial Relationship Hallucinations in Foreground-conditioned Inpainting via Diverse Preference Optimization
2025-12-16 17:55:22Qirui Li, Yizhe Tang, Ran Yi, Guangben Lu, Fangyuan Zou, Peng Shu, Huan Yu, Jie JiangAbstract
Foreground-conditioned inpainting, which aims at generating a harmonious background for a given foreground subject based on the text prompt, is an important subfield in controllable image generation. A common challenge in current methods, however, is the occurrence of Spatial Relationship Hallucinations between the foreground subject and the generated background, including inappropriate scale, positional relationships, and viewpoints. Critically, the subjective nature of spatial rationality makes it challenging to quantify, hindering the use of traditional reward-based RLHF methods. To address this issue, we propose InpaintDPO, the first Direct Preference Optimization (DPO) based framework dedicated to spatial rationality in foreground-conditioned inpainting, ensuring plausible spatial relationships between foreground and background elements. To resolve the gradient conflicts in standard DPO caused by identical foreground in win-lose pairs, we propose MaskDPO, which confines preference optimization exclusively to the background to enhance background spatial relationships, while retaining the inpainting loss in the foreground region for robust foreground preservation. To enhance coherence at the foreground-background boundary, we propose Conditional Asymmetric Preference Optimization, which samples pairs with differentiated cropping operations and applies global preference optimization to promote contextual awareness and enhance boundary coherence. Finally, based on the observation that winning samples share a commonality in plausible spatial relationships, we propose Shared Commonality Preference Optimization to enhance the model's understanding of spatial commonality across high-quality winning samples, further promoting shared spatial rationality.
Abstract (translated)
前景条件下的图像修复（Foreground-conditioned inpainting）旨在根据文本提示生成与给定前景主体相协调的背景，这是可控图像生成领域的一个重要子领域。然而，当前方法面临的一个常见挑战是前景主体和生成背景之间会出现空间关系幻觉问题，包括不适当的尺度、位置关系以及视角问题。特别地，空间合理性的主观性使其难以量化，从而阻碍了传统基于奖励的RLHF（Reinforcement Learning from Human Feedback）方法的应用。 为解决这一问题，我们提出了InpaintDPO——第一个专注于前景条件修复中空间合理性问题的直接偏好优化(Direct Preference Optimization, DPO)框架，以确保前景和背景元素之间有合理的空间关系。为了缓解标准DPO在胜者与败者配对时由于前景相同而造成的梯度冲突问题，我们提出了MaskDPO方法，该方法将偏好优化严格限制在背景部分以增强背景的空间关系，同时保留前景区域的修复损失以确保稳固地保持前景内容。 为提高前景和背景边界处的一致性，我们提出了一种条件非对称偏好优化（Conditional Asymmetric Preference Optimization）方法。这种方法通过采用不同的裁剪操作来采样配对样本，并应用全局偏好优化以增强上下文感知能力并改善边界一致性。 最后，基于观察到获胜的样本在合理的空间关系上具有共同点这一发现，我们提出了共享共性偏奷新框架（Shared Commonality Preference Optimization）。该方法通过分析高质量的胜利样本之间的空间共性来提高模型对空间普遍性的理解，并进一步推广这种共享的空间合理性。
URL
https://arxiv.org/abs/2512.15644
PDF
-
End-to-End Learning-based Video Streaming Enhancement Pipeline: A Generative AI Approach
2025-12-16 08:28:53Emanuele Artioli, Farzad Tashtarian, Christian TimmererAbstract
The primary challenge of video streaming is to balance high video quality with smooth playback. Traditional codecs are well tuned for this trade-off, yet their inability to use context means they must encode the entire video data and transmit it to the client. This paper introduces ELVIS (End-to-end Learning-based VIdeo Streaming Enhancement Pipeline), an end-to-end architecture that combines server-side encoding optimizations with client-side generative in-painting to remove and reconstruct redundant video data. Its modular design allows ELVIS to integrate different codecs, inpainting models, and quality metrics, making it adaptable to future innovations. Our results show that current technologies achieve improvements of up to 11 VMAF points over baseline benchmarks, though challenges remain for real-time applications due to computational demands. ELVIS represents a foundational step toward incorporating generative AI into video streaming pipelines, enabling higher quality experiences without increased bandwidth requirements.
Abstract (translated)
视频流传输的主要挑战在于平衡高质量的视频与流畅播放之间的关系。传统编解码器在这一权衡中表现良好，但由于它们无法利用上下文信息，必须对整个视频数据进行编码并将其传送到客户端。本文介绍了 ELVIS（基于端到端学习的视频流增强管道），这是一种结合了服务器端编码优化和客户端生成式修复技术来移除和重建冗余视频数据的端到端架构。ELVIS 的模块化设计使其能够整合不同的编解码器、修补模型以及质量指标，从而适应未来的创新需求。我们的结果表明，当前的技术相比基准测试可提升高达 11 个 VMAF 分数（一种视频质量评估标准），尽管实时应用仍面临计算资源的挑战。ELVIS 标志着将生成式人工智能融入视频流传输管道的一个基础性步骤，能够在不增加带宽需求的情况下提供更高质量的观看体验。
URL
https://arxiv.org/abs/2512.14185
PDF
-
TWLR: Text-Guided Weakly-Supervised Lesion Localization and Severity Regression for Explainable Diabetic Retinopathy Grading
2025-12-15 06:08:16Xi Luo, Shixin Xu, Ying Xie, JianZhong Hu, Yuwei He, Yuhui Deng, Huaxiong HuangAbstract
Accurate medical image analysis can greatly assist clinical diagnosis, but its effectiveness relies on high-quality expert annotations Obtaining pixel-level labels for medical images, particularly fundus images, remains costly and time-consuming. Meanwhile, despite the success of deep learning in medical imaging, the lack of interpretability limits its clinical adoption. To address these challenges, we propose TWLR, a two-stage framework for interpretable diabetic retinopathy (DR) assessment. In the first stage, a vision-language model integrates domain-specific ophthalmological knowledge into text embeddings to jointly perform DR grading and lesion classification, effectively linking semantic medical concepts with visual features. The second stage introduces an iterative severity regression framework based on weakly-supervised semantic segmentation. Lesion saliency maps generated through iterative refinement direct a progressive inpainting mechanism that systematically eliminates pathological features, effectively downgrading disease severity toward healthier fundus appearances. Critically, this severity regression approach achieves dual benefits: accurate lesion localization without pixel-level supervision and providing an interpretable visualization of disease-to-healthy transformations. Experimental results on the FGADR, DDR, and a private dataset demonstrate that TWLR achieves competitive performance in both DR classification and lesion segmentation, offering a more explainable and annotation-efficient solution for automated retinal image analysis.
Abstract (translated)
准确的医学影像分析能够极大地辅助临床诊断，但其有效性依赖于高质量的专业标注。获取医学图像（尤其是眼底图像）的像素级标签既耗时又昂贵。尽管深度学习在医学成像领域的成功已经取得突破，但缺乏可解释性限制了其临床应用。为了解决这些问题，我们提出了TWLR，这是一种用于可解释的糖尿病视网膜病变（DR）评估的两阶段框架。 第一阶段中，视觉-语言模型整合特定领域的专业眼科知识到文本嵌入中，从而同时执行DR分级和病变分类。这有效地将语义医学概念与视觉特征联系起来。 第二阶段引入了一个基于弱监督语义分割的迭代严重程度回归框架。通过迭代细化生成的病变显著图引导渐进式填充机制，系统地消除病理特征，使疾病严重程度逐步向更健康的眼底外观退化。关键在于这一严重度回归方法实现了双重效果：在没有像素级监督的情况下实现准确的病变定位，并提供可解释的疾病到健康转变可视化。 实验结果表明，在FGADR、DDR以及一个私有数据集上，TWLR在DR分类和病变分割方面均达到了竞争性的性能，为自动视网膜图像分析提供了更加可解释且标注效率更高的解决方案。
URL
https://arxiv.org/abs/2512.13008
PDF
-
UnReflectAnything: RGB-Only Highlight Removal by Rendering Synthetic Specular Supervision
2025-12-10 12:22:37Alberto Rota, Mert Kiray, Mert Asim Karaoglu, Patrick Ruhkamp, Elena De Momi, Nassir Navabm, Benjamin BusamAbstract
Specular highlights distort appearance, obscure texture, and hinder geometric reasoning in both natural and surgical imagery. We present UnReflectAnything, an RGB-only framework that removes highlights from a single image by predicting a highlight map together with a reflection-free diffuse reconstruction. The model uses a frozen vision transformer encoder to extract multi-scale features, a lightweight head to localize specular regions, and a token-level inpainting module that restores corrupted feature patches before producing the final diffuse image. To overcome the lack of paired supervision, we introduce a Virtual Highlight Synthesis pipeline that renders physically plausible specularities using monocular geometry, Fresnel-aware shading, and randomized lighting which enables training on arbitrary RGB images with correct geometric structure. UnReflectAnything generalizes across natural and surgical domains where non-Lambertian surfaces and non-uniform lighting create severe highlights and it achieves competitive performance with state-of-the-art results on several benchmarks. Project Page: this https URL
Abstract (translated)
镜面高光会扭曲外观，遮蔽纹理，并阻碍几何推理，在自然和外科图像中均会出现这种情况。我们提出了 UnReflectAnything，这是一个仅基于 RGB 的框架，通过预测高光图并生成无反射的漫反射重建来从单张图像中移除高光。该模型使用冻结的视觉变换器编码器提取多尺度特征、一个轻量级头部用于定位镜面区域，并且在输出最终漫反射图像之前，采用令牌级别的修复模块恢复被破坏的特征块。 为了克服配对监督不足的问题，我们引入了一个虚拟高光合成流水线，该流水线使用单目几何、符合菲涅尔定律的着色以及随机光线来生成物理上合理的镜面效果。这使得模型可以利用具有正确几何结构的任意 RGB 图像进行训练。 UnReflectAnything 能够在自然和外科领域中跨域泛化，在非朗伯表面和不均匀照明条件下产生的严重高光情况下表现良好，并且在多个基准测试中取得了与现有最先进的方法相当甚至更好的性能。项目页面：[请在此处插入正确的网址链接]
URL
https://arxiv.org/abs/2512.09583
PDF
-
DirectSwap: Mask-Free Cross-Identity Training and Benchmarking for Expression-Consistent Video Head Swapping
2025-12-10 08:31:28Yanan Wang, Shengcai Liao, Panwen Hu, Xin Li, Fan Yang, Xiaodan LiangAbstract
Video head swapping aims to replace the entire head of a video subject, including facial identity, head shape, and hairstyle, with that of a reference image, while preserving the target body, background, and motion dynamics. Due to the lack of ground-truth paired swapping data, prior methods typically train on cross-frame pairs of the same person within a video and rely on mask-based inpainting to mitigate identity leakage. Beyond potential boundary artifacts, this paradigm struggles to recover essential cues occluded by the mask, such as facial pose, expressions, and motion dynamics. To address these issues, we prompt a video editing model to synthesize new heads for existing videos as fake swapping inputs, while maintaining frame-synchronized facial poses and expressions. This yields HeadSwapBench, the first cross-identity paired dataset for video head swapping, which supports both training (\TrainNum{} videos) and benchmarking (\TestNum{} videos) with genuine outputs. Leveraging this paired supervision, we propose DirectSwap, a mask-free, direct video head-swapping framework that extends an image U-Net into a video diffusion model with a motion module and conditioning inputs. Furthermore, we introduce the Motion- and Expression-Aware Reconstruction (MEAR) loss, which reweights the diffusion loss per pixel using frame-difference magnitudes and facial-landmark proximity, thereby enhancing cross-frame coherence in motion and expressions. Extensive experiments demonstrate that DirectSwap achieves state-of-the-art visual quality, identity fidelity, and motion and expression consistency across diverse in-the-wild video scenes. We will release the source code and the HeadSwapBench dataset to facilitate future research.
Abstract (translated)
视频头部交换的目标是用参考图像中的整个头部（包括面部识别、头形和发型）替换视频主体的头部，同时保留目标身体、背景及运动动态。由于缺乏配对的真实交换数据，先前的方法通常在来自同一视频内不同帧的跨帧配对上进行训练，并依赖基于掩码的修复技术来减少身份信息泄漏。除了可能存在的边界伪影之外，这种范式难以恢复被掩码遮挡的重要线索，例如面部姿态、表情和运动动态。 为了解决这些问题，我们提示一个视频编辑模型生成用于现有视频的人造头部交换输入，同时保持逐帧同步的面部姿势和表情。这产生了HeadSwapBench，这是首个用于视频头部交换的真实配对数据集，支持训练（\TrainNum{}个视频）和基准测试（\TestNum{}个视频），并提供真实输出。利用这种配对监督，我们提出了DirectSwap，这是一种无掩码的直接视频头部交换框架，它将图像U-Net扩展为带有运动模块和条件输入的视频扩散模型。此外，我们引入了基于运动和表情感知重建（MEAR）损失，该损失通过帧差值幅度和面部特征点接近度来重新加权扩散损失每像素权重，从而增强跨帧在动作与表达上的一致性。 大量的实验表明，DirectSwap实现了视觉质量、身份忠实度以及在各种野外视频场景中的运动和表情一致性方面的最新技术水平。我们将发布源代码和HeadSwapBench数据集以促进未来的研究工作。
URL
https://arxiv.org/abs/2512.09417
PDF
-
H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos
2025-12-10 07:59:45Hai Ci, Xiaokang Liu, Pei Yang, Yiren Song, Mike Zheng ShouAbstract
Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: this https URL
Abstract (translated)
从日常人类视频中学习操作技能的机器人可以无需繁琐的数据收集过程就能获得广泛的能力。我们提出了一种视频到视频转换框架，该框架能够将普通的“人-物”互动视频转化为带有真实且物理上合理的机器人操作动作的视频。我们的方法在训练时不需要配对的人类和机器人的视频数据，仅需要一组未配对的机器人视频，使得系统易于扩展。 我们引入了一种可转移表示形式，以弥合实体差距：通过在训练视频中绘制机械臂来获得干净的背景，并叠加一个简单的视觉线索（标记物和指示夹持器位置和方向的箭头），我们可以使生成模型有条件地将机械臂重新插入场景。在测试阶段，我们对人类视频执行同样的过程（擦除人物并叠加人体姿态提示），从而能够产生高质量的机器人视频，这些视频模仿了人类的动作。 我们将最先进的视频扩散模型（Wan 2.2）以上下文学习的方式进行微调，确保其时间一致性，并充分利用其中丰富的先验知识。实证结果表明，与基线方法相比，我们的方法实现了更真实和基于物理的操作动作，预示着从未标记的人类视频中大规模训练机器人具有广阔前景。 项目主页：[此链接](https://this-url.com)
URL
https://arxiv.org/abs/2512.09406
PDF
