Inpainting
-
Bridging Knowledge Gap Between Image Inpainting and Large-Area Visible Watermark Removal
2025-04-07 02:37:14Yicheng Leng, Chaowei Fang, Junye Chen, Yixiang Fang, Sheng Li, Guanbin LiAbstract
Visible watermark removal which involves watermark cleaning and background content restoration is pivotal to evaluate the resilience of watermarks. Existing deep neural network (DNN)-based models still struggle with large-area watermarks and are overly dependent on the quality of watermark mask prediction. To overcome these challenges, we introduce a novel feature adapting framework that leverages the representation modeling capacity of a pre-trained image inpainting model. Our approach bridges the knowledge gap between image inpainting and watermark removal by fusing information of the residual background content beneath watermarks into the inpainting backbone model. We establish a dual-branch system to capture and embed features from the residual background content, which are merged into intermediate features of the inpainting backbone model via gated feature fusion modules. Moreover, for relieving the dependence on high-quality watermark masks, we introduce a new training paradigm by utilizing coarse watermark masks to guide the inference process. This contributes to a visible image removal model which is insensitive to the quality of watermark mask during testing. Extensive experiments on both a large-scale synthesized dataset and a real-world dataset demonstrate that our approach significantly outperforms existing state-of-the-art methods. The source code is available in the supplementary materials.
Abstract (translated)
可见水印去除技术，包括水印清洗和背景内容恢复，在评估水印的鲁棒性方面至关重要。现有的基于深度神经网络（DNN）的模型在处理大面积水印时仍然面临挑战，并且过度依赖于水印掩码预测的质量。为了解决这些问题，我们引入了一个新颖的功能适应框架，该框架利用了预训练图像修复模型的表示建模能力。我们的方法通过将残余背景内容的信息融入到图像修复主干模型中，填补了图像修复和水印去除之间的知识空白。我们建立了一个双分支系统来捕获并嵌入来自水印下的剩余背景内容的功能，并通过门控特征融合模块将其合并到图像修复主干模型的中间特性中。此外，为了减轻对高质量水印掩码的依赖性，我们引入了一种新的训练范式，利用粗略水印掩码来指导推理过程。这有助于创建一个在测试过程中对水印掩码质量不敏感的可见图片去除模型。广泛的实验结果表明，在大规模合成数据集和真实世界数据集上，我们的方法显著优于现有的最先进的方法。源代码可在补充材料中找到。
URL
https://arxiv.org/abs/2504.04687
PDF
-
VIP: Video Inpainting Pipeline for Real World Human Removal
2025-04-03 21:40:10Huiming Sun, Yikang Li, Kangning Yang, Ruineng Li, Daitao Xing, Yangbo Xie, Lan Fu, Kaiyu Zhang, Ming Chen, Jiaming Ding, Jiang Geng, Jie Cai, Zibo Meng, Chiuman HoAbstract
Inpainting for real-world human and pedestrian removal in high-resolution video clips presents significant challenges, particularly in achieving high-quality outcomes, ensuring temporal consistency, and managing complex object interactions that involve humans, their belongings, and their shadows. In this paper, we introduce VIP (Video Inpainting Pipeline), a novel promptless video inpainting framework for real-world human removal applications. VIP enhances a state-of-the-art text-to-video model with a motion module and employs a Variational Autoencoder (VAE) for progressive denoising in the latent space. Additionally, we implement an efficient human-and-belongings segmentation for precise mask generation. Sufficient experimental results demonstrate that VIP achieves superior temporal consistency and visual fidelity across diverse real-world scenarios, surpassing state-of-the-art methods on challenging datasets. Our key contributions include the development of the VIP pipeline, a reference frame integration technique, and the Dual-Fusion Latent Segment Refinement method, all of which address the complexities of inpainting in long, high-resolution video sequences.
Abstract (translated)
在高分辨率视频片段中去除真实世界中的行人和人类，以实现高质量的结果、确保时间一致性以及处理涉及人类及其随身物品和阴影的复杂对象交互，面临着重大挑战。本文介绍了VIP（Video Inpainting Pipeline），这是一个新颖的无提示视频修复框架，专门用于现实世界的行人移除应用。VIP通过引入一个运动模块增强了最先进的文本到视频模型，并利用变分自编码器（VAE）在潜在空间中进行渐进式去噪处理。此外，我们还实施了一个高效的分割方法来生成精确的人及其随身物品的掩码。充分的实验结果表明，VIP在各种现实世界场景中实现了卓越的时间一致性和视觉保真度，在具有挑战性的数据集上超越了现有最先进的方法。我们的主要贡献包括开发VIP管道、参考帧集成技术以及双融合潜在段细化方法，这些都解决了长时间高分辨率视频序列中的修复复杂性问题。
URL
https://arxiv.org/abs/2504.03041
PDF
-
WonderTurbo: Generating Interactive 3D World in 0.72 Seconds
2025-04-03 04:10:47Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Weijie Wang, Haoyun Li, Guosheng Zhao, Jie Li, Wenkang Qin, Guan Huang, Wenjun MeiAbstract
Interactive 3D generation is gaining momentum and capturing extensive attention for its potential to create immersive virtual experiences. However, a critical challenge in current 3D generation technologies lies in achieving real-time interactivity. To address this issue, we introduce WonderTurbo, the first real-time interactive 3D scene generation framework capable of generating novel perspectives of 3D scenes within 0.72 seconds. Specifically, WonderTurbo accelerates both geometric and appearance modeling in 3D scene generation. In terms of geometry, we propose StepSplat, an innovative method that constructs efficient 3D geometric representations through dynamic updates, each taking only 0.26 seconds. Additionally, we design QuickDepth, a lightweight depth completion module that provides consistent depth input for StepSplat, further enhancing geometric accuracy. For appearance modeling, we develop FastPaint, a 2-steps diffusion model tailored for instant inpainting, which focuses on maintaining spatial appearance consistency. Experimental results demonstrate that WonderTurbo achieves a remarkable 15X speedup compared to baseline methods, while preserving excellent spatial consistency and delivering high-quality output.
Abstract (translated)
交互式3D生成技术正逐渐兴起，并因其创造沉浸式虚拟体验的潜力而备受关注。然而，当前3D生成技术的一个关键挑战在于实现实时互动性。为了解决这一问题，我们推出了WonderTurbo——首个能够实现实时交互式3D场景生成框架，它能够在0.72秒内生成新的3D场景视角。具体而言，WonderTurbo通过加速几何和外观建模来提高3D场景生成的速度。 在几何方面，我们提出了StepSplat方法，这是一种创新的动态更新技术，能够构建高效的3D几何表示，并且每次仅需花费0.26秒。此外，我们设计了QuickDepth轻量级深度完成模块，为StepSplat提供了一致性的深度输入，从而进一步提高几何精度。 在外观建模方面，我们开发了FastPaint——一个专为即时修复绘制而定制的两步扩散模型，专注于保持空间外观的一致性。实验结果表明，与基准方法相比，WonderTurbo实现了15倍的速度提升，并且能够维持出色的空间一致性并输出高质量的结果。
URL
https://arxiv.org/abs/2504.02261
PDF
-
A$^text{T}$A: Adaptive Transformation Agent for Text-Guided Subject-Position Variable Background Inpainting
2025-04-02 11:13:46Yizhe Tang, Zhimin Sun, Yuzhen Du, Ran Yi, Guangben Lu, Teng Hu, Luying Li, Lizhuang Ma, Fangyuan ZouAbstract
Image inpainting aims to fill the missing region of an image. Recently, there has been a surge of interest in foreground-conditioned background inpainting, a sub-task that fills the background of an image while the foreground subject and associated text prompt are provided. Existing background inpainting methods typically strictly preserve the subject's original position from the source image, resulting in inconsistencies between the subject and the generated background. To address this challenge, we propose a new task, the "Text-Guided Subject-Position Variable Background Inpainting", which aims to dynamically adjust the subject position to achieve a harmonious relationship between the subject and the inpainted background, and propose the Adaptive Transformation Agent (A$^\text{T}$A) for this task. Firstly, we design a PosAgent Block that adaptively predicts an appropriate displacement based on given features to achieve variable subject-position. Secondly, we design the Reverse Displacement Transform (RDT) module, which arranges multiple PosAgent blocks in a reverse structure, to transform hierarchical feature maps from deep to shallow based on semantic information. Thirdly, we equip A$^\text{T}$A with a Position Switch Embedding to control whether the subject's position in the generated image is adaptively predicted or fixed. Extensive comparative experiments validate the effectiveness of our A$^\text{T}$A approach, which not only demonstrates superior inpainting capabilities in subject-position variable inpainting, but also ensures good performance on subject-position fixed inpainting.
Abstract (translated)
图像修复技术旨在填补图像中的缺失区域。近期，人们对前景条件下的背景修复（即在给定前景对象和相关文本提示的情况下填充图像的背景）产生了浓厚的兴趣。现有的背景修复方法通常严格保持主体在源图中的原始位置不变，这会导致修复后的背景与主体之间产生不协调的问题。为解决这一挑战，我们提出了一项新任务：“基于文本指导的位置可变背景修复”，旨在动态调整对象位置，以实现生成的背景与对象之间的和谐关系，并为此任务提出了自适应变换代理（Adaptive Transformation Agent, A$^\text{T}$A）。具体来说： 1. 我们设计了一个PosAgent模块，该模块根据给定的功能特性自适应地预测一个合适的位移量来实现位置可变性。 2. 我们还设计了逆向位移变换（Reverse Displacement Transform, RDT）模块。这个模块采用了多个PosAgent块的反向结构，能够基于语义信息将深层次特征图转化为浅层特征图。 3. 为了控制生成图像中对象的位置是自适应预测还是固定不变，我们在A$^\text{T}$A模型中添加了位置切换嵌入(Position Switch Embedding)。 通过广泛的对比实验验证了我们提出的A$^\text{T}$A方法的有效性。我们的方法不仅在处理位置可变修复时表现出卓越的填充能力，而且还在确保对象位置固定不变的情况下保持良好的性能。
URL
https://arxiv.org/abs/2504.01603
PDF
-
Instance Migration Diffusion for Nuclear Instance Segmentation in Pathology
2025-04-02 10:29:31Lirui Qi, Hongliang He, Tong Wang, Siwei Feng, Guohong FuAbstract
Nuclear instance segmentation plays a vital role in disease diagnosis within digital pathology. However, limited labeled data in pathological images restricts the overall performance of nuclear instance segmentation. To tackle this challenge, we propose a novel data augmentation framework Instance Migration Diffusion Model (IM-Diffusion), IM-Diffusion designed to generate more varied pathological images by constructing diverse nuclear layouts and internuclear spatial relationships. In detail, we introduce a Nuclear Migration Module (NMM) which constructs diverse nuclear layouts by simulating the process of nuclear migration. Building on this, we further present an Internuclear-regions Inpainting Module (IIM) to generate diverse internuclear spatial relationships by structure-aware inpainting. On the basis of the above, IM-Diffusion generates more diverse pathological images with different layouts and internuclear spatial relationships, thereby facilitating downstream tasks. Evaluation on the CoNSeP and GLySAC datasets demonstrate that the images generated by IM-Diffusion effectively enhance overall instance segmentation performance. Code will be made public later.
Abstract (translated)
核实例分割在数字病理学中的疾病诊断中扮演着至关重要的角色。然而，由于病理图像中的标注数据有限，这限制了核实例分割的整体性能。为了解决这一挑战，我们提出了一种新颖的数据增强框架——实例迁移扩散模型（IM-Diffusion）。IM-Diffusion 旨在通过构建多样化的核布局和细胞间空间关系来生成更多样化的病理图像。 具体来说，我们引入了一个名为核迁移模块（NMM）的组件，该模块通过模拟核迁移的过程来构造多样的核布局。在此基础上，进一步提出了一种细胞间区域修复模块（IIM），用于通过结构感知的修补技术生成多样化的细胞间空间关系。基于上述设计，IM-Diffusion 能够生成具有不同布局和细胞间空间关系的多样化病理图像，从而为下游任务提供支持。 在 CoNSeP 和 GLySAC 数据集上的评估表明，由 IM-Diffusion 生成的图像有效提升了整体实例分割性能。代码将在后续公开发布。
URL
https://arxiv.org/abs/2504.01577
PDF
-
Pro-DG: Procedural Diffusion Guidance for Architectural Facade Generation
2025-04-02 10:16:19Aleksander Plocharski, Jan Swidzinski, Przemyslaw MusialskiAbstract
We present Pro-DG, a framework for procedurally controllable photo-realistic facade generation that combines a procedural shape grammar with diffusion-based image synthesis. Starting from a single input image, we reconstruct its facade layout using grammar rules, then edit that structure through user-defined transformations. As facades are inherently multi-hierarchical structures, we introduce hierarchical matching procedure that aligns facade structures at different levels which is used to introduce control maps to guide a generative diffusion pipeline. This approach retains local appearance fidelity while accommodating large-scale edits such as floor duplication or window rearrangement. We provide a thorough evaluation, comparing Pro-DG against inpainting-based baselines and synthetic ground truths. Our user study and quantitative measurements indicate improved preservation of architectural identity and higher edit accuracy. Our novel method is the first to integrate neuro-symbolically derived shape-grammars for modeling with modern generative model and highlights the broader potential of such approaches for precise and controllable image manipulation.
Abstract (translated)
我们介绍了一种名为Pro-DG的框架，该框架用于生成程序可控的逼真建筑立面，它结合了过程形状语法和基于扩散的图像合成技术。从单张输入图片开始，我们可以使用语法规则重建其立面布局，并通过用户定义的转换编辑这一结构。鉴于立面本质上是多层次结构，我们引入了一种层次匹配过程，该过程对不同层级上的立面结构进行对齐，以用于引导生成扩散管道的控制图。这种方法在保持局部外观真实性的前提下，还能容纳大规模编辑，例如楼层复制或窗户重新排列。 我们进行了全面的评估，并将Pro-DG与基于图像修复的方法和合成的真实数据进行了比较。我们的用户研究和定量测量表明，在保留建筑身份的同时提高了编辑准确性。我们的新方法是首个集成神经符号派生形状语法用于现代生成模型建模的技术，突显了此类方法在精确且可控的图像操作方面的广泛应用潜力。
URL
https://arxiv.org/abs/2504.01571
PDF
-
TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting
2025-04-01 17:33:29Liangbin Xie, Daniil Pakhomov, Zhonghao Wang, Zongze Wu, Ziyan Chen, Yuqian Zhou, Haitian Zheng, Zhifei Zhang, Zhe Lin, Jiantao Zhou, Chao DongAbstract
This paper introduces TurboFill, a fast image inpainting model that enhances a few-step text-to-image diffusion model with an inpainting adapter for high-quality and efficient inpainting. While standard diffusion models generate high-quality results, they incur high computational costs. We overcome this by training an inpainting adapter on a few-step distilled text-to-image model, DMD2, using a novel 3-step adversarial training scheme to ensure realistic, structurally consistent, and visually harmonious inpainted regions. To evaluate TurboFill, we propose two benchmarks: DilationBench, which tests performance across mask sizes, and HumanBench, based on human feedback for complex prompts. Experiments show that TurboFill outperforms both multi-step BrushNet and few-step inpainting methods, setting a new benchmark for high-performance inpainting tasks. Our project page: this https URL
Abstract (translated)
这篇论文介绍了TurboFill，这是一种快速的图像修复模型，通过增强几个步骤的文本到图像扩散模型，并加入一个图像修复适配器来实现高质量和高效的图像修补。虽然标准的扩散模型可以生成高质量的结果，但它们会带来高昂的计算成本。我们通过在几步蒸馏后的文本到图像模型DMD2上训练一个图像修复适配器来克服这一问题，并使用一种新颖的三步骤对抗性训练方案，以确保修复区域的真实感、结构一致性和视觉和谐。 为了评估TurboFill的效果，我们提出了两个基准测试：DilationBench用于检测不同遮罩大小下的性能表现；HumanBench则是基于复杂指令的人类反馈。实验结果表明，TurboFill在性能上超过了多步骤的BrushNet和少步骤图像修复方法，在高质量图像修补任务中设立了新的标准。 我们的项目页面链接：[this https URL](https://example.com) （注意：实际使用时请替换为正确的网址）
URL
https://arxiv.org/abs/2504.00996
PDF
-
Self-Supervised Pretraining for Aerial Road Extraction
2025-03-31 17:14:08Rupert Polley, Sai Vignesh Abishek Deenadayalan, J. Marius Z\"ollnerAbstract
Deep neural networks for aerial image segmentation require large amounts of labeled data, but high-quality aerial datasets with precise annotations are scarce and costly to produce. To address this limitation, we propose a self-supervised pretraining method that improves segmentation performance while reducing reliance on labeled data. Our approach uses inpainting-based pretraining, where the model learns to reconstruct missing regions in aerial images, capturing their inherent structure before being fine-tuned for road extraction. This method improves generalization, enhances robustness to domain shifts, and is invariant to model architecture and dataset choice. Experiments show that our pretraining significantly boosts segmentation accuracy, especially in low-data regimes, making it a scalable solution for aerial image analysis.
Abstract (translated)
针对高空图像分割的深度神经网络需要大量的标注数据，但高质量的带有精确注释的高空图像数据集稀缺且成本高昂。为了解决这一限制，我们提出了一种自监督预训练方法，该方法可以在减少对标注数据依赖的同时提升分割性能。我们的方法采用基于图像修复（inpainting）的预训练方式，其中模型学习如何重建高空图像中的缺失区域，在捕捉其内在结构后进行道路提取的微调。这种方法提升了泛化能力，增强了领域偏移时的鲁棒性，并且对模型架构和数据集的选择具有不变性。实验结果表明，我们的预训练方法显著提高了分割精度，尤其是在低标注数据的情况下表现尤为出色，使其成为高空图像分析中的可扩展解决方案。
URL
https://arxiv.org/abs/2503.24326
PDF
-
Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering
2025-03-29 02:16:15Yugen Sato, Tomohiro TakagiAbstract
Recent advances in large language models (LLMs) have led to the development of multimodal LLMs (MLLMs) in the fields of natural language processing (NLP) and computer vision. Although these models allow for integrated visual and language understanding, they present challenges such as opaque internal processing and the generation of hallucinations and misinformation. Therefore, there is a need for a method to clarify the location of knowledge in MLLMs. In this study, we propose a method to identify neurons associated with specific knowledge using MiniGPT-4, a Transformer-based MLLM. Specifically, we extract knowledge neurons through two stages: activation differences filtering using inpainting and gradient-based filtering using GradCAM. Experiments on the image caption generation task using the MS COCO 2017 dataset, BLEU, ROUGE, and BERTScore quantitative evaluation, and qualitative evaluation using an activation heatmap showed that our method is able to locate knowledge with higher accuracy than existing methods. This study contributes to the visualization and explainability of knowledge in MLLMs and shows the potential for future knowledge editing and control.
Abstract (translated)
近期，在自然语言处理（NLP）和计算机视觉领域，大型语言模型（LLM）的进展促使了多模态大型语言模型（MLLMs）的发展。尽管这些模型能够实现集成化的视觉与语言理解，但它们也带来了诸如内部处理不透明、生成幻觉及错误信息等挑战。因此，需要一种方法来明确多模态大语言模型中知识的位置。本研究提出了一种利用基于Transformer架构的MiniGPT-4模型识别特定知识相关神经元的方法。具体而言，我们通过两个阶段提取知识神经元：第一阶段使用修复（inpainting）技术进行激活差异过滤；第二阶段采用GradCAM方法进行梯度基过滤。 在图像描述生成任务中，我们利用MS COCO 2017数据集进行了实验，并使用BLEU、ROUGE和BERTScore等定量评估指标以及通过激活热图的定性评估来验证我们的方法。结果显示，相较于现有技术，本研究的方法能够以更高的准确性定位知识位置。 这项研究为多模态大语言模型的知识可视化及可解释性做出了贡献，并展示了未来在知识编辑与控制方面的潜在应用。
URL
https://arxiv.org/abs/2503.22941
PDF
-
Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
2025-03-28 15:03:41Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal, Felix HeideAbstract
We introduce Scenario Dreamer, a fully data-driven generative simulator for autonomous vehicle planning that generates both the initial traffic scene - comprising a lane graph and agent bounding boxes - and closed-loop agent behaviours. Existing methods for generating driving simulation environments encode the initial traffic scene as a rasterized image and, as such, require parameter-heavy networks that perform unnecessary computation due to many empty pixels in the rasterized scene. Moreover, we find that existing methods that employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer instead employs a novel vectorized latent diffusion model for initial scene generation that directly operates on the vectorized scene elements and an autoregressive Transformer for data-driven agent behaviour simulation. Scenario Dreamer additionally supports scene extrapolation via diffusion inpainting, enabling the generation of unbounded simulation environments. Extensive experiments show that Scenario Dreamer outperforms existing generative simulators in realism and efficiency: the vectorized scene-generation base model achieves superior generation quality with around 2x fewer parameters, 6x lower generation latency, and 10x fewer GPU training hours compared to the strongest baseline. We confirm its practical utility by showing that reinforcement learning planning agents are more challenged in Scenario Dreamer environments than traditional non-generative simulation environments, especially on long and adversarial driving environments.
Abstract (translated)
我们介绍了Scenario Dreamer，这是一种完全基于数据驱动的自动驾驶车辆规划生成模拟器。它可以生成初始交通场景（包括车道图和代理边框）以及闭环代理行为。现有的用于生成驾驶仿真环境的方法将初始交通场景编码为栅格化图像，并因此需要参数繁重的网络来执行不必要的计算，因为栅格化场景中有许多空白像素。此外，我们发现使用基于规则的代理行为方法缺乏多样性和现实性。 与现有方法不同，Scenario Dreamer采用了一种新颖的向量化潜在扩散模型进行初始场景生成，该模型直接操作向量化的场景元素，并且使用自回归Transformer来进行数据驱动的代理行为模拟。此外，Scenario Dreamer通过扩散填补技术支持场景扩展，从而能够生成不受限的仿真环境。 大量的实验表明，与现有的生成式模拟器相比，Scenario Dreamer在真实性和效率方面表现更佳：向量化场景生成基础模型实现了更高的生成质量，参数数量减少了大约2倍，生成延迟降低了6倍，并且GPU训练时间也减少了10倍。我们通过展示强化学习规划代理在Scenario Dreamer环境中面临更大挑战的事实来确认其实际效用，特别是在长时间和对抗性驾驶环境中的表现优于传统的非生成式仿真环境。
URL
https://arxiv.org/abs/2503.22496
PDF
-
Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting
2025-03-27 17:59:33Anand Bhattad, Konpat Preechakul, Alexei A. EfrosAbstract
This paper proposes a novel scene understanding task called Visual Jenga. Drawing inspiration from the game Jenga, the proposed task involves progressively removing objects from a single image until only the background remains. Just as Jenga players must understand structural dependencies to maintain tower stability, our task reveals the intrinsic relationships between scene elements by systematically exploring which objects can be removed while preserving scene coherence in both physical and geometric sense. As a starting point for tackling the Visual Jenga task, we propose a simple, data-driven, training-free approach that is surprisingly effective on a range of real-world images. The principle behind our approach is to utilize the asymmetry in the pairwise relationships between objects within a scene and employ a large inpainting model to generate a set of counterfactuals to quantify the asymmetry.
Abstract (translated)
这篇论文提出了一项新的场景理解任务，称为视觉积木（Visual Jenga）。这项任务借鉴了游戏Jenga中的原理，即逐步从单张图片中移除对象直到只剩下背景。就像Jenga玩家必须理解结构依赖关系以保持塔的稳定性一样，我们的任务通过系统性地探索在保持场景物理和几何连贯性的前提下可以移除哪些物体，来揭示场景元素之间的内在联系。 为了开始解决视觉积木的任务，我们提出了一种简单、数据驱动且无需训练的方法，在各种现实世界图片上表现出惊人的效果。该方法背后的原理是利用场景中对象之间成对关系的不对称性，并使用一个大型图像修复模型生成一组反事实情况来量化这种不对称性。
URL
https://arxiv.org/abs/2503.21770
PDF
-
MVFNet: Multipurpose Video Forensics Network using Multiple Forms of Forensic Evidence
2025-03-26 21:11:06Tai D. Nguyen, Matthew C. StammAbstract
While videos can be falsified in many different ways, most existing forensic networks are specialized to detect only a single manipulation type (e.g. deepfake, inpainting). This poses a significant issue as the manipulation used to falsify a video is not known a priori. To address this problem, we propose MVFNet - a multipurpose video forensics network capable of detecting multiple types of manipulations including inpainting, deepfakes, splicing, and editing. Our network does this by extracting and jointly analyzing a broad set of forensic feature modalities that capture both spatial and temporal anomalies in falsified videos. To reliably detect and localize fake content of all shapes and sizes, our network employs a novel Multi-Scale Hierarchical Transformer module to identify forensic inconsistencies across multiple spatial scales. Experimental results show that our network obtains state-of-the-art performance in general scenarios where multiple different manipulations are possible, and rivals specialized detectors in targeted scenarios.
Abstract (translated)
虽然视频可以通过多种方式被伪造，但现有的大多数取证网络仅专注于检测单一类型的操纵（例如深度伪造、修补）。这在实践中带来了重大问题，因为在验证视频的真实性时通常不知道是哪种方法进行了篡改。为了解决这个问题，我们提出了MVFNet——一种多功能的视频取证网络，能够检测包括修补、深度伪造、拼接和编辑在内的多种类型的操作。我们的网络通过提取并联合分析一套广泛的取证特征模态来实现这一目标，这些特征可以捕捉到被篡改视频中的空间和时间异常。 为了可靠地检测并定位所有形状和大小的虚假内容，我们的网络采用了一种创新的多尺度分层变换器模块，在多个空间尺度上识别取证上的不一致性。实验结果表明，我们的网络在多种不同类型的操纵都可能发生的一般场景中达到了最先进的性能，并且在针对特定类型伪造手段的场景中也能与专用检测器相媲美。
URL
https://arxiv.org/abs/2503.20991
PDF
-
Unified Multimodal Discrete Diffusion
2025-03-26 17:59:51Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, Katerina FragkiadakiAbstract
Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches, which process tokens sequentially from left to right, or top to bottom. These models jointly handle images, text, video, and audio for various tasks such as image captioning, question answering, and image generation. In this work, we explore discrete diffusion models as a unified generative formulation in the joint text and image domain, building upon their recent success in text generation. Discrete diffusion models offer several advantages over AR models, including improved control over quality versus diversity of generated samples, the ability to perform joint multimodal inpainting (across both text and image domains), and greater controllability in generation through guidance. Leveraging these benefits, we present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images for a variety of downstream tasks. We compare UniDisc to multimodal AR models, performing a scaling analysis and demonstrating that UniDisc outperforms them in terms of both performance and inference-time compute, enhanced controllability, editability, inpainting, and flexible trade-off between inference time and generation quality. Code and additional visualizations are available at this https URL.
Abstract (translated)
多模态生成模型能够理解和生成跨多种模式（如图像、文本、视频和音频）的数据，在任务如图像描述、问答和图像生成中发挥作用。这些模型主要采用自回归（AR）方法，按从左到右或从上到下的顺序处理标记。在这项工作中，我们探讨了离散扩散模型作为联合文本和图像领域的统一生成框架，并在此前成功应用于文本生成的基础上进行研究。离散扩散模型相比自回归模型具有若干优势，包括对生成样本质量和多样性的更好控制、能够在文本和图像域内执行联合多模态插补以及通过指导实现更可控的生成能力。 利用这些优点，我们提出了首个统一多模态离散扩散（UniDisc）模型，该模型能够同时理解和生成用于各种下游任务的文本和图像。我们将UniDisc与多模态自回归模型进行比较，并进行了规模分析，证明在性能、推理时间计算、增强可控性、可编辑性和插补以及在推理时间和生成质量之间的灵活权衡方面，UniDisc优于它们。 代码及相关可视化内容可在以下链接获得：[请在此处插入实际URL]。
URL
https://arxiv.org/abs/2503.20853
PDF
-
SparseGS-W: Sparse-View 3D Gaussian Splatting in the Wild with Generative Priors
2025-03-25 08:40:40Yiqing Li, Xuan Wang, Jiawei Wu, Yikun Ma, Zhi JinAbstract
Synthesizing novel views of large-scale scenes from unconstrained in-the-wild images is an important but challenging task in computer vision. Existing methods, which optimize per-image appearance and transient occlusion through implicit neural networks from dense training views (approximately 1000 images), struggle to perform effectively under sparse input conditions, resulting in noticeable artifacts. To this end, we propose SparseGS-W, a novel framework based on 3D Gaussian Splatting that enables the reconstruction of complex outdoor scenes and handles occlusions and appearance changes with as few as five training images. We leverage geometric priors and constrained diffusion priors to compensate for the lack of multi-view information from extremely sparse input. Specifically, we propose a plug-and-play Constrained Novel-View Enhancement module to iteratively improve the quality of rendered novel views during the Gaussian optimization process. Furthermore, we propose an Occlusion Handling module, which flexibly removes occlusions utilizing the inherent high-quality inpainting capability of constrained diffusion priors. Both modules are capable of extracting appearance features from any user-provided reference image, enabling flexible modeling of illumination-consistent scenes. Extensive experiments on the PhotoTourism and Tanks and Temples datasets demonstrate that SparseGS-W achieves state-of-the-art performance not only in full-reference metrics, but also in commonly used non-reference metrics such as FID, ClipIQA, and MUSIQ.
Abstract (translated)
合成大规模场景的新视角是从不受约束的野外图像中一个重要的但具有挑战性的任务，在计算机视觉领域。现有的方法通过隐式神经网络从密集训练视图（约1000张图片）来优化每个图像的外观和瞬时遮挡，但在稀疏输入条件下难以有效运行，导致明显的伪影出现。 为此，我们提出了SparseGS-W框架，这是一个基于3D高斯点阵的新方法，它能够仅使用五张训练图像就重建复杂的室外场景，并处理遮挡和外观变化。我们利用了几何先验知识和受限制的扩散先验来补偿极度稀疏输入中多视角信息的不足。 具体来说，我们提出了一种即插即用的Constrained Novel-View Enhancement模块，在高斯优化过程中迭代提高渲染的新视图质量。此外，我们还提出了一个Occlusion Handling模块，它灵活地利用受限制扩散先验固有的高质量填图能力来去除遮挡。 两个模块都能够从任何用户提供的参考图像中提取外观特征，使光照一致的场景建模更加灵活。在PhotoTourism和Tanks and Temples数据集上的大量实验表明，SparseGS-W不仅在全参照度量标准上取得了最先进的性能，在常用的无参照度量标准（如FID、ClipIQA和MUSIQ）上也表现出色。
URL
https://arxiv.org/abs/2503.19452
PDF
-
Tuning-Free Amodal Segmentation via the Occlusion-Free Bias of Inpainting Models
2025-03-24 17:59:56Jae Joong Lee, Bedrich Benes, Raymond A. YehAbstract
Amodal segmentation aims to predict segmentation masks for both the visible and occluded regions of an object. Most existing works formulate this as a supervised learning problem, requiring manually annotated amodal masks or synthetic training data. Consequently, their performance depends on the quality of the datasets, which often lack diversity and scale. This work introduces a tuning-free approach that repurposes pretrained diffusion-based inpainting models for amodal segmentation. Our approach is motivated by the "occlusion-free bias" of inpainting models, i.e., the inpainted objects tend to be complete objects without occlusions. Specifically, we reconstruct the occluded regions of an object via inpainting and then apply segmentation, all without additional training or fine-tuning. Experiments on five datasets demonstrate the generalizability and robustness of our approach. On average, our approach achieves 5.3% more accurate masks over the state-of-the-art.
Abstract (translated)
非模态分割的目标是为物体的可见和被遮挡区域预测分割掩码。大多数现有的方法将这个问题定义为一个监督学习问题，需要手动标注非模态掩码或合成训练数据。因此，它们的表现依赖于数据集的质量，而这些数据集往往缺乏多样性和规模。这项工作提出了一种无需调优的方法，该方法重新利用了预先训练的基于扩散的修复模型来进行非模态分割。我们的方法受到“无遮挡偏置”（即修复的对象倾向于是完整的、没有被遮挡的对象）这一特性启发。具体来说，我们通过修复来重建物体的遮挡区域，并在此基础上进行分割，整个过程无需额外的训练或微调。在五个数据集上的实验表明了我们的方法具有良好的泛化能力和鲁棒性。平均而言，与现有最佳方法相比，我们的方法实现了5.3%更准确的掩码效果。
URL
https://arxiv.org/abs/2503.18947
PDF
-
OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video Diffusion Models
2025-03-23 11:26:48Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-AriAbstract
Omnimatte aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. We accomplish this by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. We then show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.
Abstract (translated)
Omnimatte旨在将给定的视频分解为语义上有意义的图层，包括背景和各个对象及其相关效果（如阴影和反射）。现有的方法通常需要大量的训练或昂贵的自监督优化。在本文中，我们提出了一种无需训练的方法——OmnimatteZero，该方法利用现成的预训练视频扩散模型进行全场景材质分解。它可以移除视频中的物体，提取单独的对象图层及其相关效果，并将这些对象合并到新的视频上。通过将零样本图像修复技术适应于视频对象去除任务（这是它们在没有修改的情况下无法有效处理的任务），我们实现了这一目标。此外，我们展示了自注意力映射可以捕获有关对象及其足迹的信息，并利用这些信息来修复对象的效果，从而留下干净的背景。通过简单的潜在算术运算，还可以将物体图层隔离并无缝地与新的视频图层重新组合以生成新视频。评估结果表明，OmnimatteZero不仅在背景重建方面表现出色，而且还创下了最快全场景材质分解方法的新纪录，在实现近乎实时性能的同时保证了最小的帧处理时间。
URL
https://arxiv.org/abs/2503.18033
PDF
-
Taste More, Taste Better: Diverse Data and Strong Model Boost Semi-Supervised Crowd Counting
2025-03-23 08:38:01Maochen Yang, Zekun Li, Jian Zhang, Lei Qi, Yinghuan ShiAbstract
Semi-supervised crowd counting is crucial for addressing the high annotation costs of densely populated scenes. Although several methods based on pseudo-labeling have been proposed, it remains challenging to effectively and accurately utilize unlabeled data. In this paper, we propose a novel framework called Taste More Taste Better (TMTB), which emphasizes both data and model aspects. Firstly, we explore a data augmentation technique well-suited for the crowd counting task. By inpainting the background regions, this technique can effectively enhance data diversity while preserving the fidelity of the entire scenes. Secondly, we introduce the Visual State Space Model as backbone to capture the global context information from crowd scenes, which is crucial for extremely crowded, low-light, and adverse weather scenarios. In addition to the traditional regression head for exact prediction, we employ an Anti-Noise classification head to provide less exact but more accurate supervision, since the regression head is sensitive to noise in manual annotations. We conduct extensive experiments on four benchmark datasets and show that our method outperforms state-of-the-art methods by a large margin. Code is publicly available on this https URL.
Abstract (translated)
半监督人群计数对于解决密集场景中的高标注成本问题至关重要。尽管已经提出了几种基于伪标签的方法，但如何有效地利用未标记数据仍是一个挑战。在本文中，我们提出了一种新颖的框架——“尝得更多，变得更好”（Taste More Taste Better, TMTB），该框架同时强调了数据和模型方面的重要性。 首先，我们探索了一种适合人群计数任务的数据增强技术。通过修复背景区域，这种技术可以有效增加数据多样性的同时保持整个场景的真实感。 其次，我们引入视觉状态空间模型作为骨干网络来捕捉从人群中获取的全局上下文信息，这对于极其拥挤、光线不足和恶劣天气场景尤为重要。除了传统的回归头用于精确预测之外，我们还采用了一种抗噪分类头提供不太精确但更准确的监督，因为回归头对人工标注中的噪声非常敏感。 我们在四个基准数据集上进行了广泛的实验，并显示我们的方法在很大程度上优于最先进的方法。代码可在[此处](https://this https URL)公开获取（请注意，实际链接应替换为有效的公共访问地址）。
URL
https://arxiv.org/abs/2503.17984
PDF
-
Depth-Aided Color Image Inpainting in Quaternion Domain
2025-03-21 03:18:41Shunki Tatsumi, Ryo Hayakawa, Youji IiguniAbstract
In this paper, we propose a depth-aided color image inpainting method in the quaternion domain, called depth-aided low-rank quaternion matrix completion (D-LRQMC). In conventional quaternion-based inpainting techniques, the color image is expressed as a quaternion matrix by using the three imaginary parts as the color channels, whereas the real part is set to zero and has no information. Our approach incorporates depth information as the real part of the quaternion representations, leveraging the correlation between color and depth to improve the result of inpainting. In the proposed method, we first restore the observed image with the conventional LRQMC and estimate the depth of the restored result. We then incorporate the estimated depth into the real part of the observed image and perform LRQMC again. Simulation results demonstrate that the proposed D-LRQMC can improve restoration accuracy and visual quality for various images compared to the conventional LRQMC. These results suggest the effectiveness of the depth information for color image processing in quaternion domain.
Abstract (translated)
在这篇论文中，我们提出了一种基于四元数域的深度辅助颜色图像修复方法，称为深度辅助低秩四元数矩阵补全（D-LRQMC）。在传统的基于四元数的修复技术中，彩色图像是通过将三个虚部部分用作色彩通道来表示为一个四元数矩阵，并且实部被设置为零且不携带信息。我们的方法则利用深度信息作为四元数表示中的实部，借助颜色和深度之间的相关性来提高图像修补的效果。 在提出的方法中，我们首先使用传统的LRQMC（低秩四元数矩阵补全）技术恢复观察到的图像，并估计该修复结果的深度。然后将估算出的深度信息整合进原始图像的实部部分，并再次执行LRQMC操作。模拟结果显示，与传统方法相比，我们的D-LRQMC方法能够提升各种图像的复原准确性和视觉质量。这些结果表明，在四元数域中使用深度信息对于彩色图象处理的有效性。
URL
https://arxiv.org/abs/2503.16818
PDF
-
A Recipe for Generating 3D Worlds From a Single Image
2025-03-20 18:06:12Katja Schwarz, Denys Rozumnyi, Samuel Rota Bul\`o, Lorenzo Porzi, Peter KontschiederAbstract
We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models. This approach requires minimal training and uses existing generative models. Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator. We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning. Tested on both synthetic and real images, our method produces high-quality 3D environments suitable for VR display. By explicitly modeling the 3D structure of the generated environment from the start, our approach consistently outperforms state-of-the-art, video synthesis-based methods along multiple quantitative image quality metrics. Project Page: this https URL
Abstract (translated)
我们介绍了一种从单张图像生成沉浸式三维世界的食谱，将此任务视为2D图像修复模型的在上下文学习问题。这种方法需要最少的训练，并利用现有的生成模型。我们的流程包括两个步骤：使用预训练的扩散模型生成连贯的全景图，然后通过度量深度估计器将其提升到3D空间中。接着，我们通过对渲染的点云进行条件设置来填充未观察到的区域，这只需要少量微调即可完成。我们在合成图像和真实图像上进行了测试，我们的方法能够产生高质量的三维环境，适合用于虚拟现实显示。 通过从一开始就明确模拟生成环境中三维结构的特点，我们的方法在多个定量图像质量指标上始终优于基于视频合成的方法。项目页面：[此链接](https://this https URL)
URL
https://arxiv.org/abs/2503.16611
PDF
-
NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes
2025-03-20 17:37:43Han-Hung Lee, Qinghong Han, Angel X. ChangAbstract
In this paper, we explore the task of generating expansive outdoor scenes, ranging from castles to high-rises. Unlike indoor scene generation, which has been a primary focus of prior work, outdoor scene generation presents unique challenges, including wide variations in scene heights and the need for a method capable of rapidly producing large landscapes. To address this, we propose an efficient approach that encodes scene chunks as uniform vector sets, offering better compression and performance than the spatially structured latents used in prior methods. Furthermore, we train an explicit outpainting model for unbounded generation, which improves coherence compared to prior resampling-based inpainting schemes while also speeding up generation by eliminating extra diffusion steps. To facilitate this task, we curate NuiScene43, a small but high-quality set of scenes, preprocessed for joint training. Notably, when trained on scenes of varying styles, our model can blend different environments, such as rural houses and city skyscrapers, within the same scene, highlighting the potential of our curation process to leverage heterogeneous scenes for joint training.
Abstract (translated)
在这篇论文中，我们探讨了生成广阔户外场景的任务，这些场景从城堡到高层建筑不等。与先前研究主要关注的室内场景生成不同，户外场景生成面临着独特的挑战，包括场景高度变化大以及需要一种能够快速生产大规模景观的方法。为了解决这些问题，我们提出了一种高效的方法，该方法将场景块编码成均匀向量集合，相比之前方法使用的空间结构化潜在变量，这种方法提供了更好的压缩和性能表现。 此外，我们还训练了一个显式的“outpainting”模型用于无边界生成，与先前基于重采样的填充方案相比，这种方案能够提高连贯性，并通过消除额外的扩散步骤加快了生成速度。为了便于这项任务，我们整理了NuiScene43，这是一个虽小但质量高的一组场景集，经过预处理以进行联合训练。 值得注意的是，在不同风格场景上训练时，我们的模型可以在同一场景中融合不同的环境，如乡村房屋和城市摩天大楼，这突显出我们的数据整理过程能够利用异构场景进行联合训练的潜力。
URL
https://arxiv.org/abs/2503.16375
PDF
