Inpainting
-
Controllable 3D Placement of Objects with Scene-Aware Diffusion Models
2025-06-26 16:31:39Mohamed Omran, Dimitris Kalatzis, Jens Petersen, Amirhossein Habibian, Auke WiggersAbstract
Image editing approaches have become more powerful and flexible with the advent of powerful text-conditioned generative models. However, placing objects in an environment with a precise location and orientation still remains a challenge, as this typically requires carefully crafted inpainting masks or prompts. In this work, we show that a carefully designed visual map, combined with coarse object masks, is sufficient for high quality object placement. We design a conditioning signal that resolves ambiguities, while being flexible enough to allow for changing of shapes or object orientations. By building on an inpainting model, we leave the background intact by design, in contrast to methods that model objects and background jointly. We demonstrate the effectiveness of our method in the automotive setting, where we compare different conditioning signals in novel object placement tasks. These tasks are designed to measure edit quality not only in terms of appearance, but also in terms of pose and location accuracy, including cases that require non-trivial shape changes. Lastly, we show that fine location control can be combined with appearance control to place existing objects in precise locations in a scene.
Abstract (translated)
随着强大的文本条件生成模型的出现，图像编辑方法已经变得更加强大和灵活。然而，在精确的位置和方向上将物体放置在环境中仍然是一项挑战，这通常需要精心设计的修补掩模或提示词。在这项工作中，我们展示了通过结合精细设计的视觉地图与粗略的对象遮罩，就足以实现高质量的物体放置。我们设计了一种条件信号，这种信号能够解决歧义问题，并且足够灵活以允许形状变化或对象方向的变化。通过建立在修补模型的基础上，我们的方法有意地保留背景不变，这与其他同时建模物体和背景的方法形成了对比。我们在汽车设置中演示了我们方法的有效性，在该设置中，我们将不同条件信号应用于新颖的物体放置任务中进行比较。这些任务旨在从外观、姿态以及位置精度等方面评估编辑质量，包括需要进行非平凡形状变化的情况。最后，我们展示了精细的位置控制可以与外观控制相结合，以将现有的对象精确地放置在一个场景中的特定位置。
URL
https://arxiv.org/abs/2506.21446
PDF
-
Video Virtual Try-on with Conditional Diffusion Transformer Inpainter
2025-06-26 13:56:27Cheng Zou, Senlin Cheng, Bolei Xu, Dandan Zheng, Xiaobo Li, Jingdong Chen, Ming YangAbstract
Video virtual try-on aims to naturally fit a garment to a target person in consecutive video frames. It is a challenging task, on the one hand, the output video should be in good spatial-temporal consistency, on the other hand, the details of the given garment need to be preserved well in all the frames. Naively using image-based try-on methods frame by frame can get poor results due to severe inconsistency. Recent diffusion-based video try-on methods, though very few, happen to coincide with a similar solution: inserting temporal attention into image-based try-on model to adapt it for video try-on task, which have shown improvements but there still exist inconsistency problems. In this paper, we propose ViTI (Video Try-on Inpainter), formulate and implement video virtual try-on as a conditional video inpainting task, which is different from previous methods. In this way, we start with a video generation problem instead of an image-based try-on problem, which from the beginning has a better spatial-temporal consistency. Specifically, at first we build a video inpainting framework based on Diffusion Transformer with full 3D spatial-temporal attention, and then we progressively adapt it for video garment inpainting, with a collection of masking strategies and multi-stage training. After these steps, the model can inpaint the masked garment area with appropriate garment pixels according to the prompt with good spatial-temporal consistency. Finally, as other try-on methods, garment condition is added to the model to make sure the inpainted garment appearance and details are as expected. Both quantitative and qualitative experimental results show that ViTI is superior to previous works.
Abstract (translated)
视频虚拟试穿的目标是将衣物自然地合身到目标人物在连续帧中的姿态上。这项任务具有挑战性，一方面输出的视频应具备良好的时空一致性；另一方面，给定衣物的所有细节都需要在每一帧中得到良好保留。简单地逐帧应用基于图像的试穿方法会导致严重的不一致问题。尽管最近有一些扩散模型（diffusion-based）的方法尝试解决这个问题，但它们通过将时间注意力机制插入到现有的基于图像的试穿模型来适应视频任务，虽然取得了一些进步，但仍存在一致性的问题。 在本文中，我们提出了ViTI（Video Try-on Inpainter），并将其定义为一个条件下的视频修复任务。这种方法不同于以往的方法，因为它从一开始就以视频生成问题而非单一图像为基础，从而更好地保证了时空的一致性。具体来说，我们首先建立了一个基于扩散变换器（Diffusion Transformer）的3D空间时间注意力框架，并逐步适应该框架用于衣物区域的修复工作，通过一系列掩码策略和多阶段训练来实现这一点。经过这些步骤后，模型可以在遮挡的衣物区域内根据提示进行填充，生成具有良好时空一致性的适当衣物像素。 最后，如同其他试穿方法一样，在这一过程中加入服装条件以确保修复后的服装外观及细节符合预期要求。无论是定量还是定性实验结果都显示ViTI优于先前的工作。
URL
https://arxiv.org/abs/2506.21270
PDF
-
DreamAnywhere: Object-Centric Panoramic 3D Scene Generation
2025-06-25 12:30:41Edoardo Alberto Dominici, Jozef Hladky, Floor Verhoeven, Lukas Radl, Thomas Deixelberger, Stefan Ainetter, Philipp Drescher, Stefan Hauswiesner, Arno Coomans, Giacomo Nazzaro, Konstantinos Vardis, Markus SteinbergerAbstract
Recent advances in text-to-3D scene generation have demonstrated significant potential to transform content creation across multiple industries. Although the research community has made impressive progress in addressing the challenges of this complex task, existing methods often generate environments that are only front-facing, lack visual fidelity, exhibit limited scene understanding, and are typically fine-tuned for either indoor or outdoor settings. In this work, we address these issues and propose DreamAnywhere, a modular system for the fast generation and prototyping of 3D scenes. Our system synthesizes a 360° panoramic image from text, decomposes it into background and objects, constructs a complete 3D representation through hybrid inpainting, and lifts object masks to detailed 3D objects that are placed in the virtual environment. DreamAnywhere supports immersive navigation and intuitive object-level editing, making it ideal for scene exploration, visual mock-ups, and rapid prototyping -- all with minimal manual modeling. These features make our system particularly suitable for low-budget movie production, enabling quick iteration on scene layout and visual tone without the overhead of traditional 3D workflows. Our modular pipeline is highly customizable as it allows components to be replaced independently. Compared to current state-of-the-art text and image-based 3D scene generation approaches, DreamAnywhere shows significant improvements in coherence in novel view synthesis and achieves competitive image quality, demonstrating its effectiveness across diverse and challenging scenarios. A comprehensive user study demonstrates a clear preference for our method over existing approaches, validating both its technical robustness and practical usefulness.
Abstract (translated)
近期在文本到3D场景生成领域的进展展示了其有潜力彻底改变多个行业的内容创作。尽管研究社区在这项复杂任务上取得了令人瞩目的进步，但现有方法通常只能生成面向前方的环境，并且缺乏视觉真实感、场景理解能力有限，而且通常是为室内或室外环境进行微调的。在本工作中，我们解决了这些问题并提出了DreamAnywhere系统——这是一个用于快速生成和原型设计3D场景的模块化系统。 我们的系统从文本中合成一个360°全景图像，将其分解成背景和物体，并通过混合填充技术构建完整的3D表示。此外，它还能够将对象掩模提升为详细且放置在虚拟环境中的3D对象。DreamAnywhere支持沉浸式导航和直观的对象级编辑功能，非常适合场景探索、视觉模拟以及快速原型设计——所有这些都可以用最少的手动建模完成。 该系统的这些特点特别适合低预算电影制作，因为它允许创作者迅速迭代场景布局和视觉风格，而无需传统3D工作流程带来的额外负担。我们的模块化管道高度可定制，因为它的组件可以独立替换。与目前最先进的基于文本和图像的3D场景生成方法相比，DreamAnywhere在新视角合成的一致性和图像质量方面显示出显著改进，证明了它在各种具有挑战性的场景中的有效性。 一项全面的用户研究显示，相对于现有方法，用户更倾向于使用我们的方法，这验证了其技术稳定性和实际应用价值。
URL
https://arxiv.org/abs/2506.20367
PDF
-
PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Scenes
2025-06-23 20:47:18Christina Ourania Tze, Daniel Dauner, Yiyi Liao, Dzmitry Tsishkou, Andreas GeigerAbstract
Large-scale 3D semantic scene generation has predominantly relied on voxel-based representations, which are memory-intensive, bound by fixed resolutions, and challenging to edit. In contrast, primitives represent semantic entities using compact, coarse 3D structures that are easy to manipulate and compose, making them an ideal representation for this task. In this paper, we introduce PrITTI, a latent diffusion-based framework that leverages primitives as the main foundational elements for generating compositional, controllable, and editable 3D semantic scene layouts. Our method adopts a hybrid representation, modeling ground surfaces in a rasterized format while encoding objects as vectorized 3D primitives. This decomposition is also reflected in a structured latent representation that enables flexible scene manipulation of ground and object components. To overcome the orientation ambiguities in conventional encoding methods, we introduce a stable Cholesky-based parameterization that jointly encodes object size and orientation. Experiments on the KITTI-360 dataset show that PrITTI outperforms a voxel-based baseline in generation quality, while reducing memory requirements by up to $3\times$. In addition, PrITTI enables direct instance-level manipulation of objects in the scene and supports a range of downstream applications, including scene inpainting, outpainting, and photo-realistic street-view synthesis.
Abstract (translated)
大规模的三维语义场景生成主要依赖于基于体素（voxel）的表示方法，这种方法内存消耗大、分辨率固定且难以编辑。相比之下，原语（primitives）使用紧凑而粗糙的3D结构来表示语义实体，易于操作和组合，成为这一任务的理想选择。在本文中，我们介绍了一种名为PrITTI的新框架，该框架采用基于潜在扩散的方法，并以原语作为生成组成化、可控性和可编辑性三维语义场景布局的主要基础元素。我们的方法采用了混合表示方式，在栅格格式下建模地面表面的同时将物体编码为向量化的3D原语。这种分解也反映在一种结构化的潜在表示中，该表示能够灵活地操作场景中的地面和对象组件。 为了克服传统编码方法中存在的方位模糊性问题，我们引入了一种基于稳定的乔累斯基（Cholesky）参数化的方法，这种方法同时编码物体的大小和方向。实验结果表明，在KITTI-360数据集上的生成质量方面，PrITTI优于基于体素的基础线模型，并且将内存需求减少了最多三倍。此外，PrITTI支持场景中对象实例级别的直接操作，并适用于包括场景修复、扩展以及合成照片级真实街道视图在内的多种下游应用。
URL
https://arxiv.org/abs/2506.19117
PDF
-
Let Your Video Listen to Your Music!
2025-06-23 17:52:16Xinyu Zhang, Dong Gong, Zicheng Duan, Anton van den Hengel, Lingqiao LiuAbstract
Aligning the rhythm of visual motion in a video with a given music track is a practical need in multimedia production, yet remains an underexplored task in autonomous video editing. Effective alignment between motion and musical beats enhances viewer engagement and visual appeal, particularly in music videos, promotional content, and cinematic editing. Existing methods typically depend on labor-intensive manual cutting, speed adjustments, or heuristic-based editing techniques to achieve synchronization. While some generative models handle joint video and music generation, they often entangle the two modalities, limiting flexibility in aligning video to music beats while preserving the full visual content. In this paper, we propose a novel and efficient framework, termed MVAA (Music-Video Auto-Alignment), that automatically edits video to align with the rhythm of a given music track while preserving the original visual content. To enhance flexibility, we modularize the task into a two-step process in our MVAA: aligning motion keyframes with audio beats, followed by rhythm-aware video inpainting. Specifically, we first insert keyframes at timestamps aligned with musical beats, then use a frame-conditioned diffusion model to generate coherent intermediate frames, preserving the original video's semantic content. Since comprehensive test-time training can be time-consuming, we adopt a two-stage strategy: pretraining the inpainting module on a small video set to learn general motion priors, followed by rapid inference-time fine-tuning for video-specific adaptation. This hybrid approach enables adaptation within 10 minutes with one epoch on a single NVIDIA 4090 GPU using CogVideoX-5b-I2V as the backbone. Extensive experiments show that our approach can achieve high-quality beat alignment and visual smoothness.
Abstract (translated)
将视频中的视觉运动节奏与给定的音乐曲目对齐是多媒体制作的实际需求，但在自主视频编辑中仍是一个未充分探索的任务。有效地将动作和音乐节拍同步可以增强观众的参与度和视觉吸引力，特别是在音乐视频、宣传内容和电影剪辑方面。现有的方法通常依赖于劳动密集型的手动裁剪、速度调整或基于启发式的编辑技术来实现同步。尽管一些生成模型能够处理联合视频和音乐生成，但它们往往将这两种模式纠缠在一起，限制了在保持完整视觉内容的同时灵活对齐视频与音乐节拍的能力。 为此，在本文中我们提出了一种新颖且高效的框架，称为MVAA（Music-Video Auto-Alignment），该框架可以自动编辑视频以使其节奏与给定的音乐曲目同步，并同时保留原始视觉内容。为了增强灵活性，我们将任务模块化为MVAA中的两步过程：首先将运动关键帧对齐到音频节拍上，然后进行感知节奏的视频修复。具体来说，我们先在时间戳处插入与音乐节拍对齐的关键帧，随后使用受条件约束的扩散模型生成连贯的中间帧，同时保留原始视频的语义内容。 鉴于全面的时间测试训练可能耗时较长，我们采用了一种两阶段策略：首先在一个小规模视频集上预训练修复模块以学习通用运动先验知识，然后进行快速推理时间微调来适应特定视频。这种混合方法能够在单个NVIDIA 4090 GPU上使用CogVideoX-5b-I2V作为骨干网络的情况下，在一个epoch内（10分钟内）完成自适应调整。 广泛的实验表明，我们的方法能够实现高质量的节拍对齐和视觉平滑度。
URL
https://arxiv.org/abs/2506.18881
PDF
-
MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
2025-06-23 15:08:03Fang-Duo Tsai, Shih-Lun Wu, Weijaw Lee, Sheng-Ping Yang, Bo-Rui Chen, Hao-Chung Cheng, Yi-Hsuan YangAbstract
We propose MuseControlLite, a lightweight mechanism designed to fine-tune text-to-music generation models for precise conditioning using various time-varying musical attributes and reference audio signals. The key finding is that positional embeddings, which have been seldom used by text-to-music generation models in the conditioner for text conditions, are critical when the condition of interest is a function of time. Using melody control as an example, our experiments show that simply adding rotary positional embeddings to the decoupled cross-attention layers increases control accuracy from 56.6% to 61.1%, while requiring 6.75 times fewer trainable parameters than state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion Transformer model of Stable Audio Open. We evaluate various forms of musical attribute control, audio inpainting, and audio outpainting, demonstrating improved controllability over MusicGen-Large and Stable Audio Open ControlNet at a significantly lower fine-tuning cost, with only 85M trainble parameters. Source code, model checkpoints, and demo examples are available at: https: //MuseControlLite.this http URL.
Abstract (translated)
我们提出了一种名为MuseControlLite的轻量级机制，旨在通过各种随时间变化的音乐属性和参考音频信号对文本到音乐生成模型进行精细调优。关键发现是，在条件随时间变化的情况下，尽管文本条件化中很少使用位置嵌入（Positional Embeddings），它们却至关重要。以旋律控制为例，我们的实验表明，只需在解耦的交叉注意力层上添加旋转位置嵌入，就可以将控制精度从56.6%提升到61.1%，同时训练参数仅为最先进微调机制的六分之一，并使用相同的预训练扩散Transformer模型（来自Stable Audio Open）。我们对各种音乐属性控制形式、音频修补和音频扩展进行了评估，证明了与MusicGen-Large和Stable Audio Open ControlNet相比，在显著降低微调成本的情况下，提高了可控性。MuseControlLite仅需85M可训练参数。源代码、模型检查点及演示示例可在以下网址获取：https://MuseControlLite.this http URL. 请注意，实际链接可能需要进一步确认或替换为正确的URL地址。
URL
https://arxiv.org/abs/2506.18729
PDF
-
VisualChef: Generating Visual Aids in Cooking via Mask Inpainting
2025-06-23 12:23:21Oleh Kuzyk, Zuoyue Li, Marc Pollefeys, Xi WangAbstract
Cooking requires not only following instructions but also understanding, executing, and monitoring each step - a process that can be challenging without visual guidance. Although recipe images and videos offer helpful cues, they often lack consistency in focus, tools, and setup. To better support the cooking process, we introduce VisualChef, a method for generating contextual visual aids tailored to cooking scenarios. Given an initial frame and a specified action, VisualChef generates images depicting both the action's execution and the resulting appearance of the object, while preserving the initial frame's environment. Previous work aims to integrate knowledge extracted from large language models by generating detailed textual descriptions to guide image generation, which requires fine-grained visual-textual alignment and involves additional annotations. In contrast, VisualChef simplifies alignment through mask-based visual grounding. Our key insight is identifying action-relevant objects and classifying them to enable targeted modifications that reflect the intended action and outcome while maintaining a consistent environment. In addition, we propose an automated pipeline to extract high-quality initial, action, and final state frames. We evaluate VisualChef quantitatively and qualitatively on three egocentric video datasets and show its improvements over state-of-the-art methods.
Abstract (translated)
烹饪不仅需要遵循指示，还需要理解、执行和监控每一个步骤——这一过程在缺乏视觉指导的情况下可能会变得具有挑战性。虽然食谱图片和视频提供了有用的提示，但它们往往在焦点、工具和设置方面缺乏一致性。为了更好地支持烹饪过程，我们引入了VisualChef，这是一种生成针对烹饪场景的上下文可视化辅助的方法。给定初始画面和指定的动作，VisualChef可以生成描绘动作执行以及对象最终外观的图像，同时保留初始画面中的环境。 以往的工作旨在通过从大型语言模型中提取知识来生成详细的文本描述以指导图像生成，这要求精细的视觉-文本对齐，并涉及额外的标注。相比之下，VisualChef通过基于掩码的视觉定位简化了对齐过程。我们的关键洞察是识别与动作相关的对象并对其进行分类，以便进行针对性修改，反映预期的动作和结果的同时保持环境的一致性。 此外，我们提出了一种自动管道来提取高质量的初始、动作和最终状态帧。我们在三个第一人称视频数据集上从定量和定性的角度评估了VisualChef，并展示了它相比现有最先进方法的进步。
URL
https://arxiv.org/abs/2506.18569
PDF
-
Frequency-Domain Fusion Transformer for Image Inpainting
2025-06-23 09:19:04Sijin He, Guangfeng Lin, Tao Li, Yajun ChenAbstract
Image inpainting plays a vital role in restoring missing image regions and supporting high-level vision tasks, but traditional methods struggle with complex textures and large occlusions. Although Transformer-based approaches have demonstrated strong global modeling capabilities, they often fail to preserve high-frequency details due to the low-pass nature of self-attention and suffer from high computational costs. To address these challenges, this paper proposes a Transformer-based image inpainting method incorporating frequency-domain fusion. Specifically, an attention mechanism combining wavelet transform and Gabor filtering is introduced to enhance multi-scale structural modeling and detail preservation. Additionally, a learnable frequency-domain filter based on the fast Fourier transform is designed to replace the feedforward network, enabling adaptive noise suppression and detail retention. The model adopts a four-level encoder-decoder structure and is guided by a novel loss strategy to balance global semantics and fine details. Experimental results demonstrate that the proposed method effectively improves the quality of image inpainting by preserving more high-frequency information.
Abstract (translated)
图像修复在恢复缺失的图像区域和支持高级视觉任务方面扮演着至关重要的角色，但传统方法在处理复杂纹理和大面积遮挡时往往表现不佳。虽然基于Transformer的方法展示了强大的全局建模能力，但由于自注意力机制的低通特性，它们通常无法很好地保留高频细节，并且计算成本较高。为了解决这些挑战，本文提出了一种结合频率域融合的基于Transformer的图像修复方法。具体来说，引入了一个结合小波变换和Gabor滤波器的注意机制，以增强多尺度结构建模和细节保留能力。此外，还设计了一种基于快速傅里叶变换（FFT）的可学习频域滤波器来替代前馈网络，从而实现自适应噪声抑制和细节保持功能。 该模型采用了四级编码-解码架构，并通过一种新的损失策略来进行指导，以平衡全局语义信息与精细细节。实验结果表明，所提出的方法有效提高了图像修复的质量，尤其是在保留更多高频信息方面表现出色。
URL
https://arxiv.org/abs/2506.18437
PDF
-
EgoWorld: Translating Exocentric View to Egocentric View using Rich Exocentric Observations
2025-06-22 04:21:48Junho Park, Andrew Sangwoo Ye, Taein KwonAbstract
Egocentric vision is essential for both human and machine visual understanding, particularly in capturing the detailed hand-object interactions needed for manipulation tasks. Translating third-person views into first-person views significantly benefits augmented reality (AR), virtual reality (VR) and robotics applications. However, current exocentric-to-egocentric translation methods are limited by their dependence on 2D cues, synchronized multi-view settings, and unrealistic assumptions such as necessity of initial egocentric frame and relative camera poses during inference. To overcome these challenges, we introduce EgoWorld, a novel two-stage framework that reconstructs an egocentric view from rich exocentric observations, including projected point clouds, 3D hand poses, and textual descriptions. Our approach reconstructs a point cloud from estimated exocentric depth maps, reprojects it into the egocentric perspective, and then applies diffusion-based inpainting to produce dense, semantically coherent egocentric images. Evaluated on the H2O and TACO datasets, EgoWorld achieves state-of-the-art performance and demonstrates robust generalization to new objects, actions, scenes, and subjects. Moreover, EgoWorld shows promising results even on unlabeled real-world examples.
Abstract (translated)
自视视觉对于人类和机器的视觉理解都至关重要，尤其是在捕捉操作任务所需的详细手部与物体互动方面。将第三人称视角转换为第一人称视角可以显著提升增强现实（AR）、虚拟现实（VR）以及机器人技术的应用效果。然而，当前从外向内（exocentric到egocentric）视图转化的方法受到依赖二维线索、同步多视角设置以及诸如需要初始自视帧和推断过程中的相对相机姿态等不切实际假设的限制。 为了解决这些挑战，我们引入了EgoWorld，这是一种新颖的两阶段框架，可以从丰富的外向观察（包括投影点云、三维手部姿势和文本描述）中重构出第一人称视角。我们的方法首先根据估计的第三人称深度图重建点云，然后将其重新投影到自视视角，并通过扩散基础插值生成稠密且语义一致的第一人称图像。 在H2O和TACO数据集上的评估显示，EgoWorld达到了最先进的性能水平，并展示了对新对象、动作、场景及主体的稳健泛化能力。此外，即使在未标记的真实世界示例上，EgoWorld也表现出令人鼓舞的结果。
URL
https://arxiv.org/abs/2506.17896
PDF
-
3DeepRep: 3D Deep Low-rank Tensor Representation for Hyperspectral Image Inpainting
2025-06-20 04:09:34Yunshan Li, Wenwu Gong, Qianqian Wang, Chao Wang, Lili YangAbstract
Recent approaches based on transform-based tensor nuclear norm (TNN) have demonstrated notable effectiveness in hyperspectral image (HSI) inpainting by leveraging low-rank structures in latent representations. Recent developments incorporate deep transforms to improve low-rank tensor representation; however, existing approaches typically restrict the transform to the spectral mode, neglecting low-rank properties along other tensor modes. In this paper, we propose a novel 3-directional deep low-rank tensor representation (3DeepRep) model, which performs deep nonlinear transforms along all three modes of the HSI tensor. To enforce low-rankness, the model minimizes the nuclear norms of mode-i frontal slices in the corresponding latent space for each direction (i=1,2,3), forming a 3-directional TNN regularization. The outputs from the three directional branches are subsequently fused via a learnable aggregation module to produce the final result. An efficient gradient-based optimization algorithm is developed to solve the model in a self-supervised manner. Extensive experiments on real-world HSI datasets demonstrate that the proposed method achieves superior inpainting performance compared to existing state-of-the-art techniques, both qualitatively and quantitatively.
Abstract (translated)
最近基于变换的张量核范数（TNN）的方法在高光谱图像（HSI）修复中展示了显著的有效性，通过利用潜在表示中的低秩结构。近期的发展将深度转换纳入其中以改进低秩张量表示；然而，现有方法通常只限制变换在光谱模式上进行，忽略了其他张量模式上的低秩特性。本文提出了一个新颖的三方向深度低秩张量表示（3DeepRep）模型，在HSI张量的所有三个模式上执行深度非线性转换。为了强制执行低秩性质，该模型通过最小化每个方向对应的潜在空间中mode-i前切片的核范数来形成三向TNN正则化（i=1,2,3）。来自三个方向分支的输出随后通过一个可学习的聚合模块融合以生成最终结果。开发了一种有效的基于梯度的优化算法，以便自监督地解决该模型。在真实世界HSI数据集上的大量实验表明，所提出的方法在定性和定量上均优于现有的最先进的技术，在图像修复性能方面表现出色。
URL
https://arxiv.org/abs/2506.16735
PDF
-
Exploring Diffusion with Test-Time Training on Efficient Image Restoration
2025-06-17 14:01:59Rongchang Lu, Tianduo Luo, Yunzhi Zhang, Conghan Yue, Pei Yang, Guibao Liu, Changyang GuAbstract
Image restoration faces challenges including ineffective feature fusion, computational bottlenecks and inefficient diffusion processes. To address these, we propose DiffRWKVIR, a novel framework unifying Test-Time Training (TTT) with efficient diffusion. Our approach introduces three key innovations: (1) Omni-Scale 2D State Evolution extends RWKV's location-dependent parameterization to hierarchical multi-directional 2D scanning, enabling global contextual awareness with linear complexity O(L); (2) Chunk-Optimized Flash Processing accelerates intra-chunk parallelism by 3.2x via contiguous chunk processing (O(LCd) complexity), reducing sequential dependencies and computational overhead; (3) Prior-Guided Efficient Diffusion extracts a compact Image Prior Representation (IPR) in only 5-20 steps, proving 45% faster training/inference than DiffIR while solving computational inefficiency in denoising. Evaluated across super-resolution and inpainting benchmarks (Set5, Set14, BSD100, Urban100, Places365), DiffRWKVIR outperforms SwinIR, HAT, and MambaIR/v2 in PSNR, SSIM, LPIPS, and efficiency metrics. Our method establishes a new paradigm for adaptive, high-efficiency image restoration with optimized hardware utilization.
Abstract (translated)
图像恢复面临包括无效特征融合、计算瓶颈和低效扩散过程等挑战。为解决这些问题，我们提出了一种新的框架——DiffRWKVIR，它将测试时间训练（TTT）与高效的扩散过程结合起来。我们的方法引入了三个关键创新： 1. **Omni-Scale 2D State Evolution**：扩展了RWKV的位置依赖参数化到层次化的多方向二维扫描，使得全局上下文感知具有线性复杂度O(L)。 2. **Chunk-Optimized Flash Processing**：通过连续的块处理加速了跨块并行性（复杂度为O(LCd)），减少了顺序依赖和计算开销，并提升了3.2倍的速度。 3. **Prior-Guided Efficient Diffusion**：在仅需5到20步的情况下提取紧凑的图像先验表示（IPR），相比DiffIR，训练和推理速度快45%，同时解决了去噪中的计算效率问题。 我们对超分辨率和补全基准（Set5、Set14、BSD100、Urban100、Places365）进行了评估。结果表明，DiffRWKVIR在PSNR、SSIM、LPIPS以及效率指标上均优于SwinIR、HAT和MambaIR/v2。 我们的方法为自适应且高效的图像恢复建立了新的范例，并实现了优化的硬件利用率。
URL
https://arxiv.org/abs/2506.14541
PDF
-
orGAN: A Synthetic Data Augmentation Pipeline for Simultaneous Generation of Surgical Images and Ground Truth Labels
2025-06-17 08:29:40Niran Nataraj, Maina Sogabe, Kenji KawashimaAbstract
Deep learning in medical imaging faces obstacles: limited data diversity, ethical issues, high acquisition costs, and the need for precise annotations. Bleeding detection and localization during surgery is especially challenging due to the scarcity of high-quality datasets that reflect real surgical scenarios. We propose orGAN, a GAN-based system for generating high-fidelity, annotated surgical images of bleeding. By leveraging small "mimicking organ" datasets, synthetic models that replicate tissue properties and bleeding, our approach reduces ethical concerns and data-collection costs. orGAN builds on StyleGAN with Relational Positional Learning to simulate bleeding events realistically and mark bleeding coordinates. A LaMa-based inpainting module then restores clean, pre-bleed visuals, enabling precise pixel-level annotations. In evaluations, a balanced dataset of orGAN and mimicking-organ images achieved 90% detection accuracy in surgical settings and up to 99% frame-level accuracy. While our development data lack diverse organ morphologies and contain intraoperative artifacts, orGAN markedly advances ethical, efficient, and cost-effective creation of realistic annotated bleeding datasets, supporting broader integration of AI in surgical practice.
Abstract (translated)
深度学习在医学影像领域面临诸多挑战，包括数据多样性有限、伦理问题、高昂的数据获取成本以及精确标注的需求。尤其是在手术过程中检测和定位出血方面，由于高质量数据集稀缺且难以反映真实的外科场景，这一任务尤为艰巨。 我们提出了一种基于生成对抗网络（GAN）的系统——orGAN，用于生成高保真度并带有注释标签的手术出血图像。通过利用小型“模拟器官”数据集，并使用合成模型复制组织特性和出血情况，我们的方法减少了伦理担忧和数据收集成本。orGAN 建立在 StyleGAN 和关系位置学习基础上，能够逼真地模拟出血事件并标记出血坐标。随后，orGAN 使用 LaMa（Large Mask）为基础的图像修复模块恢复干净、无出血的原始视图，从而实现精确的像素级注释。 评估结果显示，在手术场景中，一个平衡混合了 orGAN 生成数据和“模拟器官”真实数据的数据集达到了90%的检测准确率，并在帧级别上实现了高达99%的准确度。尽管我们的开发数据缺乏多样化的器官形态并且包含术中的图像伪影，但orGAN 显著推进了伦理、高效且低成本地创建逼真的标注出血数据集的方法。这种方法支持人工智能技术更广泛应用于外科实践中。
URL
https://arxiv.org/abs/2506.14303
PDF
-
VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models
2025-06-16 17:58:00Edward Li, Zichen Wang, Jiahe Huang, Jeong Joon ParkAbstract
We present a unified framework for solving partial differential equations (PDEs) using video-inpainting diffusion transformer models. Unlike existing methods that devise specialized strategies for either forward or inverse problems under full or partial observation, our approach unifies these tasks under a single, flexible generative framework. Specifically, we recast PDE-solving as a generalized inpainting problem, e.g., treating forward prediction as inferring missing spatiotemporal information of future states from initial conditions. To this end, we design a transformer-based architecture that conditions on arbitrary patterns of known data to infer missing values across time and space. Our method proposes pixel-space video diffusion models for fine-grained, high-fidelity inpainting and conditioning, while enhancing computational efficiency through hierarchical modeling. Extensive experiments show that our video inpainting-based diffusion model offers an accurate and versatile solution across a wide range of PDEs and problem setups, outperforming state-of-the-art baselines.
Abstract (translated)
我们提出了一种使用视频修复扩散变换器模型解决偏微分方程（PDE）的统一框架。与现有方法不同，后者为完整或部分观测下的正向问题或逆向问题设计了专门策略，我们的方法将这些任务统一在一个单一且灵活的生成框架下进行处理。具体来说，我们将求解PDE的问题重新表述为一个广义修复问题，例如，将其视为从初始条件推断未来状态缺失的空间-时间信息的过程。 为此，我们设计了一种基于变换器架构的方法，它可以根据任意已知数据模式，在时间和空间上推测出缺失的值。我们的方法采用像素空间视频扩散模型进行精细化、高保真度的修复和条件设定，并通过分层建模提高计算效率。 大量的实验表明，我们提出的基于视频修复的扩散模型在广泛的PDE问题设置中提供了准确且灵活的解决方案，超过了现有的最先进基准。
URL
https://arxiv.org/abs/2506.13754
PDF
-
Exploiting the Exact Denoising Posterior Score in Training-Free Guidance of Diffusion Models
2025-06-16 15:43:28Gregory BellchambersAbstract
The success of diffusion models has driven interest in performing conditional sampling via training-free guidance of the denoising process to solve image restoration and other inverse problems. A popular class of methods, based on Diffusion Posterior Sampling (DPS), attempts to approximate the intractable posterior score function directly. In this work, we present a novel expression for the exact posterior score for purely denoising tasks that is tractable in terms of the unconditional score function. We leverage this result to analyze the time-dependent error in the DPS score for denoising tasks and compute step sizes on the fly to minimize the error at each time step. We demonstrate that these step sizes are transferable to related inverse problems such as colorization, random inpainting, and super resolution. Despite its simplicity, this approach is competitive with state-of-the-art techniques and enables sampling with fewer time steps than DPS.
Abstract (translated)
扩散模型的成功激发了人们通过无训练的去噪过程引导进行条件采样，以解决图像恢复和其他逆向问题的兴趣。基于扩散后验采样（DPS）的一类流行方法试图直接近似难以处理的后验评分函数。在这项工作中，我们提出了一个全新的、对于纯去噪任务而言可操作的确切后验评分表达式，并且该表达式可以用无条件评分函数来表示。利用这一结果，我们分析了DPS在去噪任务中的时间依赖误差，并实时计算步长以减少每个时间点的误差。我们展示了这些步长可以转移到相关逆向问题中，例如着色、随机插值和超分辨率。 尽管这种方法简单，但它与最先进的技术竞争性很强，并且可以在比DPS更少的时间步骤内进行采样。
URL
https://arxiv.org/abs/2506.13614
PDF
-
DicFace: Dirichlet-Constrained Variational Codebook Learning for Temporally Coherent Video Face Restoration
2025-06-16 10:54:28Yan Chen, Hanlin Shang, Ce Liu, Yuxuan Chen, Hui Li, Weihao Yuan, Hao Zhu, Zilong Dong, Siyu ZhuAbstract
Video face restoration faces a critical challenge in maintaining temporal consistency while recovering fine facial details from degraded inputs. This paper presents a novel approach that extends Vector-Quantized Variational Autoencoders (VQ-VAEs), pretrained on static high-quality portraits, into a video restoration framework through variational latent space modeling. Our key innovation lies in reformulating discrete codebook representations as Dirichlet-distributed continuous variables, enabling probabilistic transitions between facial features across frames. A spatio-temporal Transformer architecture jointly models inter-frame dependencies and predicts latent distributions, while a Laplacian-constrained reconstruction loss combined with perceptual (LPIPS) regularization enhances both pixel accuracy and visual quality. Comprehensive evaluations on blind face restoration, video inpainting, and facial colorization tasks demonstrate state-of-the-art performance. This work establishes an effective paradigm for adapting intensive image priors, pretrained on high-quality images, to video restoration while addressing the critical challenge of flicker artifacts. The source code has been open-sourced and is available at this https URL.
Abstract (translated)
视频面部修复面临的一个关键挑战是在恢复降质输入中的精细面部细节的同时保持时间一致性。本文提出了一种新颖的方法，该方法通过变分潜在空间建模将基于静态高质量肖像预训练的向量量化变分自动编码器（VQ-VAEs）扩展到视频修复框架中。我们的关键技术创新在于重新表述离散代码本表示为狄利克雷分布连续变量，从而使得在帧之间进行面部特征的概率转换成为可能。一种时空变换架构同时建模帧间的依赖关系，并预测潜在分布，而拉普拉斯约束重建损失与感知（LPIPS）正则化的结合增强了像素准确性和视觉质量。在盲面修复、视频插值和面部着色任务上的全面评估表明了该方法处于业界领先地位。这项工作建立了一种有效的范式，将基于高质量图像的密集图像先验知识适应到视频修复中，并解决了闪烁伪影的关键挑战。源代码已开源并可在提供的URL获取。
URL
https://arxiv.org/abs/2506.13355
PDF
-
ContextBench: Modifying Contexts for Targeted Latent Activation
2025-06-15 16:54:09Robert Graham, Edward Stevinson, Leo Richter, Alexander Chia, Joseph Miller, Joseph Isaac BloomAbstract
Identifying inputs that trigger specific behaviours or latent features in language models could have a wide range of safety use cases. We investigate a class of methods capable of generating targeted, linguistically fluent inputs that activate specific latent features or elicit model behaviours. We formalise this approach as context modification and present ContextBench -- a benchmark with tasks assessing core method capabilities and potential safety applications. Our evaluation framework measures both elicitation strength (activation of latent features or behaviours) and linguistic fluency, highlighting how current state-of-the-art methods struggle to balance these objectives. We enhance Evolutionary Prompt Optimisation (EPO) with LLM-assistance and diffusion model inpainting, and demonstrate that these variants achieve state-of-the-art performance in balancing elicitation effectiveness and fluency.
Abstract (translated)
识别能够触发特定行为或潜在特征的语言模型输入可能具有广泛的安全应用场景。我们研究了一类可以生成针对性强、语言流利的输入的方法，这些输入能激活特定的潜在特性或引发模型的行为。我们将这种方法形式化为上下文修改，并提出了ContextBench——一个基准测试集，其中包括评估核心方法能力和潜在安全应用的任务。我们的评估框架衡量了激发强度（即激活潜在特征或行为）和语言流畅性，强调了当前最先进的方法在平衡这些目标方面所面临的挑战。我们通过引入大模型辅助和扩散模型修复技术来增强进化提示优化（EPO），并证明这些变体在平衡激发有效性和流畅性方面的性能达到了最新水平。
URL
https://arxiv.org/abs/2506.15735
PDF
-
EraserDiT: Fast Video Inpainting with Diffusion Transformer Model
2025-06-15 13:59:57Jie Liu, Zheng HuiAbstract
Video object removal and inpainting are critical tasks in the fields of computer vision and multimedia processing, aimed at restoring missing or corrupted regions in video sequences. Traditional methods predominantly rely on flow-based propagation and spatio-temporal Transformers, but these approaches face limitations in effectively leveraging long-term temporal features and ensuring temporal consistency in the completion results, particularly when dealing with large masks. Consequently, performance on extensive masked areas remains suboptimal. To address these challenges, this paper introduces a novel video inpainting approach leveraging the Diffusion Transformer (DiT). DiT synergistically combines the advantages of diffusion models and transformer architectures to maintain long-term temporal consistency while ensuring high-quality inpainting results. We propose a Circular Position-Shift strategy to further enhance long-term temporal consistency during the inference stage. Additionally, the proposed method automatically detects objects within videos, interactively removes specified objects, and generates corresponding prompts. In terms of processing speed, it takes only 180 seconds (testing on one NVIDIA A100 GPU) to complete a video with a resolution of $1080 \times 1920$ with 121 frames without any acceleration method. Experimental results indicate that the proposed method demonstrates superior performance in content fidelity, texture restoration, and temporal consistency. Project page: this https URL.
Abstract (translated)
视频对象移除和修复是计算机视觉与多媒体处理领域中的关键任务，旨在恢复视频序列中丢失或损坏的区域。传统方法主要依赖于基于流的传播和时空Transformer，但这些方法在有效利用长期时间特征及确保完成结果的时间一致性方面存在局限性，特别是在处理大范围遮罩时尤为明显。因此，在广泛的被屏蔽区域内性能仍然不理想。 为解决这些问题，本文介绍了一种新颖的视频修复方法，该方法利用扩散变换器（Diffusion Transformer, DiT）。DiT 结合了扩散模型和变换器架构的优势，在保持长期时间一致性的同时确保高质量的修复结果。我们还提出了一种圆形位置偏移策略，以进一步增强推理阶段中的长时一致性。此外，所提出的这种方法能够自动检测视频内的对象，并交互式地移除指定的对象并生成相应的提示。 在处理速度方面，在没有加速方法的情况下使用一个 NVIDIA A100 GPU 测试分辨率为 1080x1920、包含 121 帧的视频，整个修复过程仅需 180 秒。实验结果表明，所提出的方法在内容保真度、纹理恢复和时间一致性方面表现出色。 项目页面：[点击这里查看](https://this-url.com/)（请将“this https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2506.12853
PDF
-
iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer
2025-06-15 13:41:43Zhelun Shen, Chenming Wu, Junsheng Zhou, Chen Zhao, Kaisiyuan Wang, Hang Zhou, Yingying Li, Haocheng Feng, Wei He, Jingdong WangAbstract
Digital human video generation is gaining traction in fields like education and e-commerce, driven by advancements in head-body animation and lip-syncing technologies. However, realistic Hand-Object Interaction (HOI) - the complex dynamics between human hands and objects - continues to pose challenges. Generating natural and believable HOI reenactments is difficult due to issues such as occlusion between hands and objects, variations in object shapes and orientations, and the necessity for precise physical interactions, and importantly, the ability to generalize to unseen humans and objects. This paper presents a novel framework iDiT-HOI that enables in-the-wild HOI reenactment generation. Specifically, we propose a unified inpainting-based token process method, called Inp-TPU, with a two-stage video diffusion transformer (DiT) model. The first stage generates a key frame by inserting the designated object into the hand region, providing a reference for subsequent frames. The second stage ensures temporal coherence and fluidity in hand-object interactions. The key contribution of our method is to reuse the pretrained model's context perception capabilities without introducing additional parameters, enabling strong generalization to unseen objects and scenarios, and our proposed paradigm naturally supports long video generation. Comprehensive evaluations demonstrate that our approach outperforms existing methods, particularly in challenging real-world scenes, offering enhanced realism and more seamless hand-object interactions.
Abstract (translated)
数字人视频生成在教育和电子商务等领域正逐渐受到重视，这得益于头部与身体动画技术及唇同步技术的进步。然而，真实的“手-物交互”（HOI）——即人类双手与物体之间的复杂动态关系——仍然面临诸多挑战。由于手部与物体间的遮挡问题、不同形状和姿态的物体变化以及对精确物理互动的需求等原因，生成自然且可信的手物交互重演变得困难重重，尤其是当需要泛化到未见过的人体和物体时。 本文提出了一种名为iDiT-HOI的新框架，能够实现野外环境中的手物交互重现。具体来说，我们提出了一个统一的基于补全的标记处理方法——Inp-TPU，并采用了一个两阶段视频扩散变压器（DiT）模型。第一阶段通过在指定的手部区域插入目标物体来生成关键帧，为后续帧提供参考；第二阶段确保手物互动在时间上的连贯性和流畅性。 本方法的关键贡献在于，无需引入额外参数的情况下重用预训练模型的上下文感知能力，从而实现对未知对象和场景的强大泛化能力。此外，我们提出的范式自然支持长视频生成。全面评估表明，我们的方法优于现有技术，在具有挑战性的现实世界场景中表现出更强的真实感和更平滑的手物交互体验。
URL
https://arxiv.org/abs/2506.12847
PDF
-
3D Hand Mesh-Guided AI-Generated Malformed Hand Refinement with Hand Pose Transformation via Diffusion Model
2025-06-15 01:30:22Chen-Bin Feng, Kangdao Liu, Jian Sun, Jiping Jin, Yiguo Jiang, Chi-Man VongAbstract
The malformed hands in the AI-generated images seriously affect the authenticity of the images. To refine malformed hands, existing depth-based approaches use a hand depth estimator to guide the refinement of malformed hands. Due to the performance limitations of the hand depth estimator, many hand details cannot be represented, resulting in errors in the generated hands, such as confusing the palm and the back of the hand. To solve this problem, we propose a 3D mesh-guided refinement framework using a diffusion pipeline. We use a state-of-the-art 3D hand mesh estimator, which provides more details of the hands. For training, we collect and reannotate a dataset consisting of RGB images and 3D hand mesh. Then we design a diffusion inpainting model to generate refined outputs guided by 3D hand meshes. For inference, we propose a double check algorithm to facilitate the 3D hand mesh estimator to obtain robust hand mesh guidance to obtain our refined results. Beyond malformed hand refinement, we propose a novel hand pose transformation method. It increases the flexibility and diversity of the malformed hand refinement task. We made the restored images mimic the hand poses of the reference images. The pose transformation requires no additional training. Extensive experimental results demonstrate the superior performance of our proposed method.
Abstract (translated)
在AI生成的图像中，畸形的手严重影响了图片的真实性。现有的基于深度的方法通过使用手部深度估计器来指导畸形手部的精炼过程。然而，由于手部深度估计器性能的限制，许多手部细节无法被准确表示，导致生成的手部出现错误，例如将手掌与手背混淆。为了解决这个问题，我们提出了一种基于3D网格引导的精炼框架，并采用扩散管道进行优化。我们使用最先进的3D手部网格估计算法，该算法提供了更多的手部细节信息。 在训练阶段，我们收集并重新标注了一个包含RGB图像和3D手部网格的数据集。然后设计一个基于扩散的修补模型，以生成由3D手部网格引导的精炼输出结果。在推断过程中，我们提出了一种双检查算法，使3D手部网格估计算法能够获得稳健的手部网格指引，从而得到我们的精炼结果。 除了畸形手部的精炼外，我们还提出了一个新颖的手势变换方法，这种方法增加了畸形手部精炼任务的灵活性和多样性。我们将恢复后的图像模仿参考图像中的手势动作，而这种姿态转换并不需要额外的训练过程。 通过广泛的实验结果证明了我们所提出的方法具有卓越性能。
URL
https://arxiv.org/abs/2506.12680
PDF
-
Towards Seamless Borders: A Method for Mitigating Inconsistencies in Image Inpainting and Outpainting
2025-06-14 15:02:56Xingzhong Hou, Jie Wu, Boxiao Liu, Yi Zhang, Guanglu Song, Yunpeng Liu, Yu Liu, Haihang YouAbstract
Image inpainting is the task of reconstructing missing or damaged parts of an image in a way that seamlessly blends with the surrounding content. With the advent of advanced generative models, especially diffusion models and generative adversarial networks, inpainting has achieved remarkable improvements in visual quality and coherence. However, achieving seamless continuity remains a significant challenge. In this work, we propose two novel methods to address discrepancy issues in diffusion-based inpainting models. First, we introduce a modified Variational Autoencoder that corrects color imbalances, ensuring that the final inpainted results are free of color mismatches. Second, we propose a two-step training strategy that improves the blending of generated and existing image content during the diffusion process. Through extensive experiments, we demonstrate that our methods effectively reduce discontinuity and produce high-quality inpainting results that are coherent and visually appealing.
Abstract (translated)
图像修复是指重建图像中丢失或损坏的部分，使其与周围内容无缝融合。随着先进生成模型（尤其是扩散模型和生成对抗网络）的出现，图像修复在视觉质量和连贯性方面取得了显著的进步。然而，实现无缝连续性仍然是一个重大挑战。在这项工作中，我们提出了两种新颖的方法来解决基于扩散模型的图像修复中的不一致问题。首先，我们引入了一种改进的变分自编码器，可以纠正颜色失衡，确保最终修复的结果没有颜色差异。其次，我们提出了一种两步训练策略，通过这种方法可以在扩散过程中改善生成内容与现有图像内容之间的融合效果。通过广泛的实验，我们证明了我们的方法能够有效减少不连续性，并产生高质量、连贯且视觉上吸引人的图像修复结果。
URL
https://arxiv.org/abs/2506.12530
PDF
