Inpainting
-
Aligned Novel View Image and Geometry Synthesis via Cross-modal Attention Instillation
2025-06-13 16:19:00Min-Seop Kwak, Junho Kim, Sangdoo Yun, Dongyoon Han, Taekyoung Kim, Seungryong Kim, Jin-Hwa KimAbstract
We introduce a diffusion-based framework that performs aligned novel view image and geometry generation via a warping-and-inpainting methodology. Unlike prior methods that require dense posed images or pose-embedded generative models limited to in-domain views, our method leverages off-the-shelf geometry predictors to predict partial geometries viewed from reference images, and formulates novel-view synthesis as an inpainting task for both image and geometry. To ensure accurate alignment between generated images and geometry, we propose cross-modal attention distillation, where attention maps from the image diffusion branch are injected into a parallel geometry diffusion branch during both training and inference. This multi-task approach achieves synergistic effects, facilitating geometrically robust image synthesis as well as well-defined geometry prediction. We further introduce proximity-based mesh conditioning to integrate depth and normal cues, interpolating between point cloud and filtering erroneously predicted geometry from influencing the generation process. Empirically, our method achieves high-fidelity extrapolative view synthesis on both image and geometry across a range of unseen scenes, delivers competitive reconstruction quality under interpolation settings, and produces geometrically aligned colored point clouds for comprehensive 3D completion. Project page is available at this https URL.
Abstract (translated)
我们提出了一种基于扩散的框架，通过扭曲和修复的方法生成与新视角图像和几何形状对齐的新视图。与之前需要密集姿态图像或局限于领域内视图的嵌入姿势的生成模型的方法不同，我们的方法利用现成的几何预测器来预测从参考图像观察到的部分几何，并将新视图合成视为同时针对图像和几何的修复任务。为了确保生成的图像和几何之间精确对齐，我们提出了跨模态注意力蒸馏技术，在训练和推理过程中，将来自图像扩散分支的关注图注入平行的几何扩散分支中。这种多任务方法实现了协同效应，有助于实现几何鲁棒的图像合成以及明确定义的几何预测。 此外，为了整合深度和法线提示，我们将基于邻近性的网格条件引入该框架，通过在点云之间进行插值，并过滤掉错误预测的几何形状对生成过程的影响。实验证明，在各种未见过的场景中，我们的方法能够在图像和几何上实现高保真的外推视图合成；在内插设置下提供具有竞争力的重建质量；并且可以产生与颜色匹配的点云以进行全面的3D完成。 项目页面可在[此处](https://this-url.com)找到。
URL
https://arxiv.org/abs/2506.11924
PDF
-
VEIGAR: View-consistent Explicit Inpainting and Geometry Alignment for 3D object Removal
2025-06-13 11:31:44Pham Khai Nguyen Do, Bao Nguyen Tran, Nam Nguyen, Duc Dung NguyenAbstract
Recent advances in Novel View Synthesis (NVS) and 3D generation have significantly improved editing tasks, with a primary emphasis on maintaining cross-view consistency throughout the generative process. Contemporary methods typically address this challenge using a dual-strategy framework: performing consistent 2D inpainting across all views guided by embedded priors either explicitly in pixel space or implicitly in latent space; and conducting 3D reconstruction with additional consistency guidance. Previous strategies, in particular, often require an initial 3D reconstruction phase to establish geometric structure, introducing considerable computational overhead. Even with the added cost, the resulting reconstruction quality often remains suboptimal. In this paper, we present VEIGAR, a computationally efficient framework that outperforms existing methods without relying on an initial reconstruction phase. VEIGAR leverages a lightweight foundation model to reliably align priors explicitly in the pixel space. In addition, we introduce a novel supervision strategy based on scale-invariant depth loss, which removes the need for traditional scale-and-shift operations in monocular depth regularization. Through extensive experimentation, VEIGAR establishes a new state-of-the-art benchmark in reconstruction quality and cross-view consistency, while achieving a threefold reduction in training time compared to the fastest existing method, highlighting its superior balance of efficiency and effectiveness.
Abstract (translated)
最近在新颖视图合成（NVS）和3D生成方面取得的进展显著提升了编辑任务的效果，特别是在整个生成过程中的跨视角一致性维护方面。当前的方法通常采用双管齐下的框架来应对这一挑战：一种是通过像素空间中显式的嵌入先验或潜在空间中的隐式嵌入先验来进行所有视图的一致性2D修补；另一种是在进行3D重建时增加了一致性的指导。以往的策略往往需要一个初始的3D重建阶段来建立几何结构，这会带来相当大的计算开销。即使在增加了这一成本的情况下，最终得到的重建质量仍然常常不尽人意。 在这篇论文中，我们提出了一种名为VEIGAR的新框架，它比现有的方法更加高效且无需依赖于最初的重建阶段。VEIGAR利用一个轻量级的基础模型，在像素空间内可靠地对齐先验。此外，我们还引入了一种基于尺度不变深度损失的新型监督策略，这种方法消除了传统单目深度正则化中所需的缩放和平移操作。 通过广泛的实验，VEIGAR在重建质量和跨视角一致性方面建立了新的行业标准，并且训练时间比最快的现有方法减少了三倍，突显了其效率与效果之间的卓越平衡。
URL
https://arxiv.org/abs/2506.15821
PDF
-
InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model
2025-06-12 17:59:55Junqi You, Chieh Hubert Lin, Weijie Lyu, Zhengbo Zhang, Ming-Hsuan YangAbstract
Recent advances in 3D scene reconstruction enable real-time viewing in virtual and augmented reality. To support interactive operations for better immersiveness, such as moving or editing objects, 3D scene inpainting methods are proposed to repair or complete the altered geometry. However, current approaches rely on lengthy and computationally intensive optimization, making them impractical for real-time or online applications. We propose InstaInpaint, a reference-based feed-forward framework that produces 3D-scene inpainting from a 2D inpainting proposal within 0.4 seconds. We develop a self-supervised masked-finetuning strategy to enable training of our custom large reconstruction model (LRM) on the large-scale dataset. Through extensive experiments, we analyze and identify several key designs that improve generalization, textural consistency, and geometric correctness. InstaInpaint achieves a 1000x speed-up from prior methods while maintaining a state-of-the-art performance across two standard benchmarks. Moreover, we show that InstaInpaint generalizes well to flexible downstream applications such as object insertion and multi-region inpainting. More video results are available at our project page: this https URL.
Abstract (translated)
最近在3D场景重建领域的进展使得虚拟现实和增强现实中实时光学成为可能。为了支持更好的沉浸感并实现诸如移动或编辑对象的互动操作，提出了一些三维场景修复方法（即3D scene inpainting），用于修正或完成被改变的几何形状。然而，目前的方法依赖于耗时且计算密集型的优化过程，这使得它们不适合实时或在线应用。 我们提出了InstaInpaint，这是一种基于参考的前馈框架，能够从一个2D修复建议中生成3D场景修补结果，在0.4秒内完成任务。为了训练我们的自定义大型重建模型（LRM）以适应大规模数据集，我们开发了一种自我监督的掩码微调策略。通过广泛的实验，我们分析并识别了几项关键设计改进，这些改进提升了泛化能力、纹理一致性和几何正确性。 InstaInpaint相较于先前的方法，在速度上实现了1000倍的提升，并且在两个标准基准测试中保持了最先进的性能水平。此外，我们展示了InstaInpaint在诸如对象插入和多区域修补等灵活的应用场景中的良好泛化能力。更多视频结果可在我们的项目页面查看：[这个链接](https://example.com)（原文中提供的URL格式）。
URL
https://arxiv.org/abs/2506.10980
PDF
-
VINCIE: Unlocking In-context Image Editing from Video
2025-06-12 17:46:54Leigang Qu, Feng Cheng, Ziyan Yang, Qi Zhao, Shanchuan Lin, Yichun Shi, Yicong Li, Wenjie Wang, Tat-Seng Chua, Lu JiangAbstract
In-context image editing aims to modify images based on a contextual sequence comprising text and previously generated images. Existing methods typically depend on task-specific pipelines and expert models (e.g., segmentation and inpainting) to curate training data. In this work, we explore whether an in-context image editing model can be learned directly from videos. We introduce a scalable approach to annotate videos as interleaved multimodal sequences. To effectively learn from this data, we design a block-causal diffusion transformer trained on three proxy tasks: next-image prediction, current segmentation prediction, and next-segmentation prediction. Additionally, we propose a novel multi-turn image editing benchmark to advance research in this area. Extensive experiments demonstrate that our model exhibits strong in-context image editing capabilities and achieves state-of-the-art results on two multi-turn image editing benchmarks. Despite being trained exclusively on videos, our model also shows promising abilities in multi-concept composition, story generation, and chain-of-editing applications.
Abstract (translated)
在上下文中的图像编辑旨在根据包括文本和先前生成的图像在内的上下文序列对图像进行修改。现有的方法通常依赖于特定任务的工作流程和专家模型（例如分割和修复）来整理训练数据。在这项工作中，我们探讨了直接从视频中学习一个上下文图像编辑模型的可能性。我们引入了一种可扩展的方法，将视频注释为交错的多模态序列。为了有效地利用这些数据进行学习，我们设计了一个基于块因果扩散变换器，并针对三个代理任务进行了训练：下一帧预测、当前分割预测和下一帧分割预测。此外，我们还提出了一项新颖的多轮图像编辑基准测试，以推动该领域的研究进展。广泛的实验表明，我们的模型展现了强大的上下文图像编辑能力，在两个多轮图像编辑基准上取得了最先进的成果。尽管仅通过视频进行训练，但我们的模型在多概念组合、故事生成和编辑链应用方面也展现出了令人鼓舞的能力。
URL
https://arxiv.org/abs/2506.10941
PDF
-
Conditional diffusion models for guided anomaly detection in brain images using fluid-driven anomaly randomization
2025-06-11 23:27:00Ana Lawry Aguila, Peirong Liu, Oula Puonti, Juan Eugenio IglesiasAbstract
Supervised machine learning has enabled accurate pathology detection in brain MRI, but requires training data from diseased subjects that may not be readily available in some scenarios, for example, in the case of rare diseases. Reconstruction-based unsupervised anomaly detection, in particular using diffusion models, has gained popularity in the medical field as it allows for training on healthy images alone, eliminating the need for large disease-specific cohorts. These methods assume that a model trained on normal data cannot accurately represent or reconstruct anomalies. However, this assumption often fails with models failing to reconstruct healthy tissue or accurately reconstruct abnormal regions i.e., failing to remove anomalies. In this work, we introduce a novel conditional diffusion model framework for anomaly detection and healthy image reconstruction in brain MRI. Our weakly supervised approach integrates synthetically generated pseudo-pathology images into the modeling process to better guide the reconstruction of healthy images. To generate these pseudo-pathologies, we apply fluid-driven anomaly randomization to augment real pathology segmentation maps from an auxiliary dataset, ensuring that the synthetic anomalies are both realistic and anatomically coherent. We evaluate our model's ability to detect pathology, using both synthetic anomaly datasets and real pathology from the ATLAS dataset. In our extensive experiments, our model: (i) consistently outperforms variational autoencoders, and conditional and unconditional latent diffusion; and (ii) surpasses on most datasets, the performance of supervised inpainting methods with access to paired diseased/healthy images.
Abstract (translated)
监督式机器学习已经在脑部MRI中实现了准确的病理检测，但需要来自患病个体的数据进行训练，在某些情况下这些数据可能难以获得，例如在罕见病的情况下。基于重建的无监督异常检测方法，特别是使用扩散模型的方法，在医学领域变得越来越受欢迎，因为它允许仅通过健康图像进行训练，从而消除了对特定疾病的大规模患者群体的需求。然而，这种方法假设一个经过正常数据训练的模型无法准确表示或重构异常情况，这在实践中经常不成立，因为模型可能无法精确地重建健康的组织或者准确地重构异常区域。 在这项工作中，我们引入了一个新颖的条件扩散模型框架用于脑部MRI中的异常检测和健康图像重构。我们的弱监督方法将通过合成生成伪病理图像来更好地指导健康图像的重构过程。为了生成这些伪病理图像，我们将流体驱动的异常随机化应用于辅助数据集中真实病理分割图，确保合成的异常既现实又解剖结构上一致。 我们在使用合成异常数据集和ATLAS数据集中的真实病理学的情况下评估了我们模型的异常检测能力。在广泛的实验中，我们的模型：（i）始终优于变分自编码器、条件和非条件潜在扩散方法；并且（ii）在大多数数据集中超过了那些可以访问患病/健康图像配对的监督式图像修复方法的表现。 通过这种方式，该研究不仅提高了无监督异常检测技术的有效性，还为处理罕见疾病的MRI分析提供了新的途径。
URL
https://arxiv.org/abs/2506.10233
PDF
-
HiSin: Efficient High-Resolution Sinogram Inpainting via Resolution-Guided Progressive Inference
2025-06-10 13:59:25Jiaze E, Srutarshi Banerjee, Tekin Bicer, Guannan Wang, Yanfu Zhang, Bin RenAbstract
High-resolution sinogram inpainting is essential for computed tomography reconstruction, as missing high-frequency projections can lead to visible artifacts and diagnostic errors. Diffusion models are well-suited for this task due to their robustness and detail-preserving capabilities, but their application to high-resolution inputs is limited by excessive memory and computational demands. To address this limitation, we propose HiSin, a novel diffusion based framework for efficient sinogram inpainting via resolution-guided progressive inference. It progressively extracts global structure at low resolution and defers high-resolution inference to small patches, enabling memory-efficient inpainting. It further incorporates frequency-aware patch skipping and structure-adaptive step allocation to reduce redundant computation. Experimental results show that HiSin reduces peak memory usage by up to 31.25% and inference time by up to 18.15%, and maintains inpainting accuracy across datasets, resolutions, and mask conditions.
Abstract (translated)
高分辨率的扇束数据（sinogram）修复对于计算机断层成像（CT）重建至关重要，因为缺少高频投影会导致可见伪影和诊断错误。扩散模型由于其鲁棒性和细节保留能力而非常适合此任务，但它们在处理高分辨率输入时会受到内存和计算需求过大的限制。为了解决这一问题，我们提出了一种新的基于扩散的框架——HiSin，用于通过分辨率引导的逐步推理来进行高效的sinogram修复。该方法逐步在低分辨率下提取全局结构，并将高分辨率推理推延到小块处理中进行，从而实现内存效率高的修复过程。此外，它还结合了频率感知的小块跳过和结构自适应步骤分配来减少冗余计算。 实验结果显示，HiSin可以将峰值内存使用量降低最多31.25%，并将推理时间缩短最多18.15%。同时，在不同数据集、分辨率以及掩膜条件下，修复准确性得以保持。
URL
https://arxiv.org/abs/2506.08809
PDF
-
Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation
2025-06-10 08:37:45Or Tal, Felix Kreuk, Yossi AdiAbstract
Recent progress in text-to-music generation has enabled models to synthesize high-quality musical segments, full compositions, and even respond to fine-grained control signals, e.g. chord progressions. State-of-the-art (SOTA) systems differ significantly across many dimensions, such as training datasets, modeling paradigms, and architectural choices. This diversity complicates efforts to evaluate models fairly and pinpoint which design choices most influence performance. While factors like data and architecture are important, in this study we focus exclusively on the modeling paradigm. We conduct a systematic empirical analysis to isolate its effects, offering insights into associated trade-offs and emergent behaviors that can guide future text-to-music generation systems. Specifically, we compare the two arguably most common modeling paradigms: Auto-Regressive decoding and Conditional Flow-Matching. We conduct a controlled comparison by training all models from scratch using identical datasets, training configurations, and similar backbone architectures. Performance is evaluated across multiple axes, including generation quality, robustness to inference configurations, scalability, adherence to both textual and temporally aligned conditioning, and editing capabilities in the form of audio inpainting. This comparative study sheds light on distinct strengths and limitations of each paradigm, providing actionable insights that can inform future architectural and training decisions in the evolving landscape of text-to-music generation. Audio sampled examples are available at: this https URL
Abstract (translated)
最近在文本到音乐生成领域的进展使得模型能够合成高质量的音乐片段、完整的乐曲，甚至响应精细控制信号（如和弦进程）。目前最先进的系统在许多维度上存在着显著差异，例如训练数据集、建模范式以及架构选择。这种多样性复杂化了公平评估模型的努力，并且难以确定哪些设计决策对性能影响最大。虽然像数据和架构这样的因素很重要，但在本研究中我们专注于探讨建模范式的影响。我们进行了一项系统性的实证分析，以隔离其效果，并提供有关相关权衡和新兴行为的见解，从而为未来的文本到音乐生成系统指明方向。具体而言，我们比较了两种最常用的建模范式：自回归解码（Auto-Regressive decoding）与条件流匹配（Conditional Flow-Matching）。为了进行有控制的比较，我们在相同的数据集、训练配置以及相似的基本架构上从头开始训练所有模型。性能评估涵盖多个维度，包括生成质量、推理配置下的鲁棒性、可扩展性、对文本和时间同步条件的遵守程度，以及以音频中涂（audio inpainting）形式出现的编辑能力。 这项比较研究揭示了每种范式的独特优势与局限，并提供了切实可行的见解，以指导未来在不断发展的文本到音乐生成领域的架构设计和训练决策。有关音频样本示例，请访问：[此链接](https://example.com)（请将此链接替换为实际提供的URL）。
URL
https://arxiv.org/abs/2506.08570
PDF
-
Highly Compressed Tokenizer Can Generate Without Training
2025-06-09 21:45:03L. Lao Beyer, T. Li, X. Chen, S. Karaman, K. HeAbstract
Commonly used image tokenizers produce a 2D grid of spatially arranged tokens. In contrast, so-called 1D image tokenizers represent images as highly compressed one-dimensional sequences of as few as 32 discrete tokens. We find that the high degree of compression achieved by a 1D tokenizer with vector quantization enables image editing and generative capabilities through heuristic manipulation of tokens, demonstrating that even very crude manipulations -- such as copying and replacing tokens between latent representations of images -- enable fine-grained image editing by transferring appearance and semantic attributes. Motivated by the expressivity of the 1D tokenizer's latent space, we construct an image generation pipeline leveraging gradient-based test-time optimization of tokens with plug-and-play loss functions such as reconstruction or CLIP similarity. Our approach is demonstrated for inpainting and text-guided image editing use cases, and can generate diverse and realistic samples without requiring training of any generative model.
Abstract (translated)
常用的图像标记化方法会产生一个二维的、空间排列的标记网格。相比之下，所谓的1D图像标记器将图像表示为高度压缩的一维序列，最少仅包含32个离散标记。我们发现，通过向量量化实现的高度压缩能力使得基于1D标记器进行图像编辑和生成功能成为可能，并且通过对标记进行启发式操作就能实现这一点。即使是最粗糙的操作——如在不同图像的潜在表示之间复制和替换标记——也能使精细的图像编辑成为现实，包括传输外观和语义属性。 受到1D标记器隐空间表达能力的启发，我们构建了一个基于梯度测试时间优化标记（使用可插拔损失函数如重构或CLIP相似性）的图像生成流水线。我们的方法在图像修复和文本引导的图像编辑场景中得到展示，并且可以生成多样而逼真的样本，无需训练任何生成模型。
URL
https://arxiv.org/abs/2506.08257
PDF
-
Difference Inversion: Interpolate and Isolate the Difference with Token Consistency for Image Analogy Generation
2025-06-09 13:34:47Hyunsoo Kim, Donghyun Kim, Suhyun KimAbstract
How can we generate an image B' that satisfies A:A'::B:B', given the input images A,A' and B? Recent works have tackled this challenge through approaches like visual in-context learning or visual instruction. However, these methods are typically limited to specific models (e.g. InstructPix2Pix. Inpainting models) rather than general diffusion models (e.g. Stable Diffusion, SDXL). This dependency may lead to inherited biases or lower editing capabilities. In this paper, we propose Difference Inversion, a method that isolates only the difference from A and A' and applies it to B to generate a plausible B'. To address model dependency, it is crucial to structure prompts in the form of a "Full Prompt" suitable for input to stable diffusion models, rather than using an "Instruction Prompt". To this end, we accurately extract the Difference between A and A' and combine it with the prompt of B, enabling a plug-and-play application of the difference. To extract a precise difference, we first identify it through 1) Delta Interpolation. Additionally, to ensure accurate training, we propose the 2) Token Consistency Loss and 3) Zero Initialization of Token Embeddings. Our extensive experiments demonstrate that Difference Inversion outperforms existing baselines both quantitatively and qualitatively, indicating its ability to generate more feasible B' in a model-agnostic manner.
Abstract (translated)
如何在给定输入图像A、A'和B的情况下生成一个满足A:A'::B:B'的图像B'? 近期的研究通过视觉上下文学习或视觉指令等方法来解决这一挑战，但这些方法通常局限于特定模型（如InstructPix2Pix、去噪模型）而不是通用扩散模型（如Stable Diffusion、SDXL）。这种依赖性可能导致继承偏见或较低的编辑能力。在本文中，我们提出了差异反转法，这是一种仅从A和A'之间提取差异并将其应用于B以生成合理的B'的方法。为了减少对特定模型的依赖，重要的是将提示结构为适用于稳定扩散模型的“完整提示”，而不是使用“指令提示”。为此，我们准确地提取了A和A'之间的差异，并将其与B的提示相结合，从而实现差异的即插即用应用。 为了精确提取差异，我们首先通过1）Delta插值来识别它。此外，为确保训练准确性，我们提出了2）Token一致性损失和3）Token嵌入零初始化。我们的大量实验表明，差异反转法在定量和定性上都优于现有的基线方法，这表明它可以以模型无关的方式生成更可行的B'。
URL
https://arxiv.org/abs/2506.07750
PDF
-
Flow-Anything: Learning Real-World Optical Flow Estimation from Large-Scale Single-view Images
2025-06-09 13:23:44Yingping Liang, Ying Fu, Yutao Hu, Wenqi Shao, Jiaming Liu, Debing ZhangAbstract
Optical flow estimation is a crucial subfield of computer vision, serving as a foundation for video tasks. However, the real-world robustness is limited by animated synthetic datasets for training. This introduces domain gaps when applied to real-world applications and limits the benefits of scaling up datasets. To address these challenges, we propose \textbf{Flow-Anything}, a large-scale data generation framework designed to learn optical flow estimation from any single-view images in the real world. We employ two effective steps to make data scaling-up promising. First, we convert a single-view image into a 3D representation using advanced monocular depth estimation networks. This allows us to render optical flow and novel view images under a virtual camera. Second, we develop an Object-Independent Volume Rendering module and a Depth-Aware Inpainting module to model the dynamic objects in the 3D representation. These two steps allow us to generate realistic datasets for training from large-scale single-view images, namely \textbf{FA-Flow Dataset}. For the first time, we demonstrate the benefits of generating optical flow training data from large-scale real-world images, outperforming the most advanced unsupervised methods and supervised methods on synthetic datasets. Moreover, our models serve as a foundation model and enhance the performance of various downstream video tasks.
Abstract (translated)
光流估计是计算机视觉中的一个重要子领域，为视频任务提供了基础。然而，由于使用动画合成数据集进行训练，其在现实世界中的鲁棒性受到了限制。这导致了应用到实际场景时出现领域差距，并且阻碍了通过扩大数据集来提升性能的效果。为了应对这些挑战，我们提出了**Flow-Anything**，这是一个大型的数据生成框架，旨在从真实世界的单视图图像中学习光流估计。我们采用了两个有效的步骤来使数据量的扩展变得可行。首先，我们利用先进的单目深度估计网络将一幅单视图图像转换为3D表示形式，这让我们能够通过虚拟相机渲染出光流和新视角下的图像。其次，我们开发了独立于物体的体积渲染模块以及基于深度感知的修复模块来模拟3D表示中的动态对象。这两个步骤使我们能够从大规模的单视图图像中生成用于训练的真实数据集，即**FA-Flow 数据集**。 首次展示了从大规模真实世界图像中生成光流训练数据的好处，在无监督方法和合成数据集上的有监督方法方面均超越了最先进的技术水平。此外，我们的模型作为基础模型增强了各种下游视频任务的性能。
URL
https://arxiv.org/abs/2506.07740
PDF
-
Real-Time Execution of Action Chunking Flow Policies
2025-06-09 01:01:59Kevin Black, Manuel Y. Galliker, Sergey LevineAbstract
Modern AI systems, especially those interacting with the physical world, increasingly require real-time performance. However, the high latency of state-of-the-art generalist models, including recent vision-language action models (VLAs), poses a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. Our method, real-time chunking (RTC), is applicable to any diffusion- or flow-based VLA out of the box with no re-training. It generates the next action chunk while executing the current one, "freezing" actions guaranteed to execute and "inpainting" the rest. To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling high success rates in precise tasks $\unicode{x2013}$ such as lighting a match $\unicode{x2013}$ even in the presence of significant latency. See this https URL for videos.
Abstract (translated)
现代人工智能系统，特别是那些与物理世界互动的系统，越来越需要实时性能。然而，包括最近出现的视觉-语言-动作模型（VLAs）在内的最新通用型模型的高延迟带来了重大挑战。虽然动作分块已经使高频控制任务中实现了时间一致性，但它并未完全解决延迟问题，导致在分块边界处会出现暂停或不规则的动作。 本文介绍了一种新颖的推理时算法，该算法能够实现动作分块策略的平滑异步执行。我们的方法称为实时分块（RTC），可以在无需重新训练的情况下应用于任何基于扩散或流的VLA模型。该方法在执行当前动作的同时生成下一个动作片段，“冻结”那些确定要执行的动作，并“填充”剩余部分。 为了测试RTC，我们引入了一个新的基准，在Kinetix模拟器中包含了12项动态任务，以及评估了6项具有挑战性的现实世界双臂操作任务。实验结果表明，RTC速度较快、性能优良且独特地对推理延迟有强健性，显著提高了任务吞吐量，并在精确任务（如点燃火柴）中实现了高成功率——即使存在较大的延迟也能做到这一点。 有关本文介绍的视频，请参阅此链接：[此处应插入实际的URL或文档参考]
URL
https://arxiv.org/abs/2506.07339
PDF
-
Restereo: Diffusion stereo video generation and restoration
2025-06-06 12:14:24Xingchang Huang, Ashish Kumar Singh, Florian Dubost, Cristina Nader Vasconcelos, Sakar Khattar, Liang Shi, Christian Theobalt, Cengiz Oztireli, Gurprit SinghAbstract
Stereo video generation has been gaining increasing attention with recent advancements in video diffusion models. However, most existing methods focus on generating 3D stereoscopic videos from monocular 2D videos. These approaches typically assume that the input monocular video is of high quality, making the task primarily about inpainting occluded regions in the warped video while preserving disoccluded areas. In this paper, we introduce a new pipeline that not only generates stereo videos but also enhances both left-view and right-view videos consistently with a single model. Our approach achieves this by fine-tuning the model on degraded data for restoration, as well as conditioning the model on warped masks for consistent stereo generation. As a result, our method can be fine-tuned on a relatively small synthetic stereo video datasets and applied to low-quality real-world videos, performing both stereo video generation and restoration. Experiments demonstrate that our method outperforms existing approaches both qualitatively and quantitatively in stereo video generation from low-resolution inputs.
Abstract (translated)
最近在视频扩散模型方面的进展使得立体视频生成越来越受到关注。然而，大多数现有的方法主要集中在从单目二维视频生成三维立体视频上。这些方法通常假设输入的单目视频质量较高，并且任务主要是修复扭曲视频中被遮挡区域的同时保留新显露出来的区域。在这篇论文中，我们介绍了一种新的流水线技术，它不仅可以生成立体视频，还可以通过单一模型同时增强左视图和右视图视频的质量。我们的方法通过在退化数据上进行微调来进行恢复，并通过对扭曲掩码进行条件设置来实现一致的立体生成。因此，我们的方法可以使用相对较小的人工合成立体视频数据集进行微调，并应用于低质量的真实世界视频中，从而同时完成立体视频生成和修复的任务。实验结果表明，在从低分辨率输入生成立体视频方面，我们的方法在定性和定量评估上都优于现有的方法。
URL
https://arxiv.org/abs/2506.06023
PDF
-
Defurnishing with X-Ray Vision: Joint Removal of Furniture from Panoramas and Mesh
2025-06-05 17:59:30Alan Dolhasz, Chen Ma, Dave Gausebeck, Kevin Chen, Gregor Miller, Lucas Hayne, Gunnar Hovden, Azwad Sabik, Olaf Brandt, Mira SlavchevaAbstract
We present a pipeline for generating defurnished replicas of indoor spaces represented as textured meshes and corresponding multi-view panoramic images. To achieve this, we first segment and remove furniture from the mesh representation, extend planes, and fill holes, obtaining a simplified defurnished mesh (SDM). This SDM acts as an ``X-ray'' of the scene's underlying structure, guiding the defurnishing process. We extract Canny edges from depth and normal images rendered from the SDM. We then use these as a guide to remove the furniture from panorama images via ControlNet inpainting. This control signal ensures the availability of global geometric information that may be hidden from a particular panoramic view by the furniture being removed. The inpainted panoramas are used to texture the mesh. We show that our approach produces higher quality assets than methods that rely on neural radiance fields, which tend to produce blurry low-resolution images, or RGB-D inpainting, which is highly susceptible to hallucinations.
Abstract (translated)
我们提出了一种生成室内空间去家具化副本的流水线，这些室内空间以带纹理的网格和相应的多视角全景图像表示。为实现这一目标，我们首先从网格表示中分割并移除家具，扩展平面，并填充空洞，从而获得一个简化的无家具网格（SDM）。这个SDM充当场景底层结构的“X光片”，引导去家具化过程。我们从SDM渲染的深度和法线图像中提取Canny边缘。然后利用这些信息作为指导，通过ControlNet插值移除全景图中的家具。这一控制信号确保了全局几何信息的可用性，即便在特定全景视图中被移除的家具所遮挡的情况下也是如此。处理后的全景图用于为网格添加纹理。 我们展示了我们的方法相较于依赖神经辐射场的方法产生更高质量的资产，后者往往生成模糊且低分辨率的图像；同时也优于RGB-D插值方法，后者容易出现幻觉（即错误生成）。
URL
https://arxiv.org/abs/2506.05338
PDF
-
DSG-World: Learning a 3D Gaussian World Model from Dual State Videos
2025-06-05 16:33:32Wenhao Hu, Xuexiang Wen, Xi Li, Gaoang WangAbstract
Building an efficient and physically consistent world model from limited observations is a long standing challenge in vision and robotics. Many existing world modeling pipelines are based on implicit generative models, which are hard to train and often lack 3D or physical consistency. On the other hand, explicit 3D methods built from a single state often require multi-stage processing-such as segmentation, background completion, and inpainting-due to occlusions. To address this, we leverage two perturbed observations of the same scene under different object configurations. These dual states offer complementary visibility, alleviating occlusion issues during state transitions and enabling more stable and complete reconstruction. In this paper, we present DSG-World, a novel end-to-end framework that explicitly constructs a 3D Gaussian World model from Dual State observations. Our approach builds dual segmentation-aware Gaussian fields and enforces bidirectional photometric and semantic consistency. We further introduce a pseudo intermediate state for symmetric alignment and design collaborative co-pruning trategies to refine geometric completeness. DSG-World enables efficient real-to-simulation transfer purely in the explicit Gaussian representation space, supporting high-fidelity rendering and object-level scene manipulation without relying on dense observations or multi-stage pipelines. Extensive experiments demonstrate strong generalization to novel views and scene states, highlighting the effectiveness of our approach for real-world 3D reconstruction and simulation.
Abstract (translated)
从有限的观察构建一个高效且物理上一致的世界模型，一直是视觉和机器人学领域长期面临的挑战。许多现有的世界建模流程依赖于隐式生成模型，这些模型难以训练并且通常缺乏三维或物理上的一致性。另一方面，基于单一状态构建的显式3D方法由于遮挡问题往往需要多阶段处理——如分割、背景补全以及修复缺失信息等步骤。为了解决这些问题，我们利用了同一场景在不同对象配置下的两个扰动观察结果。这些双重状态提供了互补的可见性，缓解了状态转换期间的遮挡问题，并促进了更稳定和完整的重建。 在这篇论文中，我们提出了一种名为DSG-World的新颖端到端框架，该框架从双状态观测数据显式构造了一个三维高斯世界模型（3D Gaussian World Model）。我们的方法构建了双重分割感知高斯场，并强制执行双向光度学和语义一致性。此外，我们还引入了伪中间态用于对称校准，并设计了一种协作共修剪策略以优化几何完整性。 DSG-World能够在仅使用显式高斯表示空间的情况下实现高效的真实到模拟转换，支持高质量渲染以及对象级别的场景操作，而无需依赖密集观察数据或多阶段流程。大量的实验展示了其在新视角和场景状态下的强大泛化能力，突显了我们方法对于真实世界3D重建与模拟的有效性。
URL
https://arxiv.org/abs/2506.05217
PDF
-
OGGSplat: Open Gaussian Growing for Generalizable Reconstruction with Expanded Field-of-View
2025-06-05 16:17:18Yanbo Wang, Ziyi Wang, Wenzhao Zheng, Jie Zhou, Jiwen LuAbstract
Reconstructing semantic-aware 3D scenes from sparse views is a challenging yet essential research direction, driven by the demands of emerging applications such as virtual reality and embodied AI. Existing per-scene optimization methods require dense input views and incur high computational costs, while generalizable approaches often struggle to reconstruct regions outside the input view cone. In this paper, we propose OGGSplat, an open Gaussian growing method that expands the field-of-view in generalizable 3D reconstruction. Our key insight is that the semantic attributes of open Gaussians provide strong priors for image extrapolation, enabling both semantic consistency and visual plausibility. Specifically, once open Gaussians are initialized from sparse views, we introduce an RGB-semantic consistent inpainting module applied to selected rendered views. This module enforces bidirectional control between an image diffusion model and a semantic diffusion model. The inpainted regions are then lifted back into 3D space for efficient and progressive Gaussian parameter optimization. To evaluate our method, we establish a Gaussian Outpainting (GO) benchmark that assesses both semantic and generative quality of reconstructed open-vocabulary scenes. OGGSplat also demonstrates promising semantic-aware scene reconstruction capabilities when provided with two view images captured directly from a smartphone camera.
Abstract (translated)
从稀疏视角重建语义感知的3D场景是一项具有挑战性但又至关重要的研究方向，这一方向受到了虚拟现实和具身AI等新兴应用的需求推动。现有的逐场景优化方法需要密集的输入视图并且计算成本高昂，而通用化的方法则往往难以重构输入视锥之外的区域。在本文中，我们提出了OGGSplat（开放高斯增长法），这是一种用于广义3D重建中的视野扩展的开放方法。我们的关键见解是：开放高斯的语义属性为图像外推提供了强有力的先验信息，这使得在保持语义一致性和视觉合理性的同时能够进行有效重构。具体来说，在从稀疏视图初始化开放高斯后，我们引入了一个RGB-语义一致性修复模块应用于选定的渲染视图中。这个模块强制执行了图像扩散模型和语义扩散模型之间的双向控制机制。然后将修补后的区域重新投射回3D空间，并进行高效且渐进式的高斯参数优化。 为了评估我们的方法，我们建立了一个高斯外推（GO）基准测试，以衡量重建开放词汇场景的语义质量和生成质量。OGGSplat还展示了当提供两张从智能手机摄像头直接捕获的视图图像时，在具有感知意识场景重构方面表现出巨大的潜力。
URL
https://arxiv.org/abs/2506.05204
PDF
-
Geological Field Restoration through the Lens of Image Inpainting
2025-06-05 10:45:27Vladislav Trifonov, Ivan Oseledets, Ekaterina MuravlevaAbstract
We present a new viewpoint on a reconstructing multidimensional geological fields from sparse observations. Drawing inspiration from deterministic image inpainting techniques, we model a partially observed spatial field as a multidimensional tensor and recover missing values by enforcing a global low-rank structure. Our approach combines ideas from tensor completion and geostatistics, providing a robust optimization framework. Experiments on synthetic geological fields demonstrate that used tensor completion method significant improvements in reconstruction accuracy over ordinary kriging for various percent of observed data.
Abstract (translated)
我们提出了一种从稀疏观测数据重建多维地质场的新视角。借鉴确定性图像插值技术的灵感，我们将部分观察到的空间场建模为一个多维张量，并通过强制执行全局低秩结构来恢复缺失值。我们的方法结合了张量补全和地质统计学的思想，提供了一个稳健的优化框架。在合成地质领域的实验表明，所使用的张量补全方法相比于普通克里金插值法，在各种观测数据百分比下显著提高了重建准确性。
URL
https://arxiv.org/abs/2506.04869
PDF
-
MARS: Radio Map Super-resolution and Reconstruction Method under Sparse Channel Measurements
2025-06-05 07:03:50Chuyun Deng, Na Liu, Wei Xie, Lianming Xu, Li WangAbstract
Radio maps reflect the spatial distribution of signal strength and are essential for applications like smart cities, IoT, and wireless network planning. However, reconstructing accurate radio maps from sparse measurements remains challenging. Traditional interpolation and inpainting methods lack environmental awareness, while many deep learning approaches depend on detailed scene data, limiting generalization. To address this, we propose MARS, a Multi-scale Aware Radiomap Super-resolution method that combines CNNs and Transformers with multi-scale feature fusion and residual connections. MARS focuses on both global and local feature extraction, enhancing feature representation across different receptive fields and improving reconstruction accuracy. Experiments across different scenes and antenna locations show that MARS outperforms baseline models in both MSE and SSIM, while maintaining low computational cost, demonstrating strong practical potential.
Abstract (translated)
无线电地图反映了信号强度的空间分布，对于智慧城市、物联网和无线网络规划等应用至关重要。然而，从稀疏测量中重建准确的无线电地图仍然具有挑战性。传统插值和修复方法缺乏环境感知能力，而许多深度学习方法依赖于详细的场景数据，这限制了其泛化能力。为解决这些问题，我们提出了MARS（Multi-scale Aware Radiomap Super-resolution），这是一种结合CNNs（卷积神经网络）和Transformers，并采用多尺度特征融合与残差连接的方法。MARS注重全局和局部特征提取，增强了不同感受野内的特征表示，从而提高了重建的准确性。 跨不同场景和天线位置进行的实验表明，MARS在均方误差(MSE)和结构相似性(SSIM)指标上都优于基线模型，并且保持了较低的计算成本，展示了其强大的实用潜力。
URL
https://arxiv.org/abs/2506.04682
PDF
-
Follow-Your-Creation: Empowering 4D Creation through Video Inpainting
2025-06-05 03:11:48Yue Ma, Kunyu Feng, Xinhua Zhang, Hongyu Liu, David Junhao Zhang, Jinbo Xing, Yinhan Zhang, Ayden Yang, Zeyu Wang, Qifeng ChenAbstract
We introduce Follow-Your-Creation, a novel 4D video creation framework capable of both generating and editing 4D content from a single monocular video input. By leveraging a powerful video inpainting foundation model as a generative prior, we reformulate 4D video creation as a video inpainting task, enabling the model to fill in missing content caused by camera trajectory changes or user edits. To facilitate this, we generate composite masked inpainting video data to effectively fine-tune the model for 4D video generation. Given an input video and its associated camera trajectory, we first perform depth-based point cloud rendering to obtain invisibility masks that indicate the regions that should be completed. Simultaneously, editing masks are introduced to specify user-defined modifications, and these are combined with the invisibility masks to create a composite masks dataset. During training, we randomly sample different types of masks to construct diverse and challenging inpainting scenarios, enhancing the model's generalization and robustness in various 4D editing and generation tasks. To handle temporal consistency under large camera motion, we design a self-iterative tuning strategy that gradually increases the viewing angles during training, where the model is used to generate the next-stage training data after each fine-tuning iteration. Moreover, we introduce a temporal packaging module during inference to enhance generation quality. Our method effectively leverages the prior knowledge of the base model without degrading its original performance, enabling the generation of 4D videos with consistent multi-view coherence. In addition, our approach supports prompt-based content editing, demonstrating strong flexibility and significantly outperforming state-of-the-art methods in both quality and versatility.
Abstract (translated)
我们介绍了一种名为“Follow-Your-Creation”的新型4D视频创作框架，该框架能够从单目视频输入中生成和编辑4D内容。通过利用强大的视频修复基础模型作为生成先验条件，我们将4D视频的创建任务重新表述为一个视频修复任务，从而使模型能够在因相机轨迹变化或用户修改导致的内容缺失时进行填充。为了支持这一过程，我们生成了合成掩码修复视频数据集，以有效地微调模型用于4D视频生成。给定输入视频及其相关联的相机轨迹后，首先通过基于深度的点云渲染获得不可视性掩码，指示需要完成的区域。同时引入编辑掩码来指定用户定义的修改，并将这些与不可视性掩码结合以创建合成掩码数据集。在训练过程中，我们随机选择不同类型的掩码构建多样且具有挑战性的修复场景，从而增强模型在各种4D编辑和生成任务中的泛化能力和鲁棒性。 为了处理大范围相机运动下的时间一致性问题，我们设计了一种自我迭代调优策略，在训练期间逐步增加视角角度。每次微调迭代后，模型用于生成下一阶段的训练数据。此外，在推理过程中引入了时序打包模块以增强生成质量。我们的方法有效利用基础模型的知识而不损害其原始性能，从而生成具有多视图一致性的4D视频。此外，我们的方法支持基于提示的内容编辑，展示了强大的灵活性，并在质量和多功能性方面显著优于现有技术。 总的来说，“Follow-Your-Creation”框架不仅能够生成高质量的4D内容，还能灵活地进行用户定义的修改，适用于各种应用场景。
URL
https://arxiv.org/abs/2506.04590
PDF
-
OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulation
2025-06-02 15:42:06Sen Liang, Zhentao Yu, Zhengguang Zhou, Teng Hu, Hongmei Wang, Yi Chen, Qin Lin, Yuan Zhou, Xin Li, Qinglin Lu, Zhibo ChenAbstract
The emergence of Diffusion Transformers (DiT) has brought significant advancements to video generation, especially in text-to-video and image-to-video tasks. Although video generation is widely applied in various fields, most existing models are limited to single scenarios and cannot perform diverse video generation and editing through dynamic content manipulation. We propose OmniV2V, a video model capable of generating and editing videos across different scenarios based on various operations, including: object movement, object addition, mask-guided video edit, try-on, inpainting, outpainting, human animation, and controllable character video synthesis. We explore a unified dynamic content manipulation injection module, which effectively integrates the requirements of the above tasks. In addition, we design a visual-text instruction module based on LLaVA, enabling the model to effectively understand the correspondence between visual content and instructions. Furthermore, we build a comprehensive multi-task data processing system. Since there is data overlap among various tasks, this system can efficiently provide data augmentation. Using this system, we construct a multi-type, multi-scenario OmniV2V dataset and its corresponding OmniV2V-Test benchmark. Extensive experiments show that OmniV2V works as well as, and sometimes better than, the best existing open-source and commercial models for many video generation and editing tasks.
Abstract (translated)
Diffusion Transformers（DiT）的出现为视频生成领域带来了显著的进步，特别是在文本到视频和图像到视频的任务中。尽管视频生成技术在各个领域的应用广泛，但大多数现有的模型仅限于单一场景的应用，并且无法通过动态内容操作来进行多样化的视频生成与编辑。我们提出了OmniV2V，这是一种能够在不同场景下根据各种操作（如物体移动、对象添加、基于蒙版的视频编辑、试穿、修复、扩展、人类动画和可控角色视频合成）进行视频生成和编辑的模型。我们探索了一个统一的动态内容操纵注入模块，该模块能够有效整合上述任务的需求。此外，我们设计了一种基于LLaVA的视觉-文本指令模块，使模型能够有效地理解视觉内容与指令之间的对应关系。另外，我们构建了全面的多任务数据处理系统。由于各任务之间存在数据重叠，这一系统可以高效地提供数据增强。利用这套系统，我们建立了一个多类型、多场景的OmniV2V数据集及其对应的OmniV2V-Test基准测试集。大量的实验表明，对于许多视频生成和编辑的任务，OmniV2V的表现与当前最佳开源及商用模型相当甚至更优。
URL
https://arxiv.org/abs/2506.01801
PDF
-
Manipulating 3D Molecules in a Fixed-Dimensional SE-Equivariant Latent Space
2025-06-01 01:30:15Zitao Chen, Yinjun Jia, Zitong Tian, Wei-Ying Ma, Yanyan LanAbstract
Medicinal chemists often optimize drugs considering their 3D structures and designing structurally distinct molecules that retain key features, such as shapes, pharmacophores, or chemical properties. Previous deep learning approaches address this through supervised tasks like molecule inpainting or property-guided optimization. In this work, we propose a flexible zero-shot molecule manipulation method by navigating in a shared latent space of 3D molecules. We introduce a Variational AutoEncoder (VAE) for 3D molecules, named MolFLAE, which learns a fixed-dimensional, SE(3)-equivariant latent space independent of atom counts. MolFLAE encodes 3D molecules using an SE(3)-equivariant neural network into fixed number of latent nodes, distinguished by learned embeddings. The latent space is regularized, and molecular structures are reconstructed via a Bayesian Flow Network (BFN) conditioned on the encoder's latent output. MolFLAE achieves competitive performance on standard unconditional 3D molecule generation benchmarks. Moreover, the latent space of MolFLAE enables zero-shot molecule manipulation, including atom number editing, structure reconstruction, and coordinated latent interpolation for both structure and properties. We further demonstrate our approach on a drug optimization task for the human glucocorticoid receptor, generating molecules with improved hydrophilicity while preserving key interactions, under computational evaluations. These results highlight the flexibility, robustness, and real-world utility of our method, opening new avenues for molecule editing and optimization.
Abstract (translated)
药物化学家在优化药物时，常常会考虑其三维结构，并设计出具有特定关键特征（如形状、药效团或化学性质）的分子。以前的深度学习方法通过监督任务（例如分子修复或属性引导优化）来解决这个问题。在这项工作中，我们提出了一种灵活的零样本分子操作方法，该方法通过在3D分子共享潜在空间中导航实现。为此，我们引入了一个针对三维分子的变分自动编码器（VAE），名为MolFLAE。MolFLAE学习一个与原子计数无关的固定维度、SE(3)等变的潜在空间。 MolFLAE使用SE(3)-等变神经网络将三维分子编码为固定的潜在节点数量，这些节点通过学习嵌入进行区分。该潜在空间受到正则化处理，并且分子结构可以通过贝叶斯流网络（BFN）在给定编码器输出的情况下重建。MolFLAE在标准无条件的3D分子生成基准上实现了具有竞争力的表现。 此外，MolFLAE的潜在空间支持零样本分子操作，包括原子数编辑、结构重构和同时进行结构与属性调整的协调潜在插值。我们进一步展示了这种方法在人类糖皮质激素受体药物优化任务中的应用，通过计算评估证明生成的分子具有改善的亲水性，同时保持了关键相互作用。 这些结果突显了该方法的灵活性、鲁棒性和现实世界的实用性，并为分子编辑和优化开辟了新的途径。
URL
https://arxiv.org/abs/2506.00771
PDF
