Inpainting
-
Satellite-Free Training for Drone-View Geo-Localization
2026-04-02 03:48:53Tao Liu, Yingzhi Zhang, Kan Ren, Xiaoqi ZhaoAbstract
Drone-view geo-localization (DVGL) aims to determine the location of drones in GPS-denied environments by retrieving the corresponding geotagged satellite tile from a reference gallery given UAV observations of a location. In many existing formulations, these observations are represented by a single oblique UAV image. In contrast, our satellite-free setting is designed for multi-view UAV sequences, which are used to construct a geometry-normalized UAV-side location representation before cross-view retrieval. Existing approaches rely on satellite imagery during training, either through paired supervision or unsupervised alignment, which limits practical deployment when satellite data are unavailable or restricted. In this paper, we propose a satellite-free training (SFT) framework that converts drone imagery into cross-view compatible representations through three main stages: drone-side 3D scene reconstruction, geometry-based pseudo-orthophoto generation, and satellite-free feature aggregation for retrieval. Specifically, we first reconstruct dense 3D scenes from multi-view drone images using 3D Gaussian splatting and project the reconstructed geometry into pseudo-orthophotos via PCA-guided orthographic projection. This rendering stage operates directly on reconstructed scene geometry without requiring camera parameters at rendering time. Next, we refine these orthophotos with lightweight geometry-guided inpainting to obtain texture-complete drone-side views. Finally, we extract DINOv3 patch features from the generated orthophotos, learn a Fisher vector aggregation model solely from drone data, and reuse it at test time to encode satellite tiles for cross-view retrieval. Experimental results on University-1652 and SUES-200 show that our SFT framework substantially outperforms satellite-free generalization baselines and narrows the gap to methods trained with satellite imagery.
Abstract (translated)
无人机视角地理定位（DVGL）旨在通过从参考图库中检索对应的地理标记卫星瓦片，来确定无人机在GPS拒止环境中的位置，其输入是无人机对某地点的观测。在许多现有方案中，这些观测由单张倾斜无人机图像表示。相比之下，本文设计的无卫星场景适用于多视角无人机序列，用于在跨视角检索前构建几何归一化的无人机端位置表示。现有方法在训练阶段依赖卫星影像，无论是通过配对监督还是无监督对齐，这限制了卫星数据不可用或受限时的实际部署。本文提出一种无卫星训练（SFT）框架，通过三个主要阶段将无人机影像转换为跨视角兼容表示：无人机端三维场景重建、基于几何的伪正射影像生成，以及用于检索的无卫星特征聚合。具体而言，首先使用三维高斯泼溅（3D Gaussian splatting）从多视角无人机图像重建稠密三维场景，并通过PCA引导的正射投影将重建几何投影为伪正射影像。该渲染阶段直接基于重建的场景几何运行，无需渲染时的相机参数。随后，使用轻量级几何引导修复技术细化这些正射影像，以获得纹理完整的无人机端视图。最后，从生成的正射影像中提取DINOv3块特征，仅从无人机数据学习Fisher向量聚合模型，并在测试时复用该模型对卫星瓦片进行编码以实现跨视角检索。在University-1652和SUES-200数据集上的实验结果表明，所提SFT框架显著优于无卫星泛化基线，并缩小了与使用卫星影像训练的方法之间的性能差距。
URL
https://arxiv.org/abs/2604.01581
PDF
-
Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos
2026-03-30 22:08:46Yujin Ham, Junho Kim, Vivek Boominathan, Guha BalakrishnanAbstract
Egocentric "walking tour" videos provide a rich source of image data to develop rich and diverse visual models of environments around the world. However, the significant presence of humans in frames of these videos due to crowds and eye-level camera perspectives mitigates their usefulness in environment modeling applications. We focus on addressing this challenge by developing a generative algorithm that can realistically remove (i.e., inpaint) humans and their associated shadow effects from walking tour videos. Key to our approach is the construction of a rich semi-synthetic dataset of video clip pairs to train this generative model. Each pair in the dataset consists of an environment-only background clip, and a composite clip of walking humans with simulated shadows overlaid on the background. We randomly sourced both foreground and background components from real egocentric walking tour videos around the world to maintain visual diversity. We then used this dataset to fine-tune the state-of-the-art Casper video diffusion model for object and effects inpainting, and demonstrate that the resulting model performs far better than Casper both qualitatively and quantitatively at removing humans from walking tour clips with significant human presence and complex backgrounds. Finally, we show that the resulting generated clips can be used to build successful 3D/4D models of urban locations.
Abstract (translated)
以自我视角拍摄的"行走游览"视频为构建全球丰富多样的环境视觉模型提供了充足的图像数据。然而，由于人群聚集和摄像机处于人眼高度，这些视频画面中频繁出现的人物显著降低了其在环境建模应用中的效用。本研究聚焦于解决这一挑战，开发了一种能够逼真地从行走游览视频中移除人物及其伴随阴影效果的生成算法。该方法的关键在于构建了一个丰富的半合成视频片段配对数据集用于训练生成模型。数据集中的每对样本均包含一个仅含环境的背景片段，以及一个将行走人物与模拟阴影叠加在背景上的合成片段。我们随机从全球真实的第一人称行走游览视频中抽取前景和背景组件以保持视觉多样性。随后利用该数据集对最先进的Casper视频扩散模型进行微调，使其具备对象与效果重绘能力。实验表明，经微调的模型在移除高人物密度、复杂背景的行走游览片段时，其定性与定量表现均显著优于原始Casper模型。最后，我们证明所生成的视频片段可成功用于构建城市地点的三维/四维模型。
URL
https://arxiv.org/abs/2603.29036
PDF
-
TGIF2: Extended Text-Guided Inpainting Forgery Dataset & Benchmark
2026-03-30 15:59:16Hannes Mareen, Dimitrios Karageorgiou, Paschalis Giakoumoglou, Peter Lambert, Symeon Papadopoulos, Glenn Van WallendaelAbstract
Generative AI has made text-guided inpainting a powerful image editing tool, but at the same time a growing challenge for media forensics. Existing benchmarks, including our text-guided inpainting forgery (TGIF) dataset, show that image forgery localization (IFL) methods can localize manipulations in spliced images but struggle not in fully regenerated (FR) images, while synthetic image detection (SID) methods can detect fully regenerated images but cannot perform localization. With new generative inpainting models emerging and the open problem of localization in FR images remaining, updated datasets and benchmarks are needed. We introduce TGIF2, an extended version of TGIF, that captures recent advances in text-guided inpainting and enables a deeper analysis of forensic robustness. TGIF2 augments the original dataset with edits generated by FLUX.1 models, as well as with random non-semantic masks. Using the TGIF2 dataset, we conduct a forensic evaluation spanning IFL and SID, including fine-tuning IFL methods on FR images and generative super-resolution attacks. Our experiments show that both IFL and SID methods degrade on FLUX.1 manipulations, highlighting limited generalization. Additionally, while fine-tuning improves localization on FR images, evaluation with random non-semantic masks reveals object bias. Furthermore, generative super-resolution significantly weakens forensic traces, demonstrating that common image enhancement operations can undermine current forensic pipelines. In summary, TGIF2 provides an updated dataset and benchmark, which enables new insights into the challenges posed by modern inpainting and AI-based image enhancements. TGIF2 is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.28613
PDF
-
On the Usefulness of Diffusion-Based Room Impulse Response Interpolation to Microphone Array Processing
2026-03-30 09:30:54Sagi Della Torre, Mirco Pezzoli, Fabio Antonacci, Sharon GannotAbstract
Room Impulse Responses estimation is a fundamental problem in spatial audio processing and speech enhancement. In this paper, we build upon our previously introduced diffusion-based inpainting framework for Room Impulse Response interpolation and demonstrate its applicability to enhancing the performance of practical multi-microphone array processing tasks. Furthermore, we validate the robustness of this method in interpolating real-world Room Impulse Responses.
Abstract (translated)
房间脉冲响应估计是空间音频处理与语音增强领域的一个基础性问题。本文在我们先前提出的基于扩散的修复框架基础上，进一步构建了用于房间脉冲响应插值的方法，并展示了其在提升实际多麦克风阵列处理任务性能方面的应用潜力。此外，我们还验证了该方法在真实世界房间脉冲响应插值任务中的鲁棒性。
URL
https://arxiv.org/abs/2603.28209
PDF
-
Understanding Semantic Perturbations on In-Processing Generative Image Watermarks
2026-03-29 04:34:44Anirudh Nakra, Min WuAbstract
The widespread deployment of high-fidelity generative models has intensified the need for reliable mechanisms for provenance and content authentication. In-processing watermarking, embedding a signature into the generative model's synthesis procedure, has been advocated as a solution and is often reported to be robust to standard post-processing (such as geometric transforms and filtering). Yet robustness to semantic manipulations that alter high-level scene content while maintaining reasonable visual quality is not well studied or understood. We introduce a simple, multi-stage framework for systematically stress-testing in-processing generative watermarks under semantic drift. The framework utilizes off-the-shelf models for object detection, mask generation, and semantically guided inpainting or regeneration to produce controlled, meaning-altering edits with minimal perceptual degradation. Based on extensive experiments on representative schemes, we find that robustness varies significantly with the degree of semantic entanglement: methods by which watermarks remain detectable under a broad suite of conventional perturbations can fail under semantic edits, with watermark detectability in many cases dropping to near zero while image quality remains high. Overall, our results reveal a critical gap in current watermarking evaluations and suggest that watermark designs and benchmarking must explicitly account for robustness against semantic manipulation.
Abstract (translated)
高保真生成模型的广泛部署加剧了对可靠溯源与内容认证机制的需求。在处理过程中嵌入水印——即在水印生成模型的合成流程中嵌入签名——已被提倡作为一种解决方案，且常被报道对标准后处理（如几何变换和滤波）具有鲁棒性。然而，对于在保持合理视觉质量的同时改变高层场景内容的语义操控，其鲁棒性尚未得到充分研究与理解。我们提出一个简单的多阶段框架，用于在语义漂移条件下系统性地压力测试处理中生成水印。该框架利用现成的目标检测、掩码生成以及语义引导修复或重生成模型，以产生受控的、意义改变的编辑操作，同时最小化感知降质。基于对代表性方案的广泛实验，我们发现鲁棒性随语义纠缠程度显著变化：在多种常规扰动下仍可检测水印的方法，可能在语义编辑下失效，许多情况下水印可检测性降至近零，而图像质量仍保持较高水平。总体而言，我们的结果揭示了当前水印评估中的关键缺口，并表明水印设计与基准测试必须明确考虑对语义操控的鲁棒性。
URL
https://arxiv.org/abs/2603.27513
PDF
-
AVControl: Efficient Framework for Training Audio-Visual Controls
2026-03-25 20:06:43Matan Ben-Yosef, Tavi Halperin, Naomi Ken Korem, Mohammad Salama, Harel Cain, Asaf Joseph, Anthony Chen, Urska Jelercic, Ofir BibiAbstract
Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
Abstract (translated)
控制视频与音频生成需要多种模态，包括深度、姿态、摄像机轨迹和音频变换等。然而现有方法要么为固定控制集训练单一整体模型，要么每新增一种模态就需要代价高昂的架构调整。我们提出AVControl，一个基于联合音视频基础模型LTX-2构建的轻量可扩展框架，其中每种控制模态均作为独立LoRA在并行画布上训练，通过注意力层中的额外令牌提供参考信号，除LoRA适配器外无需任何架构改动。我们证明，简单将基于图像的上下文学习方法扩展到视频会导致结构控制失效，而我们的并行画布方法解决了这一问题。在VACE基准测试中，我们在深度与姿态引导生成、图像修复与扩展任务上超越所有评估的基线方法，并在摄像机控制与音视频基准测试中展现竞争力。我们的框架支持多种独立训练的模态：空间对齐控制（如深度、姿态、边缘）、带内参的摄像机轨迹、稀疏运动控制、视频编辑，以及据我们所知首个联合生成模型的模块化音视频控制。我们的方法兼具计算与数据效率：每种模态仅需小型数据集，并在数百至数千次训练步骤内收敛，仅为整体替代方案预算的一小部分。我们公开发布代码与训练好的LoRA检查点。
URL
https://arxiv.org/abs/2603.24793
PDF
-
InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting
2026-03-24 17:32:55Duc Vu, Kien Nguyen, Trong-Tung Nguyen, Ngan Nguyen, Phong Nguyen, Khoi Nguyen, Cuong Pham, Anh TranAbstract
Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization.
Abstract (translated)
近期基于扩散的模型在图像修复中已能实现照片级真实感，但需要大量采样步骤，限制了实际应用。少步数文生图模型虽能加速生成，但直接将其用于修复会导致背景与修复区域间融合不佳并产生伪影。我们追溯此问题根源在于随机高斯噪声初始化——在低函数评估次数下，这会导致语义错位并降低保真度。为克服此问题，我们提出InverFill，一种专为修复设计的一步反转方法，它能将输入掩码图像的语义信息注入初始噪声，从而实现高保真度的少步数修复。InverFill无需训练修复模型，而是通过语义对齐噪声作为输入，在混合采样流程中利用少步数文生图模型，显著改进了原始混合采样，甚至在低函数评估次数下能媲美专业修复模型。此外，InverFill无需真实图像监督，且仅增加极小的推理开销。大量实验表明，InverFill能持续提升基线少步数模型，改善图像质量与文本一致性，且无需代价高昂的重新训练或繁重的迭代优化。
URL
https://arxiv.org/abs/2603.23463
PDF
-
FontCrafter: High-Fidelity Element-Driven Artistic Font Creation with Visual In-Context Generation
2026-03-23 14:53:12Wuyang Luo, Chengkai Tan, Chang Ge, Binye Hong, Su Yang, Yongjiu MaAbstract
Artistic font generation aims to synthesize stylized glyphs based on a reference style. However, existing approaches suffer from limited style diversity and coarse control. In this work, we explore the potential of element-driven artistic font generation. Elements are the fundamental visual units of a font, serving as reference images for the desired style. Conceptually, we categorize elements into object elements (e.g., flowers or stones) with distinct structures and amorphous elements (e.g., flames or clouds) with unstructured textures. We introduce FontCrafter, an element-driven framework for font creation, and construct a large-scale dataset, ElementFont, which contains diverse element types and high-quality glyph images. However, achieving high-fidelity reconstruction of both texture and structure of reference elements remains challenging. To address this, we propose an in-context generation strategy that treats element images as visual context and uses an inpainting model to transfer element styles into glyph regions at the pixel level. To further control glyph shapes, we design a lightweight Context-aware Mask Adapter (CMA) that injects shape information. Moreover, a training-free attention redirection mechanism enables region-aware style control and suppresses stroke hallucination. In addition, edge repainting is applied to make boundaries more natural. Extensive experiments demonstrate that FontCrafter achieves strong zero-shot generation performance, particularly in preserving structural and textural fidelity, while also supporting flexible controls such as style mixture.
Abstract (translated)
艺术字体生成旨在基于参考风格合成风格化的字形。然而，现有方法存在风格多样性有限且控制粗糙的问题。本研究探索了元素驱动的艺术字体生成潜力。元素是字体的基础视觉单元，可作为期望风格的参考图像。概念上，我们将元素分为具有明确结构的实体元素（如花朵或石头）与具有非结构化纹理的无形元素（如火焰或云朵）。我们引入了FontCrafter——一个用于字体创作的元素驱动框架，并构建了包含多样化元素类型与高质量字形图像的大规模数据集ElementFont。然而，实现参考元素纹理与结构的高保真重建仍具挑战性。为此，我们提出一种上下文生成策略，将元素图像作为视觉上下文，并利用修复模型在像素层面将元素风格迁移至字形区域。为进一步控制字形形状，我们设计了轻量级的上下文感知掩模适配器（CMA）以注入形状信息。此外，一种无需训练的注意力重定向机制可实现区域感知的风格控制并抑制笔画幻觉。同时应用边缘重绘以使边界更自然。大量实验表明，FontCrafter实现了强大的零样本生成性能，尤其在保持结构与纹理保真度方面表现突出，并支持风格混合等灵活控制。
URL
https://arxiv.org/abs/2603.22054
PDF
-
SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing
2026-03-19 17:59:51Xinyao Zhang, Wenkai Dong, Yuxin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong WangAbstract
Current instruction-guided video editing models struggle to simultaneously balance precise semantic modifications with faithful motion preservation. While existing approaches rely on injecting explicit external priors (e.g., VLM features or structural conditions) to mitigate these issues, this reliance severely bottlenecks model robustness and generalization. To overcome this limitation, we present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling. First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames, enabling purely instruction-aware structural planning. Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks (cube inpainting, speed perturbation, and tube shuffle), enabling the model to internalize temporal dynamics directly from raw videos. SAMA is optimized with a two-stage pipeline: a factorized pre-training stage that learns inherent semantic-motion representations without paired video-instruction editing data, followed by supervised fine-tuning on paired editing data. Remarkably, the factorized pre-training alone already yields strong zero-shot video editing ability, validating the proposed factorization. SAMA achieves state-of-the-art performance among open-source models and is competitive with leading commercial systems (e.g., Kling-Omni). Code, models, and datasets will be released.
Abstract (translated)
URL
https://arxiv.org/abs/2603.19228
PDF
-
EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing
2026-03-19 17:59:22Yang Fu, Yike Zheng, Ziyun Dai, Henghui DingAbstract
Video object removal aims to eliminate dynamic target objects and their visual effects, such as deformation, shadows, and reflections, while restoring seamless backgrounds. Recent diffusion-based video inpainting and object removal methods can remove the objects but often struggle to erase these effects and to synthesize coherent backgrounds. Beyond method limitations, progress is further hampered by the lack of a comprehensive dataset that systematically captures common object effects across varied environments for training and evaluation. To address this, we introduce VOR (Video Object Removal), a large-scale dataset that provides diverse paired videos, each consisting of one video where the target object is present with its effects and a counterpart where the object and effects are absent, with corresponding object masks. VOR contains 60K high-quality video pairs from captured and synthetic sources, covers five effects types, and spans a wide range of object categories as well as complex, dynamic multi-object scenes. Building on VOR, we propose EffectErase, an effect-aware video object removal method that treats video object insertion as the inverse auxiliary task within a reciprocal learning scheme. The model includes task-aware region guidance that focuses learning on affected areas and enables flexible task switching. Then, an insertion-removal consistency objective that encourages complementary behaviors and shared localization of effect regions and structural cues. Trained on VOR, EffectErase achieves superior performance in extensive experiments, delivering high-quality video object effect erasing across diverse scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2603.19224
PDF
-
Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priors
2026-03-19 11:33:27Jiatong Xia, Zicheng Duan, Anton van den Hengel, Lingqiao LiuAbstract
Recent progress in 3D generation has been driven largely by models conditioned on images or text, while readily available 3D priors are still underused. In many real-world scenarios, the visible-region point cloud are easy to obtain from active sensors such as LiDAR or from feed-forward predictors like VGGT, offering explicit geometric constraints that current methods fail to exploit. In this work, we introduce Points-to-3D, a diffusion-based framework that leverages point cloud priors for geometry-controllable 3D asset and scene generation. Built on a latent 3D diffusion model TRELLIS, Points-to-3D first replaces pure-noise sparse structure latent initialization with a point cloud priors tailored input formulation.A structure inpainting network, trained within the TRELLIS framework on task-specific data designed to learn global structural inpainting, is then used for inference with a staged sampling strategy (structural inpainting followed by boundary refinement), completing the global geometry while preserving the visible regions of the input this http URL practice, Points-to-3D can take either accurate point-cloud priors or VGGT-estimated point clouds from single images as input. Experiments on both objects and scene scenarios consistently demonstrate superior performance over state-of-the-art baselines in terms of rendering quality and geometric fidelity, highlighting the effectiveness of explicitly embedding point-cloud priors for achieving more accurate and structurally controllable 3D generation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.18782
PDF
-
LGESynthNet: Controlled Scar Synthesis for Improved Scar Segmentation in Cardiac LGE-MRI Imaging
2026-03-18 23:40:33Athira J. Jacob, Puneet Sharma, Daniel RueckertAbstract
Segmentation of enhancement in LGE cardiac MRI is critical for diagnosing various ischemic and non-ischemic cardiomyopathies. However, creating pixel-level annotations for these images is challenging and labor-intensive, leading to limited availability of annotated data. Generative models, particularly diffusion models, offer promise for synthetic data generation, yet many rely on large training datasets and often struggle with fine-grained conditioning control, especially for small or localized features. We introduce LGESynthNet, a latent diffusion-based framework for controllable enhancement synthesis, enabling explicit control over size, location, and transmural extent. Formulated as inpainting using a ControlNet-based architecture, the model integrates: (a) a reward model for conditioning-specific supervision, (b) a captioning module for anatomically descriptive text prompts, and (c) a biomedical text encoder. Trained on just 429 images (79 patients), it produces realistic, anatomically coherent samples. A quality control filter selects outputs with high conditioning-fidelity, which when used for training augmentation, improve downstream segmentation and detection performance, by up-to 6 and 20 points respectively.
Abstract (translated)
URL
https://arxiv.org/abs/2603.18356
PDF
-
ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation
2026-03-18 15:04:57Dmitriy Rivkin, Parker Ewen, Lili Gao, Julian Ost, Stefanie Walz, Rasika Kangutkar, Mario Bijelic, Felix HeideAbstract
Recent video diffusion models achieve high-quality generation through recurrent frame processing where each frame generation depends on previous frames. However, this recurrent mechanism means that training such models in the pixel domain incurs prohibitive memory costs, as activations accumulate across the entire video sequence. This fundamental limitation also makes fine-tuning these models with pixel-wise losses computationally intractable for long or high-resolution videos. This paper introduces ChopGrad, a truncated backpropagation scheme for video decoding, limiting gradient computation to local frame windows while maintaining global consistency. We provide a theoretical analysis of this approximation and show that it enables efficient fine-tuning with frame-wise losses. ChopGrad reduces training memory from scaling linearly with the number of video frames (full backpropagation) to constant memory, and compares favorably to existing state-of-the-art video diffusion models across a suite of conditional video generation tasks with pixel-wise losses, including video super-resolution, video inpainting, video enhancement of neural-rendered scenes, and controlled driving video generation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17812
PDF
-
TAPESTRY: From Geometry to Appearance via Consistent Turntable Videos
2026-03-18 14:02:09Yan Zeng, Haoran Jiang, Kaixin Yao, Qixuan Zhang, Longwen Zhang, Lan Xu, Jingyi YuAbstract
Automatically generating photorealistic and self-consistent appearances for untextured 3D models is a critical challenge in digital content creation. The advancement of large-scale video generation models offers a natural approach: directly synthesizing 360-degree turntable videos (TTVs), which can serve not only as high-quality dynamic previews but also as an intermediate representation to drive texture synthesis and neural rendering. However, existing general-purpose video diffusion models struggle to maintain strict geometric consistency and appearance stability across the full range of views, making their outputs ill-suited for high-quality 3D reconstruction. To this end, we introduce TAPESTRY, a framework for generating high-fidelity TTVs conditioned on explicit 3D geometry. We reframe the 3D appearance generation task as a geometry-conditioned video diffusion problem: given a 3D mesh, we first render and encode multi-modal geometric features to constrain the video generation process with pixel-level precision, thereby enabling the creation of high-quality and consistent TTVs. Building upon this, we also design a method for downstream reconstruction tasks from the TTV input, featuring a multi-stage pipeline with 3D-Aware Inpainting. By rotating the model and performing a context-aware secondary generation, this pipeline effectively completes self-occluded regions to achieve full surface coverage. The videos generated by TAPESTRY are not only high-quality dynamic previews but also serve as a reliable, 3D-aware intermediate representation that can be seamlessly back-projected into UV textures or used to supervise neural rendering methods like 3DGS. This enables the automated creation of production-ready, complete 3D assets from untextured meshes. Experimental results demonstrate that our method outperforms existing approaches in both video consistency and final reconstruction quality.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17735
PDF
-
VisionNVS: Self-Supervised Inpainting for Novel View Synthesis under the Virtual-Shift Paradigm
2026-03-18 05:57:46Hongbo Lu, Liang Yao, Chenghao He, Fan Liu, Wenlong Liao, Tao He, Pai PengAbstract
A fundamental bottleneck in Novel View Synthesis (NVS) for autonomous driving is the inherent supervision gap on novel trajectories: models are tasked with synthesizing unseen views during inference, yet lack ground truth images for these shifted poses during training. In this paper, we propose VisionNVS, a camera-only framework that fundamentally reformulates view synthesis from an ill-posed extrapolation problem into a self-supervised inpainting task. By introducing a ``Virtual-Shift'' strategy, we use monocular depth proxies to simulate occlusion patterns and map them onto the original view. This paradigm shift allows the use of raw, recorded images as pixel-perfect supervision, effectively eliminating the domain gap inherent in previous approaches. Furthermore, we address spatial consistency through a Pseudo-3D Seam Synthesis strategy, which integrates visual data from adjacent cameras during training to explicitly model real-world photometric discrepancies and calibration errors. Experiments demonstrate that VisionNVS achieves superior geometric fidelity and visual quality compared to LiDAR-dependent baselines, offering a robust solution for scalable driving simulation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17382
PDF
-
Stereo World Model: Camera-Guided Stereo Video Generation
2026-03-18 05:42:22Yang-Tian Sun, Zehuan Huang, Yifan Niu, Lin Ma, Yan-Pei Cao, Yuewen Ma, Xiaojuan QiAbstract
We present StereoWorld, a camera-conditioned stereo world model that jointly learns appearance and binocular geometry for end-to-end stereo video this http URL monocular RGB or RGBD approaches, StereoWorld operates exclusively within the RGB modality, while simultaneously grounding geometry directly from disparity. To efficiently achieve consistent stereo generation, our approach introduces two key designs: (1) a unified camera-frame RoPE that augments latent tokens with camera-aware rotary positional encoding, enabling relative, view- and time-consistent conditioning while preserving pretrained video priors via a stable attention initialization; and (2) a stereo-aware attention decomposition that factors full 4D attention into 3D intra-view attention plus horizontal row attention, leveraging the epipolar prior to capture disparity-aligned correspondences with substantially lower compute. Across benchmarks, StereoWorld improves stereo consistency, disparity accuracy, and camera-motion fidelity over strong monocular-then-convert pipelines, achieving more than 3x faster generation with an additional 5% gain in viewpoint consistency. Beyond benchmarks, StereoWorld enables end-to-end binocular VR rendering without depth estimation or inpainting, enhances embodied policy learning through metric-scale depth grounding, and is compatible with long-video distillation for extended interactive stereo synthesis.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17375
PDF
-
Search2Motion: Training-Free Object-Level Motion Control via Attention-Consensus Search
2026-03-17 16:02:38Sainan Liu, Tz-Ying Wu, Hector A Valdez, Subarna TripathiAbstract
We present Search2Motion, a training-free framework for object-level motion editing in image-to-video generation. Unlike prior methods requiring trajectories, bounding boxes, masks, or motion fields, Search2Motion adopts target-frame-based control, leveraging first-last-frame motion priors to realize object relocation while preserving scene stability without fine-tuning. Reliable target-frame construction is achieved through semantic-guided object insertion and robust background inpainting. We further show that early-step self-attention maps predict object and camera dynamics, offering interpretable user feedback and motivating ACE-Seed (Attention Consensus for Early-step Seed selection), a lightweight search strategy that improves motion fidelity without look-ahead sampling or external evaluators. Noting that existing benchmarks conflate object and camera motion, we introduce S2M-DAVIS and S2M-OMB for stable-camera, object-only evaluation, alongside FLF2V-obj metrics that isolate object artifacts without requiring ground-truth trajectories. Search2Motion consistently outperforms baselines on FLF2V-obj and VBench.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16711
PDF
-
LICA: Layered Image Composition Annotations for Graphic Design Research
2026-03-17 03:43:13Elad Hirsch, Shubham Yadav, Mohit Garg, Purvanshi MehtaAbstract
We introduce LICA (Layered Image Composition Annotations), a large-scale dataset of 1,550,244 multi-layer graphic design compositions designed to advance structured understanding and generation of graphic layouts1. In addition to ren- dered PNG images, LICA represents each design as a hierarchical composition of typed components including text, image, vector, and group elements, each paired with rich per-element metadata such as spatial geometry, typographic attributes, opacity, and visibility. The dataset spans 20 design categories and 971,850 unique templates, providing broad coverage of real-world design structures. We further introduce graphic design video as a new and largely unexplored challenge for current vision-language models through 27,261 animated layouts annotated with per-component keyframes and motion parameters. Beyond scale, LICA establishes a new paradigm of research tasks for graphic design, enabling structured investiga- tions into problems such as layer-aware inpainting, structured layout generation, controlled design editing, and temporally-aware generative modeling. By repre- senting design as a system of compositional layers and relationships, the dataset supports research on models that operate directly on design structure rather than pixels alone.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16098
PDF
-
UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priors
2026-03-16 22:44:52Xiaoyan Cong, Zekun Li, Zhiyang Dou, Hongyu Li, Omid Taheri, Chuan Guo, Abhay Mittal, Sizhe An, Taku Komura, Wojciech Matusik, Michael J. Black, Srinath SridharAbstract
Large-scale foundation models (LFMs) have recently made impressive progress in text-to-motion generation by learning strong generative priors from massive 3D human motion datasets and paired text descriptions. However, how to effectively and efficiently leverage such single-purpose motion LFMs, i.e., text-to-motion synthesis, in more diverse cross-modal and in-context motion generation downstream tasks remains largely unclear. Prior work typically adapts pretrained generative priors to individual downstream tasks in a task-specific manner. In contrast, our goal is to unlock such priors to support a broad spectrum of downstream motion generation tasks within a single unified framework. To bridge this gap, we present UMO, a simple yet general unified formulation that casts diverse downstream tasks into compositions of atomic per-frame operations, enabling in-context adaptation to unlock the generative priors of pretrained DiT-based motion LFMs. Specifically, UMO introduces three learnable frame-level meta-operation embeddings to specify per-frame intent and employs lightweight temporal fusion to inject in-context cues into the pretrained backbone, with negligible runtime overhead compared to the base model. With this design, UMO finetunes the pretrained model, originally limited to text-to-motion generation, to support diverse previously unsupported tasks, including temporal inpainting, text-guided motion editing, text-serialized geometric constraints, and multi-identity reaction generation. Experiments demonstrate that UMO consistently outperforms task-specific and training-free baselines across a wide range of benchmarks, despite using a single unified model. Code and model will be publicly available. Project Page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.15975
PDF
-
MV2UV: Generating High-quality UV Texture Maps with Multiview Prompts
2026-03-16 15:42:00Zheng Zhang, Qinchuan Zhang, Yuteng Ye, Zhi Chen, Penglei Ji, Mengfei Li, Wenxiao Zhang, Yuan LiuAbstract
Generating high-quality textures for 3D assets is a challenging task. Existing multiview texture generation methods suffer from the multiview inconsistency and missing textures on unseen parts, while UV inpainting texture methods do not generalize well due to insufficient UV data and cannot well utilize 2D image diffusion priors. In this paper, we propose a new method called MV2UV that combines 2D generative priors from multiview generation and the inpainting ability of UV refinement to get high-quality texture maps. Our key idea is to adopt a UV space generative model that simultaneously inpaints unseen parts of multiview images while resolving the inconsistency of multiview images. Experiments show that our method enables a better texture generation quality than existing methods, especially in unseen occluded and multiview-inconsistent parts.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15436
PDF
