Inpainting
-
Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors
2025-08-11 18:11:42Mutian Tong, Rundi Wu, Changxi ZhengAbstract
Indoor lighting estimation from a single image or video remains a challenge due to its highly ill-posed nature, especially when the lighting condition of the scene varies spatially and temporally. We propose a method that estimates from an input video a continuous light field describing the spatiotemporally varying lighting of the scene. We leverage 2D diffusion priors for optimizing such light field represented as a MLP. To enable zero-shot generalization to in-the-wild scenes, we fine-tune a pre-trained image diffusion model to predict lighting at multiple locations by jointly inpainting multiple chrome balls as light probes. We evaluate our method on indoor lighting estimation from a single image or video and show superior performance over compared baselines. Most importantly, we highlight results on spatiotemporally consistent lighting estimation from in-the-wild videos, which is rarely demonstrated in previous works.
Abstract (translated)
从单张图像或视频中估计室内照明仍然是一项挑战，尤其是当场景的光照条件在空间和时间上变化时。我们提出了一种方法，可以从输入视频中估算出连续光场，描述场景中的时空光照变化。我们将二维扩散先验用于优化表示为多层感知机（MLP）的这种光场。为了实现对野外场景的零样本泛化，我们微调了一个预训练的图像扩散模型，通过联合绘制多个铬球作为照明探针来预测多个位置处的照明情况。 我们在单张图像或视频上的室内照明估计上评估了该方法，并显示其性能优于比较基准。最重要的是，我们强调了在野外视频中一致地估算时空照明的效果，这在以前的工作中很少被展示。
URL
https://arxiv.org/abs/2508.08384
PDF
-
S^2VG: 3D Stereoscopic and Spatial Video Generation via Denoising Frame Matrix
2025-08-11 14:50:03Peng Dai, Feitong Tan, Qiangeng Xu, Yihua Huang, David Futschik, Ruofei Du, Sean Fanello, Yinda Zhang, Xiaojuan QiAbstract
While video generation models excel at producing high-quality monocular videos, generating 3D stereoscopic and spatial videos for immersive applications remains an underexplored challenge. We present a pose-free and training-free method that leverages an off-the-shelf monocular video generation model to produce immersive 3D videos. Our approach first warps the generated monocular video into pre-defined camera viewpoints using estimated depth information, then applies a novel \textit{frame matrix} inpainting framework. This framework utilizes the original video generation model to synthesize missing content across different viewpoints and timestamps, ensuring spatial and temporal consistency without requiring additional model fine-tuning. Moreover, we develop a \dualupdate~scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. The resulting multi-view videos are then adapted into stereoscopic pairs or optimized into 4D Gaussians for spatial video synthesis. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, such as Sora, Lumiere, WALT, and Zeroscope. The experiments demonstrate that our method has a significant improvement over previous methods. Project page at: this https URL
Abstract (translated)
虽然视频生成模型在制作高质量的单目视频方面表现出色，但生成用于沉浸式应用的3D立体和空间视频仍是一个尚未充分探索的挑战。我们提出了一种无需姿势信息且无需训练的方法，该方法利用现成的单目视频生成模型来生产沉浸式的3D视频。我们的方法首先使用估计出的深度信息将生成的单目视频变形到预定义的摄像机视角中，然后应用一种新颖的“帧矩阵”插值框架。这一框架借助原始的视频生成模型在不同的视角和时间戳间合成缺失的内容，从而确保空间和时间的一致性，无需额外调整模型参数。此外，我们开发了一种\dualupdate（双更新）方案，通过减轻从遮挡区域传递到潜在空间中的负面影响来进一步提高视频插值的质量。最终的多视图视频随后被转换为立体对或优化成4D高斯分布，用于空间视频合成。我们在各种生成模型（如Sora、Lumiere、WALT和Zeroscope）生成的视频上进行了实验验证，结果表明我们提出的方法在效果上明显优于以往方法。 项目页面在此链接：[请将此URL替换为实际提供的链接]
URL
https://arxiv.org/abs/2508.08048
PDF
-
Improving Diagnostic Accuracy for Oral Cancer with inpainting Synthesis Lesions Generated Using Diffusion Models
2025-08-08 09:15:02Yong Oh Lee, JeeEun Kim, Jung Woo LeeAbstract
In oral cancer diagnostics, the limited availability of annotated datasets frequently constrains the performance of diagnostic models, particularly due to the variability and insufficiency of training data. To address these challenges, this study proposed a novel approach to enhance diagnostic accuracy by synthesizing realistic oral cancer lesions using an inpainting technique with a fine-tuned diffusion model. We compiled a comprehensive dataset from multiple sources, featuring a variety of oral cancer images. Our method generated synthetic lesions that exhibit a high degree of visual fidelity to actual lesions, thereby significantly enhancing the performance of diagnostic algorithms. The results show that our classification model achieved a diagnostic accuracy of 0.97 in differentiating between cancerous and non-cancerous tissues, while our detection model accurately identified lesion locations with 0.85 accuracy. This method validates the potential for synthetic image generation in medical diagnostics and paves the way for further research into extending these methods to other types of cancer diagnostics.
Abstract (translated)
在口腔癌诊断中，由于注释数据集的有限可用性和训练数据的变化性及不足，限制了诊断模型的表现。为解决这些问题，本研究提出了一种新方法，利用经过微调的扩散模型和图像修复技术生成逼真的口腔癌症病变图像，以提高诊断准确性。我们从多个来源编译了一个包含多种类型口腔癌图像的综合数据集。我们的方法生成的合成病变与实际病变具有高度视觉相似性，从而显著提高了诊断算法的表现。 实验结果显示，我们的分类模型在区分癌症和非癌症组织方面的准确率为0.97，而检测模型在识别病灶位置上的准确率达到了0.85。这一方法验证了使用合成图像生成技术在医学诊断中的潜力，并为将这些方法扩展应用于其他类型癌症的诊断研究铺平道路。
URL
https://arxiv.org/abs/2508.06151
PDF
-
Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attention
2025-08-08 06:40:59Shiyang Zhou, Haijin Zeng, Yunfan Lu, Yongyong Chen, Jie Liu, Jingyong SuAbstract
Event cameras like the Hybrid Event-based Vision Sensor (HybridEVS) camera capture brightness changes as asynchronous "events" instead of frames, offering advanced application on mobile photography. However, challenges arise from combining a Quad Bayer Color Filter Array (CFA) sensor with event pixels lacking color information, resulting in aliasing and artifacts on the demosaicing process before downstream application. Current methods struggle to address these issues, especially on resource-limited mobile devices. In response, we introduce \textbf{TSANet}, a lightweight \textbf{T}wo-stage network via \textbf{S}tate space augmented cross-\textbf{A}ttention, which can handle event pixels inpainting and demosaicing separately, leveraging the benefits of dividing complex tasks into manageable subtasks. Furthermore, we introduce a lightweight Cross-Swin State Block that uniquely utilizes positional prior for demosaicing and enhances global dependencies through the state space model with linear complexity. In summary, TSANet demonstrates excellent demosaicing performance on both simulated and real data of HybridEVS while maintaining a lightweight model, averaging better results than the previous state-of-the-art method DemosaicFormer across seven diverse datasets in both PSNR and SSIM, while respectively reducing parameter and computation costs by $1.86\times$ and $3.29\times$. Our approach presents new possibilities for efficient image demosaicing on mobile devices. Code is available in the supplementary materials.
Abstract (translated)
事件相机，如混合事件基视觉传感器（HybridEVS）相机，通过捕捉亮度变化作为异步的“事件”，而不是帧的方式工作，为移动摄影提供了高级应用。然而，将四重拜耳色彩滤镜阵列（CFA）传感器与缺乏颜色信息的事件像素结合使用时，会导致去马赛克处理前出现混叠和伪影的问题。现有的方法在资源受限的移动设备上难以解决这些问题。 为此，我们提出了一种轻量级的方法——\textbf{TSANet}，即通过状态空间增强跨注意力机制的两阶段网络（\textbf{T}wo-stage network via \textbf{S}tate space augmented cross-\textbf{A}ttention）。此方法能够分别处理事件像素的填充和去马赛克化，将复杂任务分解为可管理的小任务。此外，我们还引入了一种轻量级的交叉-斯温状态块，该块独特地利用了位置先验进行去马赛克，并通过具有线性复杂度的状态空间模型增强了全局依赖关系。 总之，TSANet 在 HybridEVS 的模拟和真实数据集上展示了卓越的去马赛克性能，并保持了一个轻量级的模型。在七个不同的数据集中，它比之前的最先进方法 DemosaicFormer 平均获得了更高的 PSNR 和 SSIM 结果，同时分别减少了 $1.86\times$ 参数成本和 $3.29\times$ 计算成本。我们的方法为移动设备上的高效图像去马赛克处理开辟了新的可能性。 代码可在补充材料中获取。
URL
https://arxiv.org/abs/2508.06058
PDF
-
GAP: Gaussianize Any Point Clouds with Text Guidance
2025-08-07 17:59:27Weiqi Zhang, Junsheng Zhou, Haotian Geng, Wenyuan Zhang, Yu-Shen LiuAbstract
3D Gaussian Splatting (3DGS) has demonstrated its advantages in achieving fast and high-quality rendering. As point clouds serve as a widely-used and easily accessible form of 3D representation, bridging the gap between point clouds and Gaussians becomes increasingly important. Recent studies have explored how to convert the colored points into Gaussians, but directly generating Gaussians from colorless 3D point clouds remains an unsolved challenge. In this paper, we propose GAP, a novel approach that gaussianizes raw point clouds into high-fidelity 3D Gaussians with text guidance. Our key idea is to design a multi-view optimization framework that leverages a depth-aware image diffusion model to synthesize consistent appearances across different viewpoints. To ensure geometric accuracy, we introduce a surface-anchoring mechanism that effectively constrains Gaussians to lie on the surfaces of 3D shapes during optimization. Furthermore, GAP incorporates a diffuse-based inpainting strategy that specifically targets at completing hard-to-observe regions. We evaluate GAP on the Point-to-Gaussian generation task across varying complexity levels, from synthetic point clouds to challenging real-world scans, and even large-scale scenes. Project Page: this https URL.
Abstract (translated)
3D高斯点绘（3DGS）已经展示了其实现快速且高质量渲染的优势。由于点云作为一种广泛应用且易于访问的三维表示形式，弥合点云与高斯分布之间的差距变得越来越重要。最近的研究探讨了如何将带颜色的点转换为高斯分布，但直接从无色的3D点云生成高斯分布仍然是一个未解决的问题。在本文中，我们提出了一种名为GAP的新方法，该方法可以利用文本引导将原始点云转化为高质量的三维高斯分布。我们的核心思想是设计一个多视角优化框架，该框架使用深度感知图像扩散模型来合成不同视图下的连贯外观。为了确保几何准确性，我们引入了表面锚定机制，在优化过程中有效限制高斯分布在3D形状表面上的位置。此外，GAP整合了一种基于扩散的修补策略，专门针对难以观察到的区域进行填充。 我们在不同程度复杂性上评估了GAP在点云至高斯分布生成任务上的表现，从合成的点云数据集到具有挑战性的现实世界扫描以及大规模场景都进行了测试。项目页面：[请在此处插入具体的URL链接]。
URL
https://arxiv.org/abs/2508.05631
PDF
-
CADD: Context aware disease deviations via restoration of brain images using normative conditional diffusion models
2025-08-05 15:59:19Ana Lawry Aguila, Ayodeji Ijishakin, Juan Eugenio Iglesias, Tomomi Takenaga, Yukihiro Nomura, Takeharu Yoshikawa, Osamu Abe, Shouhei HanaokaAbstract
Applying machine learning to real-world medical data, e.g. from hospital archives, has the potential to revolutionize disease detection in brain images. However, detecting pathology in such heterogeneous cohorts is a difficult challenge. Normative modeling, a form of unsupervised anomaly detection, offers a promising approach to studying such cohorts where the ``normal'' behavior is modeled and can be used at subject level to detect deviations relating to disease pathology. Diffusion models have emerged as powerful tools for anomaly detection due to their ability to capture complex data distributions and generate high-quality images. Their performance relies on image restoration; differences between the original and restored images highlight potential abnormalities. However, unlike normative models, these diffusion model approaches do not incorporate clinical information which provides important context to guide the disease detection process. Furthermore, standard approaches often poorly restore healthy regions, resulting in poor reconstructions and suboptimal detection performance. We present CADD, the first conditional diffusion model for normative modeling in 3D images. To guide the healthy restoration process, we propose a novel inference inpainting strategy which balances anomaly removal with retention of subject-specific features. Evaluated on three challenging datasets, including clinical scans, which may have lower contrast, thicker slices, and motion artifacts, CADD achieves state-of-the-art performance in detecting neurological abnormalities in heterogeneous cohorts.
Abstract (translated)
将机器学习应用于实际的医疗数据（例如医院档案）有可能彻底改变脑部图像中的疾病检测方式。然而，在这种异质群体中检测病理是一项艰巨的任务。规范建模，一种无监督异常检测的形式，提供了一种有前景的方法来研究这些群体，其中“正常”的行为被建模，并可用于在个体层面检测与疾病病理相关的偏差。扩散模型由于其捕捉复杂数据分布和生成高质量图像的能力而成为异常检测的强大工具。它们的性能依赖于图像恢复；原始图像与恢复后的图像之间的差异突出了潜在的异常情况。然而，与规范模型不同的是，这些基于扩散的方法不包含临床信息，这为指导疾病检测过程提供了重要的背景。 此外，标准方法通常较差地恢复健康区域，导致重建质量低下和检测性能不佳。我们提出了CADD（条件性扩散模型），这是首个用于3D图像中规范建模的条件性扩散模型。为了引导健康的恢复过程，我们提出了一种新颖的推理修补策略，该策略平衡了异常移除与保留个体特征之间的关系。在三个具有挑战性的数据集上进行了评估，包括可能具有较低对比度、较厚切片和运动伪影的临床扫描，CADD在异质群体中检测神经学异常方面达到了最佳性能。
URL
https://arxiv.org/abs/2508.03594
PDF
-
Quality-Aware Language-Conditioned Local Auto-Regressive Anomaly Synthesis and Detection
2025-08-05 15:07:32Long Qian, Bingke Zhu, Yingying Chen, Ming Tang, Jinqiao WangAbstract
Despite substantial progress in anomaly synthesis methods, existing diffusion-based and coarse inpainting pipelines commonly suffer from structural deficiencies such as micro-structural discontinuities, limited semantic controllability, and inefficient generation. To overcome these limitations, we introduce ARAS, a language-conditioned, auto-regressive anomaly synthesis approach that precisely injects local, text-specified defects into normal images via token-anchored latent editing. Leveraging a hard-gated auto-regressive operator and a training-free, context-preserving masked sampling kernel, ARAS significantly enhances defect realism, preserves fine-grained material textures, and provides continuous semantic control over synthesized anomalies. Integrated within our Quality-Aware Re-weighted Anomaly Detection (QARAD) framework, we further propose a dynamic weighting strategy that emphasizes high-quality synthetic samples by computing an image-text similarity score with a dual-encoder model. Extensive experiments across three benchmark datasets-MVTec AD, VisA, and BTAD, demonstrate that our QARAD outperforms SOTA methods in both image- and pixel-level anomaly detection tasks, achieving improved accuracy, robustness, and a 5 times synthesis speedup compared to diffusion-based alternatives. Our complete code and synthesized dataset will be publicly available.
Abstract (translated)
尽管在异常合成方法方面取得了重大进展，现有的基于扩散和粗略修复的流程通常会遇到结构缺陷，如微观结构不连续、语义控制有限以及生成效率低下等问题。为了克服这些限制，我们引入了ARAS（条件语言自回归异常合成法），这是一种通过标记锚定潜在编辑精确注入局部、文本指定缺陷到正常图像的方法。借助硬门控的自回归运算符和无训练需求、上下文保留的掩码采样内核，ARAS显著提升了缺陷的真实感，保持了细微材料纹理，并提供了对合成异常的连续语义控制。 在我们提出的质量感知再加权异常检测（QARAD）框架中，我们进一步提出了一种动态加权策略，该策略通过双编码器模型计算图像文本相似度得分来强调高质量的合成样本。我们在三个基准数据集——MVTec AD、VisA和BTAD上进行了广泛的实验，结果表明我们的QARAD方法在图像级和像素级异常检测任务中超越了现有的最佳方法（SOTA），实现了更高的准确性和鲁棒性，并且与基于扩散的方法相比，合成速度提升了五倍。我们将公开发布完整的代码和合成数据集。
URL
https://arxiv.org/abs/2508.03539
PDF
-
MILD: Multi-Layer Diffusion Strategy for Complex and Precise Multi-IP Aware Human Erasing
2025-08-05 13:56:24Jinghan Yu, Zhiyuan Ma, Yue Ma, Kaiqi Liu, Yuhan Wang, Jianjun LiAbstract
Recent years have witnessed the success of diffusion models in image-customized tasks. Prior works have achieved notable progress on human-oriented erasing using explicit mask guidance and semantic-aware inpainting. However, they struggle under complex multi-IP scenarios involving human-human occlusions, human-object entanglements, and background interferences. These challenges are mainly due to: 1) Dataset limitations, as existing datasets rarely cover dense occlusions, camouflaged backgrounds, and diverse interactions; 2) Lack of spatial decoupling, where foreground instances cannot be effectively disentangled, limiting clean background restoration. In this work, we introduce a high-quality multi-IP human erasing dataset with diverse pose variations and complex backgrounds. We then propose Multi-Layer Diffusion (MILD), a novel strategy that decomposes generation into semantically separated pathways for each instance and the background. To enhance human-centric understanding, we introduce Human Morphology Guidance, integrating pose, parsing, and spatial relations. We further present Spatially-Modulated Attention to better guide attention flow. Extensive experiments show that MILD outperforms state-of-the-art methods on challenging human erasing benchmarks.
Abstract (translated)
近年来，扩散模型在图像定制任务中取得了显著成功。此前的工作已经在使用显式遮罩引导和语义感知修复进行人像擦除方面取得了重要进展。然而，在涉及人物间遮挡、人与物体纠缠以及背景干扰的复杂多IP场景下（其中IP指涉及个体或多个独立实体的情况），这些方法仍面临挑战。这些问题主要源于：1）数据集限制，现有数据集中很少包含密集遮挡、伪装背景和多样化的互动情况；2）缺乏空间解耦能力，前景实例不能有效分离，从而限制了对干净背景的恢复。在这项工作中，我们引入了一个高质量的多IP人物擦除数据集，其中包含了多种姿势变化及复杂的背景。随后，我们提出了多层次扩散（MILD），这是一种新型策略，将生成过程分解为每个实例和背景的语义独立路径。为了增强以人类为中心的理解，我们还介绍了人体形态引导，整合了姿态、解析以及空间关系。此外，我们提出了一种空间调制注意力机制，以更好地指导注意力流。广泛的实验表明，在具有挑战性的人像擦除基准测试中，MILD优于现有的最先进技术。
URL
https://arxiv.org/abs/2508.06543
PDF
-
CogniPlan: Uncertainty-Guided Path Planning with Conditional Generative Layout Prediction
2025-08-05 03:12:02Yizhuo Wang, Haodong He, Jingsong Liang, Yuhong Cao, Ritabrata Chakraborty, Guillaume SartorettiAbstract
Path planning in unknown environments is a crucial yet inherently challenging capability for mobile robots, which primarily encompasses two coupled tasks: autonomous exploration and point-goal navigation. In both cases, the robot must perceive the environment, update its belief, and accurately estimate potential information gain on-the-fly to guide planning. In this work, we propose CogniPlan, a novel path planning framework that leverages multiple plausible layouts predicted by a COnditional GeNerative Inpainting model, mirroring how humans rely on cognitive maps during navigation. These predictions, based on the partially observed map and a set of layout conditioning vectors, enable our planner to reason effectively under uncertainty. We demonstrate strong synergy between generative image-based layout prediction and graph-attention-based path planning, allowing CogniPlan to combine the scalability of graph representations with the fidelity and predictiveness of occupancy maps, yielding notable performance gains in both exploration and navigation. We extensively evaluate CogniPlan on two datasets (hundreds of maps and realistic floor plans), consistently outperforming state-of-the-art planners. We further deploy it in a high-fidelity simulator and on hardware, showcasing its high-quality path planning and real-world applicability.
Abstract (translated)
在未知环境中进行路径规划是移动机器人的一项关键且具有挑战性的能力，主要包含两个相互关联的任务：自主探索和目标点导航。在这两种情况下，机器人都必须感知环境、更新其信念，并实时准确地估计潜在的信息收益以指导规划。 在此研究中，我们提出了一种新的路径规划框架——CogniPlan，该框架利用条件生成式图像修复模型预测出的多个可能布局，这类似于人类在导航过程中依赖认知地图的做法。这些基于部分观察的地图和一组布局约束向量的预测使我们的规划器能够在不确定性条件下有效推理。 本研究展示了生成式图像布局预测与图注意力路径规划之间存在强大的协同作用，使CogniPlan能够结合图表示法的可扩展性以及占用地图的精确度和预见能力，在探索和导航方面实现了显著性能提升。我们对两个数据集（数百张地图和现实场景中的平面图）进行了详尽评估，并且始终优于现有最先进技术。 此外，我们在高保真模拟器和硬件上部署了CogniPlan，展示了其高质量的路径规划能力和在真实世界的适用性。
URL
https://arxiv.org/abs/2508.03027
PDF
-
DreamPainter: Image Background Inpainting for E-commerce Scenarios
2025-08-04 07:54:37Sijie Zhao, Jing Cheng, Yaoyao Wu, Hao Xu, Shaohui JiaoAbstract
Although diffusion-based image genenation has been widely explored and applied, background generation tasks in e-commerce scenarios still face significant challenges. The first challenge is to ensure that the generated products are consistent with the given product inputs while maintaining a reasonable spatial arrangement, harmonious shadows, and reflections between foreground products and backgrounds. Existing inpainting methods fail to address this due to the lack of domain-specific data. The second challenge involves the limitation of relying solely on text prompts for image control, as effective integrating visual information to achieve precise control in inpainting tasks remains underexplored. To address these challenges, we introduce DreamEcom-400K, a high-quality e-commerce dataset containing accurate product instance masks, background reference images, text prompts, and aesthetically pleasing product images. Based on this dataset, we propose DreamPainter, a novel framework that not only utilizes text prompts for control but also flexibly incorporates reference image information as an additional control signal. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods, maintaining high product consistency while effectively integrating both text prompt and reference image information.
Abstract (translated)
尽管基于扩散模型的图像生成技术已经被广泛研究和应用，但在电子商务场景中的背景生成任务仍然面临重大挑战。第一个挑战是确保生成的产品与给定的产品输入一致，同时保持合理的空间布局、和谐的阴影以及前景产品和背景之间的反射效果。现有的修复算法（inpainting methods）由于缺乏特定领域的数据而无法解决这个问题。第二个挑战则是依赖于仅使用文本提示进行图像控制的局限性，有效地将视觉信息整合以实现精确的修复任务控制仍是一个未被充分探索的问题。 为了解决这些挑战，我们引入了DreamEcom-400K，这是一个高质量的电子商务数据集，其中包含准确的产品实例掩码、背景参考图片、文本提示以及美观的产品图像。基于这个数据集，我们提出了DreamPainter，一种新的框架，它不仅利用文本提示进行控制，还灵活地将参考图信息作为额外的控制信号集成进来。 广泛的实验表明，我们的方法在保持高产品一致性的同时显著优于最先进的方法，并且能够有效地整合文本提示和参考图片的信息。
URL
https://arxiv.org/abs/2508.02155
PDF
-
LRTuckerRep: Low-rank Tucker Representation Model for Multi-dimensional Data Completion
2025-08-04 06:57:50Wenwu Gong, Lili YangAbstract
Multi-dimensional data completion is a critical problem in computational sciences, particularly in domains such as computer vision, signal processing, and scientific computing. Existing methods typically leverage either global low-rank approximations or local smoothness regularization, but each suffers from notable limitations: low-rank methods are computationally expensive and may disrupt intrinsic data structures, while smoothness-based approaches often require extensive manual parameter tuning and exhibit poor generalization. In this paper, we propose a novel Low-Rank Tucker Representation (LRTuckerRep) model that unifies global and local prior modeling within a Tucker decomposition. Specifically, LRTuckerRep encodes low rankness through a self-adaptive weighted nuclear norm on the factor matrices and a sparse Tucker core, while capturing smoothness via a parameter-free Laplacian-based regularization on the factor spaces. To efficiently solve the resulting nonconvex optimization problem, we develop two iterative algorithms with provable convergence guarantees. Extensive experiments on multi-dimensional image inpainting and traffic data imputation demonstrate that LRTuckerRep achieves superior completion accuracy and robustness under high missing rates compared to baselines.
Abstract (translated)
多维数据补全在计算科学中是一个关键问题，尤其是在计算机视觉、信号处理和科学计算等领域。现有方法通常利用全局低秩近似或局部光滑性正则化，但每种方法都存在显著的局限性：低秩方法计算成本高且可能破坏内在的数据结构，而基于光滑性的方法往往需要大量的手动参数调整，并表现出较差的泛化能力。在本文中，我们提出了一种新颖的低秩Tucker表示（LRTuckerRep）模型，该模型在一个Tucker分解框架内统一了全局和局部先验建模。具体来说，LRTuckerRep通过因子矩阵上的自适应加权核范数以及稀疏的Tucker核心来编码低秩性，同时使用基于拉普拉斯算子的无参数正则化来捕捉光滑性。为了有效地解决由此产生的非凸优化问题，我们开发了两个迭代算法，并证明了它们具有收敛保证。在多维图像修复和交通数据插补上的广泛实验表明，在高缺失率情况下，LRTuckerRep比基准方法表现出更优越的完成准确性和鲁棒性。
URL
https://arxiv.org/abs/2508.03755
PDF
-
Generating Synthetic Invoices via Layout-Preserving Content Replacement
2025-08-04 06:19:34Bevin V, Ananthakrishnan P V, Ragesh KR, Sanjay M, Vineeth S, Bibin WilsonAbstract
The performance of machine learning models for automated invoice processing is critically dependent on large-scale, diverse datasets. However, the acquisition of such datasets is often constrained by privacy regulations and the high cost of manual annotation. To address this, we present a novel pipeline for generating high-fidelity, synthetic invoice documents and their corresponding structured data. Our method first utilizes Optical Character Recognition (OCR) to extract the text content and precise spatial layout from a source invoice. Select data fields are then replaced with contextually realistic, synthetic content generated by a large language model (LLM). Finally, we employ an inpainting technique to erase the original text from the image and render the new, synthetic text in its place, preserving the exact layout and font characteristics. This process yields a pair of outputs: a visually realistic new invoice image and a perfectly aligned structured data file (JSON) reflecting the synthetic content. Our approach provides a scalable and automated solution to amplify small, private datasets, enabling the creation of large, varied corpora for training more robust and accurate document intelligence models.
Abstract (translated)
机器学习模型在自动化发票处理中的性能严重依赖于大规模、多样化的数据集。然而，由于隐私法规的限制和人工标注高昂的成本，获取这样的数据集变得非常困难。为了解决这个问题，我们提出了一种新颖的数据生成流程，用于创建高保真度的合成发票文档及其对应的结构化数据。 我们的方法首先利用光学字符识别（OCR）技术从原始发票中提取文本内容和精确的空间布局信息。然后，选择性地用大型语言模型（LLM）生成的情境真实性的合成内容替换部分数据字段。最后，我们使用一种图像修复技术去除原图中的文字，并在相同的位置渲染新的合成文字，保留了原有的版式和字体特征。 这一过程会产生一对输出：一张视觉上逼真的新发票图片以及一个与其合成内容完全对齐的结构化数据文件（JSON）。我们的方法提供了一种可扩展且自动化的解决方案来放大小规模、私有的数据集，从而能够创建大规模、多样化的语料库，用于训练更加强大和准确的文档智能模型。
URL
https://arxiv.org/abs/2508.03754
PDF
-
Trans-Adapter: A Plug-and-Play Framework for Transparent Image Inpainting
2025-08-01 22:27:21Yuekun Dai, Haitian Li, Shangchen Zhou, Chen Change LoyAbstract
RGBA images, with the additional alpha channel, are crucial for any application that needs blending, masking, or transparency effects, making them more versatile than standard RGB images. Nevertheless, existing image inpainting methods are designed exclusively for RGB images. Conventional approaches to transparent image inpainting typically involve placing a background underneath RGBA images and employing a two-stage process: image inpainting followed by image matting. This pipeline, however, struggles to preserve transparency consistency in edited regions, and matting can introduce jagged edges along transparency boundaries. To address these challenges, we propose Trans-Adapter, a plug-and-play adapter that enables diffusion-based inpainting models to process transparent images directly. Trans-Adapter also supports controllable editing via ControlNet and can be seamlessly integrated into various community models. To evaluate our method, we introduce LayerBench, along with a novel non-reference alpha edge quality evaluation metric for assessing transparency edge quality. We conduct extensive experiments on LayerBench to demonstrate the effectiveness of our approach.
Abstract (translated)
RGBA图像，由于其额外的alpha通道，在需要混合、遮罩或透明度效果的应用中至关重要。这使得它们比标准RGB图像更具灵活性和适用性。然而，现有的图像修复方法（即图像插值技术）主要是针对RGB图像设计的，并未专门考虑包含透明信息的RGBA图像。 对于处理含有透明度的图像时，传统的做法通常是将背景图放置在RGBA图像下方，并采用两阶段流程：首先是图像修复（inpainting），其次是图像遮罩(matting)。这种方案的问题在于它难以保持编辑区域中透明度的一致性，并且可能在透明边界处引入锯齿状边缘。 为了解决这些问题，我们提出了Trans-Adapter——一种即插即用的适配器，它可以使得基于扩散模型（diffusion-based models）的图像修复技术能够直接处理包含透明信息的RGBA图像。此外，Trans-Adapter还支持通过ControlNet进行可控编辑，并且可以无缝集成到各种社区模型中。 为了评估我们提出的方法的有效性，我们引入了LayerBench这一全新的基准测试框架以及一种新颖的非参考alpha边缘质量评价指标来专门衡量透明度边缘的质量水平。我们在LayerBench上进行了广泛实验，以证明我们的方法在处理包含透明信息的图像修复任务中的优越性能。
URL
https://arxiv.org/abs/2508.01098
PDF
-
Diffusion-Based User-Guided Data Augmentation for Coronary Stenosis Detection
2025-08-01 08:52:43Sumin Seo, In Kyu Lee, Hyun-Woo Kim, Jaesik Min, Chung-Hwan JungAbstract
Coronary stenosis is a major risk factor for ischemic heart events leading to increased mortality, and medical treatments for this condition require meticulous, labor-intensive analysis. Coronary angiography provides critical visual cues for assessing stenosis, supporting clinicians in making informed decisions for diagnosis and treatment. Recent advances in deep learning have shown great potential for automated localization and severity measurement of stenosis. In real-world scenarios, however, the success of these competent approaches is often hindered by challenges such as limited labeled data and class imbalance. In this study, we propose a novel data augmentation approach that uses an inpainting method based on a diffusion model to generate realistic lesions, allowing user-guided control of severity. Extensive evaluation on lesion detection and severity classification across various synthetic dataset sizes shows superior performance of our method on both a large-scale in-house dataset and a public coronary angiography dataset. Furthermore, our approach maintains high detection and classification performance even when trained with limited data, highlighting its clinical importance in improving the assessment of severity of stenosis and optimizing data utilization for more reliable decision support.
Abstract (translated)
冠状动脉狭窄是导致缺血性心脏病事件并增加死亡率的主要风险因素，治疗这种状况的医疗方法需要仔细且耗时的分析。冠状动脉造影为评估狭窄提供了关键的视觉线索，支持临床医生做出明智的诊断和治疗决策。近年来，深度学习的进步在自动定位和测量狭窄程度方面显示出巨大的潜力。然而，在实际应用中，由于标注数据有限及类别不平衡等问题，这些方法的成功往往受到阻碍。 在此研究中，我们提出了一种新的数据增强方法，该方法利用基于扩散模型的修复技术生成逼真的病变，并允许用户根据严重程度进行引导控制。在不同合成数据集大小下的病灶检测和严重度分类广泛评估表明，在大规模内部数据集以及公开冠状动脉造影数据集中，我们的方法表现出了优越性能。此外，即使使用有限的数据训练时，我们这种方法仍能保持高水平的检出率和分类精度，突显其在改善狭窄严重程度评估中的临床重要性，并优化数据利用以提供更可靠的决策支持。
URL
https://arxiv.org/abs/2508.00438
PDF
-
Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpainting
2025-08-01 08:33:45Seunggeun Chi, Enna Sachdeva, Pin-Hao Huang, Kwonjoon LeeAbstract
Amodal completion, which is the process of inferring the full appearance of objects despite partial occlusions, is crucial for understanding complex human-object interactions (HOI) in computer vision and robotics. Existing methods, such as those that use pre-trained diffusion models, often struggle to generate plausible completions in dynamic scenarios because they have a limited understanding of HOI. To solve this problem, we've developed a new approach that uses physical prior knowledge along with a specialized multi-regional inpainting technique designed for HOI. By incorporating physical constraints from human topology and contact information, we define two distinct regions: the primary region, where occluded object parts are most likely to be, and the secondary region, where occlusions are less probable. Our multi-regional inpainting method uses customized denoising strategies across these regions within a diffusion model. This improves the accuracy and realism of the generated completions in both their shape and visual detail. Our experimental results show that our approach significantly outperforms existing methods in HOI scenarios, moving machine perception closer to a more human-like understanding of dynamic environments. We also show that our pipeline is robust even without ground-truth contact annotations, which broadens its applicability to tasks like 3D reconstruction and novel view/pose synthesis.
Abstract (translated)
模态完成（amodal completion）是指即使面对物体的部分遮挡，也能推断出其完整外观的过程。这一过程对于理解计算机视觉和机器人领域中复杂的“人与物互动”（human-object interaction, HOI）至关重要。现有的方法，例如使用预训练扩散模型的方法，在处理动态场景时往往难以生成可信的完成图，因为它们对HOI的理解有限。为了解决这个问题，我们开发了一种新的方法，该方法结合了物理先验知识，并设计了一种专门用于HOI的多区域修复技术。 通过引入人体拓扑和接触信息的物理约束条件，我们将需要修复的区域划分为两个不同的部分：主要区域，在这里被遮挡的部分最有可能出现；次要区域，这里的遮挡可能性较小。我们的多区域修复方法在扩散模型中针对这两个区域内采用定制化的降噪策略，从而提升生成完成图的准确性和真实感，无论是形状还是视觉细节。 实验结果显示，相比现有的方法，我们在处理HOI场景时的方法有显著优势，使机器感知向更加接近人类对动态环境的理解迈进了一步。此外，我们还证明了即使在没有地面实况接触标注的情况下，我们的管道依然具有稳健性，这扩展了其应用于三维重建和新视图/姿态合成等任务的可能性。
URL
https://arxiv.org/abs/2508.00427
PDF
-
IN2OUT: Fine-Tuning Video Inpainting Model for Video Outpainting Using Hierarchical Discriminator
2025-08-01 08:15:14Sangwoo Youn, Minji Lee, Nokap Tony Park, Yeonggyoo Jeon, Taeyoung NaAbstract
Video outpainting presents a unique challenge of extending the borders while maintaining consistency with the given content. In this paper, we suggest the use of video inpainting models that excel in object flow learning and reconstruction in outpainting rather than solely generating the background as in existing methods. However, directly applying or fine-tuning inpainting models to outpainting has shown to be ineffective, often leading to blurry results. Our extensive experiments on discriminator designs reveal that a critical component missing in the outpainting fine-tuning process is a discriminator capable of effectively assessing the perceptual quality of the extended areas. To tackle this limitation, we differentiate the objectives of adversarial training into global and local goals and introduce a hierarchical discriminator that meets both objectives. Additionally, we develop a specialized outpainting loss function that leverages both local and global features of the discriminator. Fine-tuning on this adversarial loss function enhances the generator's ability to produce both visually appealing and globally coherent outpainted scenes. Our proposed method outperforms state-of-the-art methods both quantitatively and qualitatively. Supplementary materials including the demo video and the code are available in SigPort.
Abstract (translated)
视频扩展（outpainting）在保持与给定内容一致性的同时，具有扩展边界的独特挑战。本文建议采用擅长于对象流动学习和重构的视频修复模型来应对这一问题，而不是像现有方法那样仅生成背景部分。然而，直接应用或微调这些修复模型到扩展任务中被证明是无效的，并且经常导致模糊的结果。我们通过大量关于判别器设计的实验发现，在微调过程中缺失的关键组件是一个能够有效评估扩展区域感知质量的判别器。 为了解决这一局限性，我们将对抗训练的目标分为全局和局部目标，并引入了一个既能满足全局又能满足局部目标的分层判别器。此外，我们开发了一种专门针对视频扩展任务的损失函数，该函数利用了判别器的本地和全球特征。在这一对抗损失函数上的微调提升了生成器的能力，使其能够产生既美观又具有全局一致性的视频扩展场景。 我们的方法在定量和定性评估中都优于现有的最先进的技术。补充材料包括演示视频和代码可在SigPort上获取。
URL
https://arxiv.org/abs/2508.00418
PDF
-
Controllable Pedestrian Video Editing for Multi-View Driving Scenarios via Motion Sequence
2025-08-01 03:56:57Danzhen Fu, Jiagao Hu, Daiguo Zhou, Fei Wang, Zepeng Wang, Wenhua LiaoAbstract
Pedestrian detection models in autonomous driving systems often lack robustness due to insufficient representation of dangerous pedestrian scenarios in training datasets. To address this limitation, we present a novel framework for controllable pedestrian video editing in multi-view driving scenarios by integrating video inpainting and human motion control techniques. Our approach begins by identifying pedestrian regions of interest across multiple camera views, expanding detection bounding boxes with a fixed ratio, and resizing and stitching these regions into a unified canvas while preserving cross-view spatial relationships. A binary mask is then applied to designate the editable area, within which pedestrian editing is guided by pose sequence control conditions. This enables flexible editing functionalities, including pedestrian insertion, replacement, and removal. Extensive experiments demonstrate that our framework achieves high-quality pedestrian editing with strong visual realism, spatiotemporal coherence, and cross-view consistency. These results establish the proposed method as a robust and versatile solution for multi-view pedestrian video generation, with broad potential for applications in data augmentation and scenario simulation in autonomous driving.
Abstract (translated)
在自主驾驶系统中，行人检测模型由于训练数据集中危险行人性状表示不足而往往缺乏鲁棒性。为了解决这一局限性，我们提出了一种结合视频修复和人体运动控制技术的新型多视角驾驶场景可控行人视频编辑框架。我们的方法首先通过多个摄像头视图来识别行人的感兴趣区域，并以固定的比率扩展检测边界框。然后将这些区域调整大小并拼接到一个统一的画布上，同时保留跨视图的空间关系。接着应用二值掩码来指定可编辑的区域，在此区域内行人编辑则由姿态序列控制条件指导。这使我们能够实现包括行人的插入、替换和移除在内的灵活编辑功能。广泛的实验表明，我们的框架在生成具有高质量行人编辑效果的同时，还具备强烈的视觉真实感、时空一致性和跨视图一致性。这些结果证明了所提出的方法作为多视角行人视频生成的稳健且多功能解决方案的有效性，并展示了它在数据增强和场景模拟方面的广泛应用潜力，在自主驾驶领域尤为突出。
URL
https://arxiv.org/abs/2508.00299
PDF
-
I2V-GS: Infrastructure-to-Vehicle View Transformation with Gaussian Splatting for Autonomous Driving Data Generation
2025-07-31 15:59:16Jialei Chen, Wuhao Xu, Sipeng He, Baoru Huang, Dongchun RenAbstract
Vast and high-quality data are essential for end-to-end autonomous driving systems. However, current driving data is mainly collected by vehicles, which is expensive and inefficient. A potential solution lies in synthesizing data from real-world images. Recent advancements in 3D reconstruction demonstrate photorealistic novel view synthesis, highlighting the potential of generating driving data from images captured on the road. This paper introduces a novel method, I2V-GS, to transfer the Infrastructure view To the Vehicle view with Gaussian Splatting. Reconstruction from sparse infrastructure viewpoints and rendering under large view transformations is a challenging problem. We adopt the adaptive depth warp to generate dense training views. To further expand the range of views, we employ a cascade strategy to inpaint warped images, which also ensures inpainting content is consistent across views. To further ensure the reliability of the diffusion model, we utilize the cross-view information to perform a confidenceguided optimization. Moreover, we introduce RoadSight, a multi-modality, multi-view dataset from real scenarios in infrastructure views. To our knowledge, I2V-GS is the first framework to generate autonomous driving datasets with infrastructure-vehicle view transformation. Experimental results demonstrate that I2V-GS significantly improves synthesis quality under vehicle view, outperforming StreetGaussian in NTA-Iou, NTL-Iou, and FID by 45.7%, 34.2%, and 14.9%, respectively.
Abstract (translated)
大规模且高质量的数据对于端到端的自动驾驶系统至关重要。然而，目前驾驶数据主要通过车辆采集，这种方式既昂贵又效率低下。一种潜在的解决方案是从现实世界的图像中合成数据。最近在三维重建方面的进展展示了能够生成逼真的新视角图像，这凸显了从道路上捕捉的真实图像生成驾驶数据的潜力。本文介绍了一种新的方法——I2V-GS（Infrastructure to Vehicle Gaussian Splatting），该方法能将基础设施视图转换为车辆视图，并利用高斯点阵技术进行细节处理。 从稀疏的基础设施视角重建场景并在大视角变化下渲染是一个极具挑战性的问题。为此，我们采用自适应深度变形生成密集训练图像。为了进一步扩大可观察角度范围，我们采用了级联策略对变换后的图像进行修补，同时也确保不同视图之间的内容一致性和连贯性。 此外，为增强扩散模型的可靠性，我们利用跨视角信息进行了基于信心指导的优化。同时，我们还推出了RoadSight数据集，这是一个多模态、多种视角的真实场景基础设施视图数据集。 据我们所知，I2V-GS是首个能够通过基础设施到车辆视角转换生成自动驾驶数据集的框架。实验结果表明，在车辆视角下，I2V-GS在NTA-Iou（新颖视角下的IoU）、NTL-Iou（新任务下的IoU）以及FID（Frechet Inception Distance）等评价指标上显著提高了合成质量，分别优于StreetGaussian方法45.7%、34.2%和14.9%。
URL
https://arxiv.org/abs/2507.23683
PDF
-
Online Estimation of Table-Top Grown Strawberry Mass in Field Conditions with Occlusions
2025-07-31 12:10:23Jinshan Zhen, Yuanyue Ge, Tianxiao Zhu, Hui Zhao, Ya XiongAbstract
Accurate mass estimation of table-top grown strawberries under field conditions remains challenging due to frequent occlusions and pose variations. This study proposes a vision-based pipeline integrating RGB-D sensing and deep learning to enable non-destructive, real-time and online mass estimation. The method employed YOLOv8-Seg for instance segmentation, Cycle-consistent generative adversarial network (CycleGAN) for occluded region completion, and tilt-angle correction to refine frontal projection area calculations. A polynomial regression model then mapped the geometric features to mass. Experiments demonstrated mean mass estimation errors of 8.11% for isolated strawberries and 10.47% for occluded cases. CycleGAN outperformed large mask inpainting (LaMa) model in occlusion recovery, achieving superior pixel area ratios (PAR) (mean: 0.978 vs. 1.112) and higher intersection over union (IoU) scores (92.3% vs. 47.7% in the [0.9-1] range). This approach addresses critical limitations of traditional methods, offering a robust solution for automated harvesting and yield monitoring with complex occlusion patterns.
Abstract (translated)
在田间条件下，准确估算桌面栽培的草莓的质量仍然具有挑战性，主要是由于频繁遮挡和姿态变化。本研究提出了一种基于视觉的流水线方法，该方法结合了RGB-D传感技术和深度学习技术，以实现非破坏性的、实时在线质量估计。 所采用的方法包括使用YOLOv8-Seg进行实例分割，使用循环一致生成对抗网络（CycleGAN）完成遮挡区域，并通过调整倾斜角度来修正正面投影面积计算。然后，多项式回归模型将几何特征映射到质量上。 实验表明，在孤立草莓的情况下，平均质量估计误差为8.11%，在有遮挡的情况下的误差为10.47%。CycleGAN在遮挡恢复方面优于大型掩码插补（LaMa）模型，实现了更优的像素面积比（PAR）（均值：0.978 vs 1.112）和更高的交并比（IoU）分数（范围[0.9-1]内的CycleGAN为92.3%，而LaMa为47.7%）。这种方法解决了传统方法的关键限制，为自动化收获和产量监测提供了复杂遮挡模式下的稳健解决方案。
URL
https://arxiv.org/abs/2507.23487
PDF
-
MagicRoad: Semantic-Aware 3D Road Surface Reconstruction via Obstacle Inpainting
2025-07-31 08:38:36Xingyue Peng, Yuandong Lyu, Lang Zhang, Jian Zhu, Songtao Wang, Jiaxin Deng, Songxin Lu, Weiliang Ma, Dangen She, Peng Jia, XianPeng LangAbstract
Road surface reconstruction is essential for autonomous driving, supporting centimeter-accurate lane perception and high-definition mapping in complex urban this http URL recent methods based on mesh rendering or 3D Gaussian splatting (3DGS) achieve promising results under clean and static conditions, they remain vulnerable to occlusions from dynamic agents, visual clutter from static obstacles, and appearance degradation caused by lighting and weather changes. We present a robust reconstruction framework that integrates occlusion-aware 2D Gaussian surfels with semantic-guided color enhancement to recover clean, consistent road surfaces. Our method leverages a planar-adapted Gaussian representation for efficient large-scale modeling, employs segmentation-guided video inpainting to remove both dynamic and static foreground objects, and enhances color coherence via semantic-aware correction in HSV space. Extensive experiments on urban-scale datasets demonstrate that our framework produces visually coherent and geometrically faithful reconstructions, significantly outperforming prior methods under real-world conditions.
Abstract (translated)
路面重建对于自动驾驶至关重要，它支持在复杂的城市环境中实现厘米级精度的车道感知和高清晰度地图绘制。尽管最近基于网格渲染或三维高斯喷射（3DGS）的方法在干净且静态条件下取得了有希望的结果，但它们仍然容易受到动态对象遮挡、静态障碍物造成的视觉混乱以及光照和天气变化引起的外观退化的影响。我们提出了一种稳健的重建框架，该框架结合了感知遮挡的二维高斯表面元素与语义引导的颜色增强，以恢复干净且一致的道路表面。 我们的方法利用了一种平面适应型的高斯表示法进行高效的大规模建模，并采用了分割引导视频修复技术来移除动态和静态前景对象。同时，通过在HSV空间中的语义感知校正来增强颜色的一致性。 在城市规模的数据集上进行的大量实验表明，我们的框架能够生成视觉连贯且几何准确的重建结果，在实际条件下显著优于先前的方法。
URL
https://arxiv.org/abs/2507.23340
PDF
