Inpainting
-
Detection and Simulation of Urban Heat Islands Using a Fine-Tuned Geospatial Foundation Model for Microclimate Impact Prediction
2025-10-21 16:21:15Jannis Fleckenstein, David Kreismann, Tamara Rosemary Govindasamy, Thomas Brunschwiler, Etienne Vos, Mattia RigottiAbstract
As urbanization and climate change progress, urban heat island effects are becoming more frequent and severe. To formulate effective mitigation plans, cities require detailed air temperature data, yet conventional machine learning models with limited data often produce inaccurate predictions, particularly in underserved areas. Geospatial foundation models trained on global unstructured data offer a promising alternative by demonstrating strong generalization and requiring only minimal fine-tuning. In this study, an empirical ground truth of urban heat patterns is established by quantifying cooling effects from green spaces and benchmarking them against model predictions to evaluate the model's accuracy. The foundation model is subsequently fine-tuned to predict land surface temperatures under future climate scenarios, and its practical value is demonstrated through a simulated inpainting that highlights its role for mitigation support. The results indicate that foundation models offer a powerful way for evaluating urban heat island mitigation strategies in data-scarce regions to support more climate-resilient cities.
Abstract (translated)
随着城市化和气候变化的加剧，城市热岛效应变得越来越频繁和严重。为了制定有效的缓解计划，城市需要详细的空气温度数据，然而，由于缺乏数据，传统的机器学习模型往往会产生不准确的预测，尤其是在服务不足的地区。基于全球非结构化数据训练的地空基础模型提供了一种有前景的替代方案，通过展示强大的泛化能力和仅需少量微调的特点来实现这一目标。 在这项研究中，通过对绿地冷却效应进行量化并将其与模型预测进行对比，以建立城市热模式的经验真实值，以此评估模型的准确性。随后对基础模型进行了微调，使其能够根据未来气候情景预测地表温度，并通过模拟修补任务展示了其实际价值，突显了它在缓解支持中的作用。 研究结果表明，基础模型为评价数据稀缺地区的城市热岛缓解策略提供了一种强大的方法，有助于建设更具气候适应性的城市。
URL
https://arxiv.org/abs/2510.18773
PDF
-
{epsilon}-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data
2025-10-21 13:41:07Sheida Rahnamai Kordasiabi, Damian Dalle Nogare, Florian JugAbstract
Semantic segmentation of electron microscopy (EM) images of biological samples remains a challenge in the life sciences. EM data captures details of biological structures, sometimes with such complexity that even human observers can find it overwhelming. We introduce {\epsilon}-Seg, a method based on hierarchical variational autoencoders (HVAEs), employing center-region masking, sparse label contrastive learning (CL), a Gaussian mixture model (GMM) prior, and clustering-free label prediction. Center-region masking and the inpainting loss encourage the model to learn robust and representative embeddings to distinguish the desired classes, even if training labels are sparse (0.05% of the total image data or less). For optimal performance, we employ CL and a GMM prior to shape the latent space of the HVAE such that encoded input patches tend to cluster wrt. the semantic classes we wish to distinguish. Finally, instead of clustering latent embeddings for semantic segmentation, we propose a MLP semantic segmentation head to directly predict class labels from latent embeddings. We show empirical results of {\epsilon}-Seg and baseline methods on 2 dense EM datasets of biological tissues and demonstrate the applicability of our method also on fluorescence microscopy data. Our results show that {\epsilon}-Seg is capable of achieving competitive sparsely-supervised segmentation results on complex biological image data, even if only limited amounts of training labels are available.
Abstract (translated)
生物样本的电子显微镜（EM）图像语义分割在生命科学领域仍是一个挑战。EM数据捕捉了生物结构的详细信息，有时复杂到即使是人类观察者也难以应对。我们提出了{\epsilon}-Seg方法，该方法基于分层变分自编码器（HVAE），采用中心区域掩码、稀疏标签对比学习（CL）、高斯混合模型（GMM）先验以及无聚类的标签预测技术。 - 中心区域掩码和修补损失促使模型学会稳健且具有代表性的嵌入，即使训练标签非常稀疏（占总图像数据的0.05%或更少），也能区分所需的类别。 - 为了达到最佳性能，我们采用CL和GMM先验来塑造HVAE的潜在空间，使编码输入补丁倾向于根据我们希望区别的语义类进行聚类。 - 最后，在不使用潜在嵌入集群的情况下进行语义分割，{\epsilon}-Seg方法提出了一种MLP（多层感知器）语义分割头直接从潜在嵌入预测类别标签。 我们在两种密集EM生物组织数据集上展示了{\epsilon}-Seg及其基线方法的实证结果，并表明我们的方法在荧光显微镜图像数据上也具有适用性。研究表明，即使只有少量训练标签可用时，{\epsilon}-Seg仍能对复杂生物图像数据实现竞争性的稀疏监督分割效果。
URL
https://arxiv.org/abs/2510.18637
PDF
-
GPTFace: Generative Pre-training of Facial-Linguistic Transformer by Span Masking and Weakly Correlated Text-image Data
2025-10-21 06:55:44Yudong Li, Hao Li, Xianxu Hou, Linlin ShenAbstract
Compared to the prosperity of pre-training models in natural image understanding, the research on large-scale pre-training models for facial knowledge learning is still limited. Current approaches mainly rely on manually assembled and annotated face datasets for training, but labeling such datasets is labor-intensive and the trained models have limited scalability beyond the training data. To address these limitations, we present a generative pre-training model for facial knowledge learning that leverages large-scale web-built data for training. We use texts and images containing human faces crawled from the internet and conduct pre-training on self-supervised tasks, including masked image/language modeling (MILM) and image-text matching (ITM). During the generation stage, we further utilize the image-text matching loss to pull the generation distribution towards the control signal for controllable image/text generation. Experimental results demonstrate that our model achieves comparable performance to state-of-the-art pre-training models for various facial downstream tasks, such as attribution classification and expression recognition. Furthermore, our approach is also applicable to a wide range of face editing tasks, including face attribute editing, expression manipulation, mask removal, and photo inpainting.
Abstract (translated)
与自然图像理解中预训练模型的繁荣相比，针对面部知识学习的大规模预训练研究仍处于初级阶段。目前的方法主要依赖于手动收集和标注的人脸数据集进行训练，但这种标注过程费时费力，并且所训练出的模型在超出训练数据范围后可扩展性有限。为了解决这些问题，我们提出了一种基于大规模网络构建的数据来训练的生成式预训练模型，用于面部知识学习。我们从互联网上抓取包含人脸的文本和图像，并进行包括掩码图像/语言建模（MILM）和图像-文本匹配（ITM）在内的自监督任务的预训练。在生成阶段，进一步利用图像-文本匹配损失将生成分布拉向控制信号，以实现可控的图像/文本生成。 实验结果表明，在各种面部下游任务中，如属性分类和表情识别方面，我们的模型实现了与当前最先进的预训练模型相当的表现。此外，该方法还适用于广泛的面部编辑任务，包括人脸属性编辑、表情操作、口罩去除以及照片修补等。
URL
https://arxiv.org/abs/2510.18345
PDF
-
Cost Savings from Automatic Quality Assessment of Generated Images
2025-10-17 19:41:03Xavier Giro-i-Nieto, Nefeli Andreou, Anqi Liang, Manel Baradad, Francesc Moreno-Noguer, Aleix MartinezAbstract
Deep generative models have shown impressive progress in recent years, making it possible to produce high quality images with a simple text prompt or a reference image. However, state of the art technology does not yet meet the quality standards offered by traditional photographic methods. For this reason, production pipelines that use generated images often include a manual stage of image quality assessment (IQA). This process is slow and expensive, especially because of the low yield of automatically generated images that pass the quality bar. The IQA workload can be reduced by introducing an automatic pre-filtering stage, that will increase the overall quality of the images sent to review and, therefore, reduce the average cost required to obtain a high quality image. We present a formula that estimates the cost savings depending on the precision and pass yield of a generic IQA engine. This formula is applied in a use case of background inpainting, showcasing a significant cost saving of 51.61% obtained with a simple AutoML solution.
Abstract (translated)
近年来，深度生成模型取得了显著进展，能够仅凭简短的文本提示或参考图像就生成高质量的图片。然而，最先进的技术尚未达到传统摄影方法所要求的质量标准。因此，在使用生成图像的工作流程中，通常会包含一个手动质量评估（IQA）阶段。这一过程既耗时又昂贵，尤其是由于自动产生的图像中只有很少一部分能通过质量审核。 可以通过引入自动预过滤阶段来减轻IQA工作量。这种预过滤可以提高提交审查的图片整体质量，并因此降低获得高质量图像所需的平均成本。我们提出了一种根据通用IQA引擎的精度和合格率估算节省成本的公式。该公式的应用案例为背景修复，其中通过一个简单的自动机器学习（AutoML）解决方案实现了51.61%的成本节约。 这种方法不仅能够显著减少手动审查工作量，还能提高生成图像的质量，从而优化整个生产流程的效率与效果。
URL
https://arxiv.org/abs/2510.16179
PDF
-
Post-Processing Methods for Improving Accuracy in MRI Inpainting
2025-10-17 03:42:23Nishad Kulkarni, Krithika Iyer, Austin Tapp, Abhijeet Parida, Daniel Capell\'an-Mart\'in, Zhifan Jiang, Mar\'ia J. Ledesma-Carbayo, Syed Muhammad Anwar, Marius George LinguraruAbstract
Magnetic Resonance Imaging (MRI) is the primary imaging modality used in the diagnosis, assessment, and treatment planning for brain pathologies. However, most automated MRI analysis tools, such as segmentation and registration pipelines, are optimized for healthy anatomies and often fail when confronted with large lesions such as tumors. To overcome this, image inpainting techniques aim to locally synthesize healthy brain tissues in tumor regions, enabling the reliable application of general-purpose tools. In this work, we systematically evaluate state-of-the-art inpainting models and observe a saturation in their standalone performance. In response, we introduce a methodology combining model ensembling with efficient post-processing strategies such as median filtering, histogram matching, and pixel averaging. Further anatomical refinement is achieved via a lightweight U-Net enhancement stage. Comprehensive evaluation demonstrates that our proposed pipeline improves the anatomical plausibility and visual fidelity of inpainted regions, yielding higher accuracy and more robust outcomes than individual baseline models. By combining established models with targeted post-processing, we achieve improved and more accessible inpainting outcomes, supporting broader clinical deployment and sustainable, resource-conscious research. Our 2025 BraTS inpainting docker is available at this https URL.
Abstract (translated)
磁共振成像（MRI）是用于诊断、评估和治疗规划脑部疾病的主要影像学方法。然而，大多数自动化的MRI分析工具，如分割和配准管道，都是为正常解剖结构优化的，在遇到大型病变（例如肿瘤）时常常失效。为了克服这一问题，图像修复技术旨在局部合成肿瘤区域内的健康脑组织，从而使通用工具能够可靠地应用。 在本研究中，我们系统性地评估了当前最先进的修复模型，并观察到它们独立性能上的饱和现象。作为回应，我们提出了一种结合模型集成与有效后处理策略（如中值滤波、直方图匹配和像素平均）的方法。通过一个轻量级的U-Net增强阶段进一步实现解剖学精炼。全面评估表明，我们的提议管道提高了修复区域的解剖合理性及视觉真实性，并且相较于单独的基础模型产生了更高的准确性和更稳健的结果。 通过结合现有的模型与针对性的后处理技术，我们实现了改进和更具可访问性的图像修复结果，支持更广泛的临床部署以及可持续、资源节约型的研究。我们的2025 BraTS 图像修复Docker在[此处](https://此链接应为实际提供的URL)可用。
URL
https://arxiv.org/abs/2510.15282
PDF
-
Deep generative priors for 3D brain analysis
2025-10-16 20:20:50Ana Lawry Aguila, Dina Zemlyanker, You Cheng, Sudeshna Das, Daniel C. Alexander, Oula Puonti, Annabel Sorby-Adams, W. Taylor Kimberly, Juan Eugenio IglesiasAbstract
Diffusion models have recently emerged as powerful generative models in medical imaging. However, it remains a major challenge to combine these data-driven models with domain knowledge to guide brain imaging problems. In neuroimaging, Bayesian inverse problems have long provided a successful framework for inference tasks, where incorporating domain knowledge of the imaging process enables robust performance without requiring extensive training data. However, the anatomical modeling component of these approaches typically relies on classical mathematical priors that often fail to capture the complex structure of brain anatomy. In this work, we present the first general-purpose application of diffusion models as priors for solving a wide range of medical imaging inverse problems. Our approach leverages a score-based diffusion prior trained extensively on diverse brain MRI data, paired with flexible forward models that capture common image processing tasks such as super-resolution, bias field correction, inpainting, and combinations thereof. We further demonstrate how our framework can refine outputs from existing deep learning methods to improve anatomical fidelity. Experiments on heterogeneous clinical and research MRI data show that our method achieves state-of-the-art performance producing consistent, high-quality solutions without requiring paired training datasets. These results highlight the potential of diffusion priors as versatile tools for brain MRI analysis.
Abstract (translated)
最近，扩散模型作为医学成像中的强大生成模型崭露头角。然而，如何将这些数据驱动的模型与领域知识相结合以指导脑部成像问题仍然是一个重大挑战。在神经影像学中，贝叶斯逆问题长期以来为推理任务提供了一个成功的框架，在此框架下，通过融入成像过程的知识能够使性能稳健且无需大量训练数据。然而，这些方法中的解剖建模部分通常依赖于经典的数学先验，这些先验往往无法捕捉到大脑结构的复杂性。 在这项工作中，我们首次将扩散模型作为先验用于解决广泛的医学成像逆问题，并进行了通用应用研究。我们的方法利用了在大量多样化的脑部MRI数据上训练出来的基于分数的扩散先验，并结合灵活的正向模型来捕捉诸如超分辨率、偏场校正、图像填充等常见图像处理任务，以及这些任务的不同组合。我们进一步展示了如何通过我们的框架改进现有深度学习方法的输出以提高解剖学准确性。 在异质性的临床和研究MRI数据上进行的实验表明，我们的方法能够实现业界领先的表现，在无需配对训练数据的情况下生成一致且高质量的结果。这些结果突显了扩散先验作为脑部MRI分析中多功能工具的巨大潜力。
URL
https://arxiv.org/abs/2510.15119
PDF
-
Inpainting the Red Planet: Diffusion Models for the Reconstruction of Martian Environments in Virtual Reality
2025-10-16 15:02:05Giuseppe Lorenzo Catalano, Agata Marta SocciniAbstract
Space exploration increasingly relies on Virtual Reality for several tasks, such as mission planning, multidisciplinary scientific analysis, and astronaut training. A key factor for the reliability of the simulations is having accurate 3D representations of planetary terrains. Extraterrestrial heightmaps derived from satellite imagery often contain missing values due to acquisition and transmission constraints. Mars is among the most studied planets beyond Earth, and its extensive terrain datasets make the Martian surface reconstruction a valuable task, although many areas remain unmapped. Deep learning algorithms can support void-filling tasks; however, whereas Earth's comprehensive datasets enables the use of conditional methods, such approaches cannot be applied to Mars. Current approaches rely on simpler interpolation techniques which, however, often fail to preserve geometric coherence. In this work, we propose a method for reconstructing the surface of Mars based on an unconditional diffusion model. Training was conducted on an augmented dataset of 12000 Martian heightmaps derived from NASA's HiRISE survey. A non-homogeneous rescaling strategy captures terrain features across multiple scales before resizing to a fixed 128x128 model resolution. We compared our method against established void-filling and inpainting techniques, including Inverse Distance Weighting, kriging, and Navier-Stokes algorithm, on an evaluation set of 1000 samples. Results show that our approach consistently outperforms these methods in terms of reconstruction accuracy (4-15% on RMSE) and perceptual similarity (29-81% on LPIPS) with the original data.
Abstract (translated)
空间探索越来越依赖虚拟现实技术来完成多项任务，如任务规划、跨学科科学分析和宇航员训练。模拟的可靠性关键在于拥有准确的三维地形表示。从卫星图像获得的地外高程图（高度地图）通常会因为获取和传输限制而包含缺失值。火星是地球以外研究最为深入的行星之一，其广泛的地形数据集使得火星表面重建成为一个有价值的课题，尽管许多地区仍未绘制完毕。深度学习算法可以支持填补空缺的任务；然而，由于地球详尽的数据集能够使用条件方法，这种方法却不能应用于火星。当前的方法依赖于较为简单的插值技术，但这些技术通常无法保持几何一致性。 在本工作中，我们提出了一种基于无条件扩散模型重建火星表面的方法。训练是在一个扩充的由NASA的HiRISE调查数据衍生出的12000张火星高程图的数据集上进行的。通过非均匀重缩放策略，在将图像调整为固定的128x128分辨率之前，捕获不同尺度下的地形特征。我们在包含1000个样本的评估集中比较了我们的方法与已建立的填补空缺和修复技术，包括反距离加权法、克里金插值和Navier-Stokes算法。结果显示，无论是在重建准确性（以RMSE衡量，提高了4-15%）还是感知相似性（以LPIPS衡量，提高了29-81%），我们的方法均优于这些方法与原始数据对比的结果。 此研究展示了利用深度学习模型在火星表面重建任务中的巨大潜力，并为未来进一步探索提供了可能的方向。
URL
https://arxiv.org/abs/2510.14765
PDF
-
Generative human motion mimicking through feature extraction in denoising diffusion settings
2025-10-16 14:41:54Alexander Okupnik, Johannes Schneider, Kyriakos FlourisAbstract
Recent success with large language models has sparked a new wave of verbal human-AI interaction. While such models support users in a variety of creative tasks, they lack the embodied nature of human interaction. Dance, as a primal form of human expression, is predestined to complement this experience. To explore creative human-AI interaction exemplified by dance, we build an interactive model based on motion capture (MoCap) data. It generates an artificial other by partially mimicking and also "creatively" enhancing an incoming sequence of movement data. It is the first model, which leverages single-person motion data and high level features in order to do so and, thus, it does not rely on low level human-human interaction data. It combines ideas of two diffusion models, motion inpainting, and motion style transfer to generate movement representations that are both temporally coherent and responsive to a chosen movement reference. The success of the model is demonstrated by quantitatively assessing the convergence of the feature distribution of the generated samples and the test set which serves as simulating the human performer. We show that our generations are first steps to creative dancing with AI as they are both diverse showing various deviations from the human partner while appearing realistic.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00011
PDF
-
TerraCodec: Compressing Earth Observations
2025-10-14 16:05:31Julen Costa-Watanabe, Isabelle Wittmann, Benedikt Blumenstiel, Konrad SchindlerAbstract
Earth observation (EO) satellites produce massive streams of multispectral image time series, posing pressing challenges for storage and transmission. Yet, learned EO compression remains fragmented, lacking publicly available pretrained models and misaligned with advances in compression for natural imagery. Image codecs overlook temporal redundancy, while video codecs rely on motion priors that fail to capture the radiometric evolution of largely static scenes. We introduce TerraCodec (TEC), a family of learned codecs tailored to EO. TEC includes efficient image-based variants adapted to multispectral inputs, as well as a Temporal Transformer model (TEC-TT) that leverages dependencies across time. To overcome the fixed-rate setting of today's neural codecs, we present Latent Repacking, a novel method for training flexible-rate transformer models that operate on varying rate-distortion settings. Trained on Sentinel-2 data, TerraCodec outperforms classical codecs, achieving 3-10x stronger compression at equivalent image quality. Beyond compression, TEC-TT enables zero-shot cloud inpainting, surpassing state-of-the-art methods on the AllClear benchmark. Our results establish bespoke, learned compression algorithms as a promising direction for Earth observation. Code and model weights will be released under a permissive license.
Abstract (translated)
地球观测（EO）卫星产生了大量的多光谱图像时间序列，这对存储和传输提出了紧迫的挑战。然而，目前的EO压缩方法仍然比较分散，缺乏公开可用的预训练模型，并且与自然影像压缩技术的进步不一致。传统的图象编解码器忽略了时间冗余，而视频编解码器依赖于运动先验信息，这些信息无法捕捉到静止场景中辐射量的变化。 我们引入了TerraCodec（TEC），这是一个专为EO设计的学习编码器家族。TEC包括针对多光谱输入进行了优化的高效图像变体，以及利用时间依赖性的时序变换模型（TEC-TT）。为了克服现有神经编解码器固定率设置的问题，我们提出了一种称为潜在重组的新方法，该方法用于训练在不同率失真设置下运行的灵活率转换模型。在Sentinel-2数据集上进行训练后，TerraCodec在等效图像质量的情况下，比传统的编解码器实现了3到10倍更强的压缩效果。 除了压缩性能外，TEC-TT还能够在无监督条件下实现云层插补，并且在AllClear基准测试中超越了现有最先进的方法。我们的研究结果表明，为地球观测量身定制的学习压缩算法是一个很有前途的研究方向。代码和模型权重将以宽松许可协议的形式发布。
URL
https://arxiv.org/abs/2510.12670
PDF
-
G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior
2025-10-14 03:06:28Junfeng Ni, Yixin Chen, Zhifei Yang, Yu Liu, Ruijie Lu, Song-Chun Zhu, Siyuan HuangAbstract
Despite recent advances in leveraging generative prior from pre-trained diffusion models for 3D scene reconstruction, existing methods still face two critical limitations. First, due to the lack of reliable geometric supervision, they struggle to produce high-quality reconstructions even in observed regions, let alone in unobserved areas. Second, they lack effective mechanisms to mitigate multi-view inconsistencies in the generated images, leading to severe shape-appearance ambiguities and degraded scene geometry. In this paper, we identify accurate geometry as the fundamental prerequisite for effectively exploiting generative models to enhance 3D scene reconstruction. We first propose to leverage the prevalence of planar structures to derive accurate metric-scale depth maps, providing reliable supervision in both observed and unobserved regions. Furthermore, we incorporate this geometry guidance throughout the generative pipeline to improve visibility mask estimation, guide novel view selection, and enhance multi-view consistency when inpainting with video diffusion models, resulting in accurate and consistent scene completion. Extensive experiments on Replica, ScanNet++, and DeepBlending show that our method consistently outperforms existing baselines in both geometry and appearance reconstruction, particularly for unobserved regions. Moreover, our method naturally supports single-view inputs and unposed videos, with strong generalizability in both indoor and outdoor scenarios with practical real-world applicability. The project page is available at this https URL.
Abstract (translated)
尽管近期利用预训练扩散模型的生成先验进行3D场景重建取得了进展，现有方法仍面临两个关键限制。首先，由于缺乏可靠的几何监督，它们在已观察区域难以产生高质量的重建效果，更不用说未观察到的区域了。其次，这些方法缺少有效机制来减少多视角图像中的不一致性问题，导致严重的形状-外观模糊性和场景几何结构退化。 在这篇论文中，我们识别出准确的几何信息是利用生成模型增强3D场景重建的关键前提条件。首先，我们提出利用平面结构的普遍性推导出精确的度量级深度图，这为已观察和未观察区域提供了可靠的监督。此外，我们将这种几何指导贯穿于整个生成管道中，以改进可见性掩码估计、引导新颖视角选择，并在使用视频扩散模型进行填补时提高多视图一致性，从而实现准确且一致的场景完成。 我们在Replica、ScanNet++ 和 DeepBlending 数据集上的大量实验表明，我们的方法在几何和外观重建方面始终优于现有的基准方法，特别是在未观察区域。此外，我们的方法自然支持单视角输入及未经定位处理的视频，并具有在室内和室外实际应用场景中强大泛化的特性。 该项目页面可在此 [URL] 访问。
URL
https://arxiv.org/abs/2510.12099
PDF
-
DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training
2025-10-13 17:59:15Haoran Feng, Dizhe Zhang, Xiangtai Li, Bo Du, Lu QiAbstract
In this work, we propose DiT360, a DiT-based framework that performs hybrid training on perspective and panoramic data for panoramic image generation. For the issues of maintaining geometric fidelity and photorealism in generation quality, we attribute the main reason to the lack of large-scale, high-quality, real-world panoramic data, where such a data-centric view differs from prior methods that focus on model design. Basically, DiT360 has several key modules for inter-domain transformation and intra-domain augmentation, applied at both the pre-VAE image level and the post-VAE token level. At the image level, we incorporate cross-domain knowledge through perspective image guidance and panoramic refinement, which enhance perceptual quality while regularizing diversity and photorealism. At the token level, hybrid supervision is applied across multiple modules, which include circular padding for boundary continuity, yaw loss for rotational robustness, and cube loss for distortion awareness. Extensive experiments on text-to-panorama, inpainting, and outpainting tasks demonstrate that our method achieves better boundary consistency and image fidelity across eleven quantitative metrics. Our code is available at this https URL.
Abstract (translated)
在这项工作中，我们提出了DiT360框架，这是一个基于DiT的框架，它在透视图和全景数据上进行混合训练以生成全景图像。对于保持几何保真度和生成质量的逼真度问题，我们将主要原因归结为缺乏大规模、高质量的真实世界全景数据，这种以数据为中心的观点与以前专注于模型设计的方法有所不同。 基本而言，DiT360有几个关键模块用于跨域转换和同域增强，在预VAE图像级别和后VAE标记级别均应用这些模块。在图像层面，我们通过透视图指导和全景细化整合了跨域知识，这提高了感知质量的同时还规范了多样性和逼真度。在标记层级上，混合监督应用于多个模块，包括圆形填充以保持边界连续性、偏航损失以增强旋转鲁棒性以及立方体损失以提高对失真的感知。 在从文本生成全景图、图像修复和扩展等任务的广泛实验中显示，我们的方法在十一项定量度量标准上实现了更好的边界一致性和图像保真度。我们的代码可在[此链接](https://this-https-url.com)获取（请将"this https URL"替换为实际的URL）。
URL
https://arxiv.org/abs/2510.11712
PDF
-
PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
2025-10-13 17:29:51Pradyumna Yalandur Muralidhar, Yuxuan Xue, Xianghui Xie, Margaret Kostyrko, Gerard Pons-MollAbstract
Reconstructing metrically accurate humans and their surrounding scenes from a single image is crucial for virtual reality, robotics, and comprehensive 3D scene understanding. However, existing methods struggle with depth ambiguity, occlusions, and physically inconsistent contacts. To address these challenges, we introduce PhySIC, a framework for physically plausible Human-Scene Interaction and Contact reconstruction. PhySIC recovers metrically consistent SMPL-X human meshes, dense scene surfaces, and vertex-level contact maps within a shared coordinate frame from a single RGB image. Starting from coarse monocular depth and body estimates, PhySIC performs occlusion-aware inpainting, fuses visible depth with unscaled geometry for a robust metric scaffold, and synthesizes missing support surfaces like floors. A confidence-weighted optimization refines body pose, camera parameters, and global scale by jointly enforcing depth alignment, contact priors, interpenetration avoidance, and 2D reprojection consistency. Explicit occlusion masking safeguards invisible regions against implausible configurations. PhySIC is efficient, requiring only 9 seconds for joint human-scene optimization and under 27 seconds end-to-end. It naturally handles multiple humans, enabling reconstruction of diverse interactions. Empirically, PhySIC outperforms single-image baselines, reducing mean per-vertex scene error from 641 mm to 227 mm, halving PA-MPJPE to 42 mm, and improving contact F1 from 0.09 to 0.51. Qualitative results show realistic foot-floor interactions, natural seating, and plausible reconstructions of heavily occluded furniture. By converting a single image into a physically plausible 3D human-scene pair, PhySIC advances scalable 3D scene understanding. Our implementation is publicly available at this https URL.
Abstract (translated)
从单张图像中重建出具有精确度量的人体模型及其周围场景对于虚拟现实、机器人技术和全面的3D场景理解至关重要。然而，现有的方法在处理深度模糊性、遮挡以及物理上不一致的接触点方面存在困难。为了解决这些问题，我们引入了PhySIC框架，该框架能够重建出符合物理原理的人体与场景交互及接触重构结果。通过这一框架，从单张RGB图像中可以恢复出具有度量一致性的人体SMPL-X网格模型、密集的场景表面以及顶点级别的接触图，并且所有这些都在一个共享的坐标系内完成。 PhySIC从粗略的单眼深度和人体姿态估计开始，通过遮挡感知填充处理、可见深度与未缩放几何结构融合以生成稳健度量框架，并合成缺少的支持面如地板。带权重优化则通过共同强制执行深度对齐、接触先验条件、避免穿透以及2D重投影一致性来细化身体姿势、相机参数和全局尺度，同时显式的遮挡掩码保护不可见区域免受不合理配置的影响。 PhySIC是高效的，在联合人体-场景优化时仅需9秒，而端到端处理时间低于27秒。它能够自然地处理多个个体，并支持重建各种复杂的互动情况。实验表明，与单图像基线方法相比，PhySIC的性能更优，将每顶点场景误差从641毫米减少至227毫米，姿势绝对平均部分关节位置误差（PA-MPJPE）减半至42毫米，并且接触F1分数从0.09提升到了0.51。定性的结果显示了现实的脚-地板互动、自然坐姿以及大量遮挡家具下的合理重建效果。 通过将单个图像转换为物理上可行的人体-场景3D对，PhySIC推进了大规模的3D场景理解技术的发展。我们的实现代码可在[此链接](https://github.com/alibaba/PhySIC)公开获取。
URL
https://arxiv.org/abs/2510.11649
PDF
-
Perspective-aware 3D Gaussian Inpainting with Multi-view Consistency
2025-10-13 04:10:39Yuxin Cheng, Binxiao Huang, Taiqiang Wu, Wenyong Zhou, Chenchen Ding, Zhengwu Liu, Graziano Chesi, Ngai WongAbstract
3D Gaussian inpainting, a critical technique for numerous applications in virtual reality and multimedia, has made significant progress with pretrained diffusion models. However, ensuring multi-view consistency, an essential requirement for high-quality inpainting, remains a key challenge. In this work, we present PAInpainter, a novel approach designed to advance 3D Gaussian inpainting by leveraging perspective-aware content propagation and consistency verification across multi-view inpainted images. Our method iteratively refines inpainting and optimizes the 3D Gaussian representation with multiple views adaptively sampled from a perspective graph. By propagating inpainted images as prior information and verifying consistency across neighboring views, PAInpainter substantially enhances global consistency and texture fidelity in restored 3D scenes. Extensive experiments demonstrate the superiority of PAInpainter over existing methods. Our approach achieves superior 3D inpainting quality, with PSNR scores of 26.03 dB and 29.51 dB on the SPIn-NeRF and NeRFiller datasets, respectively, highlighting its effectiveness and generalization capability.
Abstract (translated)
3D高斯插值是一种关键技术，广泛应用于虚拟现实和多媒体领域，并且通过预训练的扩散模型已经取得了显著进展。然而，确保多视角一致性仍然是实现高质量插值的一个重要挑战，这是高质量图像修复的关键要求之一。在本研究中，我们提出了PAInpainter，这是一种新颖的方法，旨在利用透视感知内容传播以及跨多个视图的插值图像的一致性验证来推进3D高斯插值技术。 我们的方法通过迭代优化从视角图自适应采样的多视角中的插值结果，并改进3D高斯表示。通过将已修复的图片作为先验信息进行传播，并检查相邻视角之间的连续性，PAInpainter能够显著提高恢复出的三维场景的整体一致性和纹理保真度。 广泛的实验表明了PAInpainter相较于现有方法的优势。我们的方法在SPIn-NeRF和NeRFiller数据集上分别取得了26.03 dB和29.51 dB的PSNR评分，这不仅展示了其卓越的质量，还证明了它具有良好的泛化能力。
URL
https://arxiv.org/abs/2510.10993
PDF
-
A Biophysically-Conditioned Generative Framework for 3D Brain Tumor MRI Synthesis
2025-10-10 13:23:33Valentin Biller, Lucas Zimmer, Can Erdur, Sandeep Nagar, Daniel R\"uckert, Niklas Bubeck, Jonas WeidnerAbstract
Magnetic resonance imaging (MRI) inpainting supports numerous clinical and research applications. We introduce the first generative model that conditions on voxel-level, continuous tumor concentrations to synthesize high-fidelity brain tumor MRIs. For the BraTS 2025 Inpainting Challenge, we adapt this architecture to the complementary task of healthy tissue restoration by setting the tumor concentrations to zero. Our latent diffusion model conditioned on both tissue segmentations and the tumor concentrations generates 3D spatially coherent and anatomically consistent images for both tumor synthesis and healthy tissue inpainting. For healthy inpainting, we achieve a PSNR of 18.5, and for tumor inpainting, we achieve 17.4. Our code is available at: this https URL
Abstract (translated)
磁共振成像（MRI）的修复技术支持众多临床和研究应用。我们介绍了首个生成模型，该模型以体素级别的、连续的肿瘤浓度为条件，合成高质量的大脑肿瘤MRI图像。对于BraTS 2025补全挑战赛，我们将此架构调整用于健康的组织恢复任务，方法是将肿瘤浓度设为零。我们的潜在扩散模型在同时考虑组织分割和肿瘤浓度的情况下，能够生成具有空间连贯性和解剖学一致性的3D图像，适用于肿瘤合成及健康组织修复。对于健康组织的修复，我们达到了18.5的PSNR值；而对于肿瘤修复，则达到17.4的PSNR值。我们的代码可在以下网址获取：[this https URL]
URL
https://arxiv.org/abs/2510.09365
PDF
-
VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioning
2025-10-09 17:58:59Minghong Cai, Qiulin Wang, Zongli Ye, Wenze Liu, Quande Liu, Weicai Ye, Xintao Wang, Pengfei Wan, Kun Gai, Xiangyu YueAbstract
We introduce the task of arbitrary spatio-temporal video completion, where a video is generated from arbitrary, user-specified patches placed at any spatial location and timestamp, akin to painting on a video canvas. This flexible formulation naturally unifies many existing controllable video generation tasks--including first-frame image-to-video, inpainting, extension, and interpolation--under a single, cohesive paradigm. Realizing this vision, however, faces a fundamental obstacle in modern latent video diffusion models: the temporal ambiguity introduced by causal VAEs, where multiple pixel frames are compressed into a single latent representation, making precise frame-level conditioning structurally difficult. We address this challenge with VideoCanvas, a novel framework that adapts the In-Context Conditioning (ICC) paradigm to this fine-grained control task with zero new parameters. We propose a hybrid conditioning strategy that decouples spatial and temporal control: spatial placement is handled via zero-padding, while temporal alignment is achieved through Temporal RoPE Interpolation, which assigns each condition a continuous fractional position within the latent sequence. This resolves the VAE's temporal ambiguity and enables pixel-frame-aware control on a frozen backbone. To evaluate this new capability, we develop VideoCanvasBench, the first benchmark for arbitrary spatio-temporal video completion, covering both intra-scene fidelity and inter-scene creativity. Experiments demonstrate that VideoCanvas significantly outperforms existing conditioning paradigms, establishing a new state of the art in flexible and unified video generation.
Abstract (translated)
我们介绍了任意时空视频补全任务，即从用户指定的在任何空间位置和时间戳放置的补丁生成视频，类似于在视频画布上作画。这种灵活的表述自然地统一了许多现有的可控视频生成任务——包括第一帧图像到视频、修复、扩展和插值——在一个单一且连贯的范式下。然而，实现这一愿景面临着现代潜在视频扩散模型的基本障碍：因果VAE（变分自编码器）引入的时间模糊性问题，在该问题中，多个像素帧被压缩成单个潜在表示，使得精确的帧级条件设置在结构上变得困难。 为了解决这个挑战，我们提出了VideoCanvas，这是一个新颖的框架，它将In-Context Conditioning (ICC)范式适应于这一细粒度控制任务，并且不引入任何新的参数。我们提出了一种混合条件策略，该策略分离了空间和时间控制：空间放置通过零填充处理，而时间对齐则通过Temporal RoPE Interpolation（时序RoPE插值）实现，为每个条件分配潜在序列中的连续分数位置。这解决了VAE的时间模糊性问题，并在冻结的骨干网络上实现了像素帧感知的控制。 为了评估这一新能力，我们开发了VideoCanvasBench，这是第一个用于任意时空视频补全的基准测试工具，涵盖了场景内的保真度和跨场景的创造性。实验表明，VideoCanvas显著优于现有的条件设置范式，在灵活且统一的视频生成方面确立了新的技术标准。
URL
https://arxiv.org/abs/2510.08555
PDF
-
One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting
2025-10-09 14:30:34Haipeng Liu, Yang Wang, Meng WangAbstract
Text-guided image inpainting aims at reconstructing the masked regions as per text prompts, where the longstanding challenges lie in the preservation for unmasked regions, while achieving the semantics consistency between unmasked and inpainted masked regions. Previous arts failed to address both of them, always with either of them to be remedied. Such facts, as we observed, stem from the entanglement of the hybrid (e.g., mid-and-low) frequency bands that encode varied image properties, which exhibit different robustness to text prompts during the denoising process. In this paper, we propose a null-text-null frequency-aware diffusion models, dubbed \textbf{NTN-Diff}, for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to circumvent two challenges in a row. Based on the diffusion process, we further divide the denoising process into early (high-level noise) and late (low-level noise) stages, where the mid-and-low frequency bands are disentangled during the denoising process. As observed, the stable mid-frequency band is progressively denoised to be semantically aligned during text-guided denoising process, which, meanwhile, serves as the guidance to the null-text denoising process to denoise low-frequency band for the masked regions, followed by a subsequent text-guided denoising process at late stage, to achieve the semantics consistency for mid-and-low frequency bands across masked and unmasked regions, while preserve the unmasked regions. Extensive experiments validate the superiority of NTN-Diff over the state-of-the-art diffusion models to text-guided diffusion models. Our code can be accessed from this https URL.
Abstract (translated)
文本引导的图像修复（Text-guided image inpainting）旨在根据文本提示重建被遮罩区域，其中长期存在的挑战在于保持未遮罩区域的同时实现遮罩与非遮罩区域之间的语义一致性。先前的方法未能同时解决这两个问题，总是在两者之间做出取舍。我们注意到这种现象的原因是混合（例如中频和低频）频带的纠缠，这些频带编码了不同的图像属性，在去噪过程中对文本提示展现出不同级别的鲁棒性。 为了应对这一挑战，我们在本文中提出了一种基于零文本-零频率感知扩散模型的方法，称为**NTN-Diff**。该方法通过将遮罩区域与非遮罩区域之间的语义一致性分解为各个频带的一致性，并保持未被遮盖的区域来依次解决这两个问题。 在扩散过程中，我们将去噪过程进一步分为早期（高层噪声）和晚期（低层噪声）两个阶段，在这一过程中分离了中频和低频。我们观察到，在文本引导的去噪处理过程中，稳定的中频带逐步减少噪声并变得语义对齐，并且它充当指导零文本去噪的过程以去除遮罩区域的低频噪声，随后在晚期阶段进行后续的文本引导去噪过程，实现被掩码与非掩码区域之间跨中、低频带的语义一致性同时保持未遮蔽区域。 广泛的实验验证了NTN-Diff相比现有的最先进的扩散模型在基于文本的图像修复任务上的优越性。我们的代码可以从这个链接访问：[请在此处插入实际的URL]。
URL
https://arxiv.org/abs/2510.08273
PDF
-
RePainter: Empowering E-commerce Object Removal via Spatial-matting Reinforcement Learning
2025-10-09 02:57:33Zipeng Guo, Lichen Ma, Xiaolong Fu, Gaojing Zhou, Lan Yang, Yuchen Zhou, Linkai Liu, Yu He, Ximan Liu, Shiping Dong, Jingling Fu, Zhen Chen, Yu Shi, Junshi Huang, Jason Li, Chao GouAbstract
In web data, product images are central to boosting user engagement and advertising efficacy on e-commerce platforms, yet the intrusive elements such as watermarks and promotional text remain major obstacles to delivering clear and appealing product visuals. Although diffusion-based inpainting methods have advanced, they still face challenges in commercial settings due to unreliable object removal and limited domain-specific adaptation. To tackle these challenges, we propose Repainter, a reinforcement learning framework that integrates spatial-matting trajectory refinement with Group Relative Policy Optimization (GRPO). Our approach modulates attention mechanisms to emphasize background context, generating higher-reward samples and reducing unwanted object insertion. We also introduce a composite reward mechanism that balances global, local, and semantic constraints, effectively reducing visual artifacts and reward hacking. Additionally, we contribute EcomPaint-100K, a high-quality, large-scale e-commerce inpainting dataset, and a standardized benchmark EcomPaint-Bench for fair evaluation. Extensive experiments demonstrate that Repainter significantly outperforms state-of-the-art methods, especially in challenging scenes with intricate compositions. We will release our code and weights upon acceptance.
Abstract (translated)
在网页数据中，产品图片对于提高电子商务平台上的用户参与度和广告效果至关重要。然而，水印和促销文字等侵入性元素仍然是提供清晰、吸引人的产品视觉效果的主要障碍。尽管基于扩散的图像修复方法已经有所进步，但在商业环境中仍然面临着对象移除不可靠以及领域特定适应能力有限的问题。为了应对这些挑战，我们提出了Repainter框架，这是一种结合了空间-遮罩轨迹细化与组相对策略优化（GRPO）的强化学习框架。我们的方法通过调节注意力机制来强调背景信息，从而生成高奖励样本并减少不需要的对象插入。此外，我们引入了一种复合奖励机制，该机制平衡全局、局部和语义约束，有效减少了视觉伪影和奖励欺骗。 另外，我们还贡献了一个高质量的大规模电子商务图像修复数据集EcomPaint-100K以及一个标准化基准测试平台EcomPaint-Bench，以实现公正的评估。广泛的实验表明，Repainter在具有复杂构图的挑战性场景中显著优于现有的最先进的方法。一旦获得接受，我们将发布我们的代码和权重。
URL
https://arxiv.org/abs/2510.07721
PDF
-
Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding
2025-10-07 17:59:20Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, Juncheng Yan, Yan Tai, Jiayi Lei, Yuewen Cao, Keqi Wang, Yibin Wang, Jinbin Bai, Qian Yu, Dengyang Jiang, Yuandong Pu, Haoxing Chen, Le Zhuo, Junjun He, Gen Luo, Tianbin Li, Ming Hu, Jin Ye, Shenglong Ye, Bo Zhang, Chang Xu, Wenhai Wang, Hongsheng Li, Guangtao Zhai, Tianfan Xue, Bin Fu, Xiaohong Liu, Yu Qiao, Yihao LiuAbstract
We introduce Lumina-DiMOO, an open-source foundational model for seamless multi-modal generation and understanding. Lumina-DiMOO sets itself apart from prior unified models by utilizing a fully discrete diffusion modeling to handle inputs and outputs across various modalities. This innovative approach allows Lumina-DiMOO to achieve higher sampling efficiency compared to previous autoregressive (AR) or hybrid AR-Diffusion paradigms and adeptly support a broad spectrum of multi-modal tasks, including text-to-image generation, image-to-image generation (e.g., image editing, subject-driven generation, and image inpainting, etc.), as well as image understanding. Lumina-DiMOO achieves state-of-the-art performance on multiple benchmarks, surpassing existing open-source unified multi-modal models. To foster further advancements in multi-modal and discrete diffusion model research, we release our code and checkpoints to the community. Project Page: this https URL.
Abstract (translated)
我们介绍了Lumina-DiMOO，这是一个开源的基础模型，用于无缝的多模态生成和理解。与之前的统一模型不同，Lumina-DiMOO通过使用全离散扩散建模方法来处理各种模式下的输入和输出。这一创新的方法使Lumina-DiMOO在采样效率方面超越了以往的自回归（AR）或混合AR-扩散范式，并且能够有效地支持广泛的多模态任务，包括文本到图像生成、图像到图像生成（例如图像编辑、主题驱动生成及图像修复等），以及图像理解。Lumina-DiMOO在多个基准测试中取得了最先进的性能，超越了现有的开源统一的多模态模型。为了推动多模态和离散扩散模型研究的进步，我们将代码和检查点发布给社区。项目页面：[此链接](https://this-url.com/)。
URL
https://arxiv.org/abs/2510.06308
PDF
-
Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptation
Abstract
Large pretrained diffusion models can provide strong priors beneficial for many graphics applications. However, generative applications such as neural rendering and inverse methods such as SVBRDF estimation and intrinsic image decomposition require additional input or output channels. Current solutions for channel expansion are often application specific and these solutions can be difficult to adapt to different diffusion models or new tasks. This paper introduces Teamwork: a flexible and efficient unified solution for jointly increasing the number of input and output channels as well as adapting a pretrained diffusion model to new tasks. Teamwork achieves channel expansion without altering the pretrained diffusion model architecture by coordinating and adapting multiple instances of the base diffusion model (\ie, teammates). We employ a novel variation of Low Rank-Adaptation (LoRA) to jointly address both adaptation and coordination between the different teammates. Furthermore Teamwork supports dynamic (de)activation of teammates. We demonstrate the flexibility and efficiency of Teamwork on a variety of generative and inverse graphics tasks such as inpainting, single image SVBRDF estimation, intrinsic decomposition, neural shading, and intrinsic image synthesis.
Abstract (translated)
大型预训练扩散模型可以为许多图形应用提供强有力的先验条件。然而，生成性应用程序（如神经渲染）和逆向方法（例如SVBRDF估计和固有图像分解）需要额外的输入或输出通道。当前针对通道扩展的解决方案通常是特定于具体应用场景的，并且这些方案很难适应不同的扩散模型或新的任务需求。本文介绍了“Teamwork”：一种灵活高效的统一解决方案，用于同时增加输入和输出通道的数量并使预训练的扩散模型能够适应新任务。“Teamwork”在不改变现有预训练扩散模型架构的情况下实现通道扩展，通过协调与调整多个基础扩散模型实例（即，“队友”）来完成这一目标。我们使用了一种新颖的低秩自适应（LoRA）变体，以同时解决不同“队友”之间的适配和协同问题。“Teamwork”还支持动态启用或停用这些“队友”。我们在多种生成性和逆向图形任务上展示了“Teamwork”的灵活性与效率，包括图像修复、单幅图像SVBRDF估计、固有分解、神经着色以及固有图像合成。
URL
https://arxiv.org/abs/2510.05532
PDF
-
Soft Disentanglement in Frequency Bands for Neural Audio Codecs
2025-10-04 08:40:56Benoit Ginies, Xiaoyu Bie, Olivier Fercoq, Ga\"el RichardAbstract
In neural-based audio feature extraction, ensuring that representations capture disentangled information is crucial for model interpretability. However, existing disentanglement methods often rely on assumptions that are highly dependent on data characteristics or specific tasks. In this work, we introduce a generalizable approach for learning disentangled features within a neural architecture. Our method applies spectral decomposition to time-domain signals, followed by a multi-branch audio codec that operates on the decomposed components. Empirical evaluations demonstrate that our approach achieves better reconstruction and perceptual performance compared to a state-of-the-art baseline while also offering potential advantages for inpainting tasks.
Abstract (translated)
在基于神经网络的音频特征提取中，确保表示能够捕捉到分离的信息对于模型的可解释性至关重要。然而，现有的分离方法往往依赖于高度依赖数据特性或特定任务的假设。在这项工作中，我们介绍了一种可以在神经架构内学习分离特征的一般化方法。我们的方法首先对时域信号进行频谱分解，然后在分解后的组件上运行一个多分支音频编解码器。经验评估表明，与最新的基准相比，我们的方法不仅实现了更好的重建和感知性能，还为插值任务提供了潜在的优势。
URL
https://arxiv.org/abs/2510.03735
PDF
