Abstract
Thermal imaging is crucial for scene understanding, particularly in low-light and nighttime conditions. However, collecting large thermal datasets is costly and labor-intensive due to the specialized equipment required for infrared image capture. To address this challenge, researchers have explored visible-to-thermal image translation. Most existing methods rely on Generative Adversarial Networks (GANs) or Diffusion Models (DMs), treating the task as a style transfer problem. As a result, these approaches attempt to learn both the modality distribution shift and underlying physical principles from limited training data. In this paper, we propose F-ViTA, a novel approach that leverages the general world knowledge embedded in foundation models to guide the diffusion process for improved translation. Specifically, we condition an InstructPix2Pix Diffusion Model with zero-shot masks and labels from foundation models such as SAM and Grounded DINO. This allows the model to learn meaningful correlations between scene objects and their thermal signatures in infrared imagery. Extensive experiments on five public datasets demonstrate that F-ViTA outperforms state-of-the-art (SOTA) methods. Furthermore, our model generalizes well to out-of-distribution (OOD) scenarios and can generate Long-Wave Infrared (LWIR), Mid-Wave Infrared (MWIR), and Near-Infrared (NIR) translations from the same visible image. Code: this https URL.
Abstract (translated)
热成像是场景理解的关键，尤其是在低光和夜间条件下。然而，收集大规模的热数据集由于需要专门的红外图像捕捉设备而成本高昂且耗时长。为了解决这一挑战，研究人员探索了可见光到热成像的转换技术。大多数现有的方法依赖于生成对抗网络（GANs）或扩散模型（DMs），将任务视为风格迁移问题。因此，这些方法试图从有限的训练数据中学习模态分布变化和潜在的物理原理。 在本文中，我们提出了F-ViTA，这是一种新颖的方法，它利用基础模型中嵌入的一般世界知识来指导扩散过程以实现更优的转换效果。具体而言，我们将InstructPix2Pix扩散模型与来自SAM（Segment Anything Model）和Grounded DINO等基础模型的零样本掩码和标签相结合进行条件化处理。这样可以使模型学习场景中的物体与其在红外图像中的热特征之间的有意义的相关性。 我们在五个公共数据集上进行了广泛的实验，结果显示F-ViTA优于当前最先进（SOTA）的方法。此外，我们的模型能够在分布外（OOD）情景下表现出良好的泛化能力，并且可以从同一张可见光图像生成长波红外（LWIR）、中波红外（MWIR）和近红外（NIR）的转换结果。 代码地址：[这个链接](https://this https URL)
URL
https://arxiv.org/abs/2504.02801
