Abstract
Real-world infrared imagery presents unique challenges for vision-language models due to the scarcity of aligned text data and domain-specific characteristics. Although existing methods have advanced the field, their reliance on synthetic infrared images generated through style transfer from visible images, which limits their ability to capture the unique characteristics of the infrared modality. To address this, we propose IRGPT, the first multi-modal large language model for real-world infrared images, built upon a large-scale InfraRed-Text Dataset (IR-TD) comprising over 260K authentic image-text pairs. The proposed IR-TD dataset contains real infrared images paired with meticulously handcrafted texts, where the initial drafts originated from two complementary processes: (1) LLM-generated descriptions of visible images, and (2) rule-based descriptions of annotations. Furthermore, we introduce a bi-cross-modal curriculum transfer learning strategy that systematically transfers knowledge from visible to infrared domains by considering the difficulty scores of both infrared-visible and infrared-text. Evaluated on a benchmark of 9 tasks (e.g., recognition, grounding), IRGPT achieves state-of-the-art performance even compared with larger-scale models.
Abstract (translated)
现实世界中的红外图像对视觉语言模型提出了独特的挑战，由于缺乏与之匹配的文本数据以及特定领域的特性。尽管现有方法已经推动了这一领域的发展，但它们依赖于通过可见光图像进行风格迁移生成的人工合成红外图像，这限制了这些模型捕捉红外模态独特特征的能力。 为了解决这个问题，我们提出了IRGPT——第一个用于真实世界红外图像的多模态大型语言模型，并且建立在大规模的“红外-文本数据集”（IR-TD）之上。该数据集包含超过260,000对真实的红外图像和精心手工制作的文字描述。所提出的IR-TD数据集包括实际的红外图像与文本配对，其初始草稿来源于两种互补的过程：(1) 大型语言模型生成可见光图像的描述；(2) 基于规则的注释描述。 此外，我们还引入了一种双向跨模态课程迁移学习策略，该策略通过考虑红外和可见光域之间以及红外和文本之间的难度得分，系统地将知识从可见光领域转移到红外领域。在涵盖9项任务（例如识别、定位等）的标准基准测试上进行评估时，IRGPT即使与更大规模的模型相比也表现出最先进的性能。
URL
https://arxiv.org/abs/2507.14449
