Abstract
Multispectral imaging plays a critical role in a range of intelligent transportation applications, including advanced driver assistance systems (ADAS), traffic monitoring, and night vision. However, accurate visible and thermal (RGB-T) image registration poses a significant challenge due to the considerable modality differences. In this paper, we present a novel joint Self-Correlation and Cross-Correspondence Estimation Framework (SC3EF), leveraging both local representative features and global contextual cues to effectively generate RGB-T correspondences. For this purpose, we design a convolution-transformer-based pipeline to extract local representative features and encode global correlations of intra-modality for inter-modality correspondence estimation between unaligned visible and thermal images. After merging the local and global correspondence estimation results, we further employ a hierarchical optical flow estimation decoder to progressively refine the estimated dense correspondence maps. Extensive experiments demonstrate the effectiveness of our proposed method, outperforming the current state-of-the-art (SOTA) methods on representative RGB-T datasets. Furthermore, it also shows competitive generalization capabilities across challenging scenarios, including large parallax, severe occlusions, adverse weather, and other cross-modal datasets (e.g., RGB-N and RGB-D).
Abstract (translated)
多光谱成像在包括高级驾驶辅助系统（ADAS）、交通监控和夜视在内的智能运输应用中发挥着关键作用。然而，由于模态差异显著，准确地注册可见光和热图像（RGB-T）构成了重大挑战。本文提出了一种新颖的联合自相关与跨模式对应估计框架（SC3EF），该框架利用局部代表性特征和全局上下文线索有效生成RGB-T对应的配对关系。为此，我们设计了一个基于卷积-变压器的管道，用于提取局部代表性特征，并编码同一模态内的全球关联以进行不同模态之间的未对齐可见光与热图像间的跨模式对应估计。在合并了局部和全局对应性估算结果后，进一步利用层次化光学流估计算法解码器逐步细化所估计的密集对应的映射图。 广泛的实验表明，我们提出的方法优于当前最先进的（SOTA）方法，在代表性的RGB-T数据集上取得了卓越的效果。此外，它还展示了在具有挑战性场景中的强大泛化能力，包括大视差、严重遮挡、恶劣天气及其他跨模态数据集（如RGB-N和RGB-D）。
URL
https://arxiv.org/abs/2504.12869
