Abstract
Remote Sensing Image Change Captioning (RSICC) aims to identify surface changes in multi-temporal remote sensing images and describe them in natural language. Current methods typically rely on an encoder-decoder architecture and focus on designing a sophisticated neck to process bi-temporal features extracted by the backbone. Recently, State Space Models (SSMs), especially Mamba, have demonstrated outstanding performance in many fields, owing to their efficient feature-selective modelling capability. However, their potential in the RSICC task remains unexplored. In this paper, we introduce Mamba into RSICC and propose a novel approach called RSCaMa (Remote Sensing Change Captioning Mamba). Specifically, we utilize Siamese backbones to extract bi-temporal features, which are then processed through multiple CaMa layers consisting of Spatial Difference-guided SSM (SD-SSM) and Temporal Traveling SSM (TT-SSM). SD-SSM uses differential features to enhance change perception, while TT-SSM promotes bitemporal interactions in a token-wise cross-scanning manner. Experimental results validate the effectiveness of CaMa layers and demonstrate the superior performance of RSCaMa, as well as the potential of Mamba in the RSICC task. Additionally, we systematically compare the effects of three language decoders, including Mamba, GPT-style decoder with causal attention mechanism, and Transformer decoder with cross-attention mechanism. This provides valuable insights for future RSICC research. The code will be available at this https URL
Abstract (translated)
远地遥感图像变化捕捉（RSICC）旨在通过自然语言描述多时程遥感图像表面的变化。目前的方法通常依赖于编码器-解码器架构，并专注于设计一个复杂的颈以处理由骨干网络提取的生物时程特征。最近，状态空间模型（SSMs）特别是Mamba在许多领域表现出卓越的性能，因为它们具有高效的特征选择建模能力。然而，在RSICC任务中，它们的表现潜力仍未经探索。在本文中，我们将Mamba引入RSICC，并提出了名为RSCaMa（远程遥感变化捕捉Mamba）的新方法。具体来说，我们利用序列到序列（Siamese）网络提取生物时程特征，然后通过多个CaMa层进行处理，包括空间差异引导的状态空间模型（SD-SSM）和时间旅行状态空间模型（TT-SSM）。SD-SSM使用差分特征来增强变化感知，而TT-SSM通过逐个扫描的token级跨扫描方式促进比特时相互作用。实验结果验证了CaMa层的有效性，并证明了RSCaMa以及Mamba在RSICC任务中的卓越表现。此外，我们系统地比较了三种语言解码器，包括Mamba、具有因果注意力机制的GPT风格解码器和具有跨注意力机制的Transformer解码器。这为未来的RSICC研究提供了宝贵的洞见。代码将在此处https:// URL中提供。
URL
https://arxiv.org/abs/2404.18895
