Abstract
Pan-sharpening aims to generate high-resolution multispectral (HRMS) images by integrating a high-resolution panchromatic (PAN) image with its corresponding low-resolution multispectral (MS) image. To achieve effective fusion, it is crucial to fully exploit the complementary information between the two modalities. Traditional CNN-based methods typically rely on channel-wise concatenation with fixed convolutional operators, which limits their adaptability to diverse spatial and spectral variations. While cross-attention mechanisms enable global interactions, they are computationally inefficient and may dilute fine-grained correspondences, making it difficult to capture complex semantic relationships. Recent advances in the Multimodal Diffusion Transformer (MMDiT) architecture have demonstrated impressive success in image generation and editing tasks. Unlike cross-attention, MMDiT employs in-context conditioning to facilitate more direct and efficient cross-modal information exchange. In this paper, we propose MMMamba, a cross-modal in-context fusion framework for pan-sharpening, with the flexibility to support image super-resolution in a zero-shot manner. Built upon the Mamba architecture, our design ensures linear computational complexity while maintaining strong cross-modal interaction capacity. Furthermore, we introduce a novel multimodal interleaved (MI) scanning mechanism that facilitates effective information exchange between the PAN and MS modalities. Extensive experiments demonstrate the superior performance of our method compared to existing state-of-the-art (SOTA) techniques across multiple tasks and benchmarks.
Abstract (translated)
光谱锐化（Pan-sharpening）的目标是通过整合高分辨率全色（PAN）图像和对应的低分辨率多光谱（MS）图像，生成高质量的高分辨多光谱（HRMS）图像。为了实现有效的融合，充分挖掘这两种模态之间的互补信息至关重要。传统的基于CNN的方法通常依赖于通道级串联以及固定的卷积操作符，这限制了它们适应各种空间和光谱变化的能力。虽然交叉注意力机制能够提供全局交互作用，但其计算效率较低，并且可能稀释精细对应关系，难以捕捉复杂的语义关系。 最近，在多模态扩散变压器（MMDiT）架构上的进展在图像生成和编辑任务中取得了令人印象深刻的成功表现。不同于交叉注意力机制，MMDiT采用上下文条件化以促进更加直接高效的跨模态信息交换。本文提出了一种新的光谱锐化框架——MMMamba，这是一种基于Mamba架构的跨模态上下文融合框架，并且能够支持零样本图像超分辨率任务。 我们的设计在确保线性计算复杂度的同时，仍然保持了强大的跨模态交互能力。此外，我们还引入了一个新颖的多模态交错（MI）扫描机制，以促进PAN和MS模态之间的有效信息交换。广泛的实验结果证明了我们的方法在多个任务和基准上的性能优于现有的最先进（SOTA）技术。
URL
https://arxiv.org/abs/2512.15261
