Abstract
Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation. However, practical systems often encounter missing modalities due to sensor failures or incomplete coverage, termed Incomplete Multimodal Semantic Segmentation (IMSS). IMSS faces three key challenges: (1) multimodal imbalance, where dominant modalities suppress fragile ones; (2) intra-class variation in scale, shape, and orientation across modalities; and (3) cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses. Existing methods rely on contrastive learning or joint optimization, which risk over-alignment, discarding modality-specific cues or imbalanced training, favoring robust modalities, while largely overlooking intra-class variation and cross-modal heterogeneity. To address these limitations, we propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance. SGMA introduces two complementary plug-and-play modules: (1) Semantic-Guided Fusion (SGF) module extracts multi-scale, class-wise semantic prototypes that capture consistent categorical representations across modalities, estimates per-modality robustness based on prototype-feature alignment, and performs adaptive fusion weighted by robustness scores to mitigate intra-class variation and cross-modal heterogeneity; (2) Modality-Aware Sampling (MAS) module leverages robustness estimations from SGF to dynamically reweight training samples, prioritizing challenging samples from fragile modalities to address modality imbalance. Extensive experiments across multiple datasets and backbones demonstrate that SGMA consistently outperforms state-of-the-art methods, with particularly significant improvements in fragile modalities.
Abstract (translated)
多模态语义分割通过整合来自不同传感器的互补信息，用于遥感地球观测。然而，在实际系统中，由于传感器故障或覆盖不完整，经常会遇到缺失模式的情况，这种情况被称为不完整的多模态语义分割（IMSS）。IMSS面临三大挑战：(1) 多模态不平衡，其中主导模式会压制脆弱模式；(2) 在不同模式下的同一类别对象在大小、形状和方向上的变化；(3) 跨模态异质性导致冲突的线索产生不一致的语义响应。现有的方法主要依赖于对比学习或联合优化，这些方法有过度对齐的风险，可能会忽略特定模态的信息或者由于训练不平衡而偏向稳健模式，同时很大程度上忽视了类别内部变化和跨模态差异。为解决这些问题，我们提出了基于语义引导的模态感知（SGMA）框架，该框架确保多模态学习平衡的同时减少类内变异性并通过语义指导来协调跨模态不一致性。 SGMA引入了两个互补的即插即用模块： (1) 语义引导融合(SGF)模块：提取跨尺度、类别级别的语义原型以捕捉不同模式下的连贯分类表示，基于原型-特征对齐估计每个模式的稳健性，并根据稳健性分数执行自适应融合以缓解类内变异性及跨模态异质性； (2) 模态感知采样(MAS)模块：利用SGF中的稳健性评估动态重新加权训练样本，优先考虑来自脆弱模态的挑战性样本，解决模态不平衡问题。 通过多个数据集和骨干网络进行广泛的实验表明，SGMA在各种方法中始终表现出色，并且对于脆弱模式改进尤为显著。
URL
https://arxiv.org/abs/2603.02505
