Abstract
Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that either use modality-specific adaptations or rely on loosely coupled interactions, thereby limiting flexibility and resulting in less effective cross-modal coordination. Moreover, these methods often struggle to balance efficient information exchange with preserving the unique characteristics of each modality across different modality combinations. To address these challenges, we propose CrossWeaver, a simple yet effective multimodal fusion framework for arbitrary-modality semantic segmentation. Its core is a Modality Interaction Block (MIB), which enables selective and reliability-aware cross-modal interaction within the encoder, while a lightweight Seam-Aligned Fusion (SAF) module further aggregates the enhanced features. Extensive experiments on multiple multimodal semantic segmentation benchmarks demonstrate that our framework achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.
Abstract (translated)
多模态语义分割在利用不同感知模态间的互补信息方面展现出巨大潜力。然而，现有方法通常依赖于精心设计的融合策略，这些策略要么采用模态特定适配，要么依赖松散耦合的交互，从而限制了灵活性并导致跨模态协调效果不佳。此外，这些方法往往难以在高效信息交换与保持不同模态组合中各模态独特特性之间取得平衡。为应对这些挑战，我们提出了CrossWeaver——一个简单而有效的任意模态语义分割多模态融合框架。其核心是模态交互块（MIB），它能够在编码器内部实现选择性且感知可靠性的跨模态交互，同时轻量级的接缝对齐融合（SAF）模块进一步聚合增强后的特征。在多个多模态语义分割基准数据集上的大量实验表明，我们的框架以极少的额外参数实现了最先进的性能，并能强泛化至未见过的模态组合。
URL
https://arxiv.org/abs/2604.02948
