Abstract
Semantic segmentation is a fundamental task in computer vision with wide-ranging applications, including autonomous driving and robotics. While RGB-based methods have achieved strong performance with CNNs and Transformers, their effectiveness degrades under fast motion, low-light, or high dynamic range conditions due to limitations of frame cameras. Event cameras offer complementary advantages such as high temporal resolution and low latency, yet lack color and texture, making them insufficient on their own. To address this, recent research has explored multimodal fusion of RGB and event data; however, many existing approaches are computationally expensive and focus primarily on spatial fusion, neglecting the temporal dynamics inherent in event streams. In this work, we propose MambaSeg, a novel dual-branch semantic segmentation framework that employs parallel Mamba encoders to efficiently model RGB images and event streams. To reduce cross-modal ambiguity, we introduce the Dual-Dimensional Interaction Module (DDIM), comprising a Cross-Spatial Interaction Module (CSIM) and a Cross-Temporal Interaction Module (CTIM), which jointly perform fine-grained fusion along both spatial and temporal dimensions. This design improves cross-modal alignment, reduces ambiguity, and leverages the complementary properties of each modality. Extensive experiments on the DDD17 and DSEC datasets demonstrate that MambaSeg achieves state-of-the-art segmentation performance while significantly reducing computational cost, showcasing its promise for efficient, scalable, and robust multimodal perception.
Abstract (translated)
语义分割是计算机视觉中的基本任务，具有广泛的应用，包括自动驾驶和机器人技术。基于RGB的方法在使用卷积神经网络（CNN）和变压器的情况下已经取得了强大的性能表现，但在快速运动、低光或高动态范围条件下，由于帧摄像头的局限性，其效果会下降。事件相机提供了诸如高时间分辨率和低延迟等互补优势，但缺乏颜色和纹理信息，在单独使用时是不够的。为了解决这个问题，最近的研究探索了RGB与事件数据的多模态融合；然而，许多现有的方法计算成本高昂，并主要集中在空间融合上，忽视了事件流中的固有时间动态特性。 在本工作中，我们提出了MambaSeg，这是一种新颖的双分支语义分割框架，它使用并行的Mamba编码器来有效地建模RGB图像和事件流。为了减少跨模态模糊性，我们引入了双维交互模块（DDIM），包括一个跨空间交互模块（CSIM）和一个跨时间交互模块（CTIM）。这两个模块共同沿空间和时间维度执行细粒度融合。这种设计提高了跨模式对齐能力，减少了混淆，并利用了每种模态的互补特性。在DDD17和DSEC数据集上进行的一系列实验表明，MambaSeg能够实现最先进的分割性能并显著降低计算成本，展示了其在高效、可扩展且稳健的多模态感知中的潜力。
URL
https://arxiv.org/abs/2512.24243
