Abstract
RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light or dark environments. However, traditional approaches often overemphasize modality balance, resulting in limited robustness and severe performance degradation when sensor signals are partially missing. Recent advances such as cross-modal knowledge distillation and modality-adaptive fine-tuning attempt to enhance cross-modal interaction, but they typically decouple modality fusion and modality adaptation, requiring multi-stage training with frozen models or teacher-student frameworks. We present RTFDNet, a three-branch encoder-decoder that unifies fusion and decoupling for robust RGB-T segmentation. Synergistic Feature Fusion (SFF) performs channel-wise gated exchange and lightweight spatial attention to inject complementary cues. Cross-Modal Decouple Regularization (CMDR) isolates modality-specific components from the fused representation and supervises unimodal decoders via stop-gradient targets. Region Decouple Regularization (RDR) enforces class-selective prediction consistency in confident regions while blocking gradients to the fusion branch. This feedback loop strengthens unimodal paths without degrading the fused stream, enabling efficient standalone inference at test time. Extensive experiments demonstrate the effectiveness of RTFDNet, showing consistent performance across varying modality conditions. Our implementation will be released to facilitate further research. Our source code are publicly available at this https URL.
Abstract (translated)
RGB-Thermal（RGB-T）语义分割对于在低光或黑暗环境中运行的机器人系统至关重要。然而，传统方法往往过度强调模态平衡，在传感器信号部分缺失的情况下会导致有限的鲁棒性和严重的性能下降。最近的研究进展如跨模式知识蒸馏和模态自适应微调旨在增强跨模态互动，但它们通常将模态融合与模态适应解耦，需要使用冻结模型或教师-学生框架进行多阶段训练。 我们提出了一种名为RTFDNet的三分支编码器-解码器结构，它可以统一地处理融合和解耦以实现鲁棒的RGB-T分割。协同特征融合（Synergistic Feature Fusion, SFF）通过通道级门控交换和轻量级空间注意来注入互补线索。跨模态解耦正则化（Cross-Modal Decouple Regularization, CMDR）从融合表示中隔离出特定于某一种模态的组件，并通过停止梯度目标监督单模态解码器。区域解耦正则化（Region Decouple Regularization, RDR）在有信心的区域内强制执行类选择性预测一致性，同时阻止对融合分支的梯度传播。这一反馈回路增强了单模态路径而不降低融合流的质量，在测试时实现了高效的独立推理。 广泛实验表明了RTFDNet的有效性，并显示了其在不同模态条件下的一致性能表现。我们将会公开发布该实现以促进进一步的研究，源代码可在此URL获取（原文中提供的具体链接需替换为实际可用的地址）。
URL
https://arxiv.org/abs/2603.09149
