Semantic_Segmentation
-
Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
2026-03-16 17:13:27Scott C. Lowe, Anthony Fuller, Sageev Oore, Evan Shelhamer, Graham W. TaylorAbstract
The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g., MAE) that reconstruct raw low-level data, and predictive approaches (e.g., I-JEPA) that predict high-level abstract embeddings. While generative methods provide strong grounding, they are computationally inefficient for high-redundancy modalities like imagery, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer from training instability due to their reliance on the non-stationary targets of final-layer self-distillation. We introduce Bootleg, a method that bridges this divide by tasking the model with predicting latent representations from multiple hidden layers of a teacher network. This hierarchical objective forces the model to capture features at varying levels of abstraction simultaneously. We demonstrate that Bootleg significantly outperforms comparable baselines (+10% over I-JEPA) on classification of ImageNet-1K and iNaturalist-21, and semantic segmentation of ADE20K and Cityscapes.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15553
PDF
-
Seeing Beyond: Extrapolative Domain Adaptive Panoramic Segmentation
2026-03-16 16:09:18Yuanfan Zheng, Kunyu Peng, Xu Zheng, Kailun YangAbstract
Cross-domain panoramic semantic segmentation has attracted growing interest as it enables comprehensive 360° scene understanding for real-world applications. However, it remains particularly challenging due to severe geometric Field of View (FoV) distortions and inconsistent open-set semantics across domains. In this work, we formulate an open-set domain adaptation setting, and propose Extrapolative Domain Adaptive Panoramic Segmentation (EDA-PSeg) framework that trains on local perspective views and tests on full 360° panoramic images, explicitly tackling both geometric FoV shifts across domains and semantic uncertainty arising from previously unseen classes. To this end, we propose the Euler-Margin Attention (EMA), which introduces an angular margin to enhance viewpoint-invariant semantic representation, while performing amplitude and phase modulation to improve generalization toward unseen classes. Additionally, we design the Graph Matching Adapter (GMA), which builds high-order graph relations to align shared semantics across FoV shifts while effectively separating novel categories through structural adaptation. Extensive experiments on four benchmark datasets under camera-shift, weather-condition, and open-set scenarios demonstrate that EDA-PSeg achieves state-of-the-art performance, robust generalization to diverse viewing geometries, and resilience under varying environmental conditions. The code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15475
PDF
-
A Tutorial on ALOS2 SAR Utilization: Dataset Preparation, Self-Supervised Pretraining, and Semantic Segmentation
2026-03-16 11:16:03Nevrez Imamoglu, Ali Caglayan, Toru KouyamaAbstract
Masked auto-encoders (MAE) and related approaches have shown promise for satellite imagery, but their application to synthetic aperture radar (SAR) remains limited due to challenges in semantic labeling and high noise levels. Building on our prior work with SAR-W-MixMAE, which adds SAR-specific intensity-weighted loss to standard MixMAE for pretraining, we also introduce SAR-W-SimMIM; a weighted variant of SimMIM applied to ALOS-2 single-channel SAR imagery. This method aims to reduce the impact of speckle and extreme intensity values during self-supervised pretraining. We evaluate its effect on semantic segmentation compared to our previous trial with SAR-W-MixMAE and random initialization, observing notable improvements. In addition, pretraining and fine-tuning models on satellite imagery pose unique challenges, particularly when developing region-specific models. Imbalanced land cover distributions such as dominant water, forest, or desert areas can introduce bias, affecting both pretraining and downstream tasks like land cover segmentation. To address this, we constructed a SAR dataset using ALOS-2 single-channel (HH polarization) imagery focused on the Japan region, marking the initial phase toward a national-scale foundation model. This dataset was used to pretrain a vision transformer-based autoencoder, with the resulting encoder fine-tuned for semantic segmentation using a task-specific decoder. Initial results demonstrate significant performance improvements compared to training from scratch with random initialization. In summary, this work provides a guide to process and prepare ALOS2 observations to create dataset so that it can be taken advantage of self-supervised pretraining of models and finetuning downstream tasks such as semantic segmentation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15119
PDF
-
Seeing Where to Deploy: Metric RGB-Based Traversability Analysis for Aerial-to-Ground Hidden Space Inspection
2026-03-15 22:20:59Seoyoung Lee, Shaekh Mohammad Shithil, Durgakant Pushp, Lantao Liu, Zhangyang WangAbstract
Inspection of confined infrastructure such as culverts often requires accessing hidden spaces whose entrances are reachable primarily from elevated viewpoints. Aerial-ground cooperation enables a UAV to deploy a compact UGV for interior exploration, but selecting a suitable deployment region from aerial observations requires metric terrain reasoning involving scale ambiguity, reconstruction uncertainty, and terrain semantics. We present a metric RGB-based geometric-semantic reconstruction and traversability analysis framework for aerial-to-ground hidden space inspection. A feed-forward multi-view RGB reconstruction backbone produces dense geometry, while temporally consistent semantic segmentation yields a 3D semantic map. To enable deployment-relevant measurements without LiDAR-based dense mapping, we introduce an embodied motion prior that recovers metric scale by enforcing consistency between predicted camera motion and onboard platform egomotion. From the metrically grounded reconstruction, we construct a confidence-aware geometric-semantic traversability map and evaluate candidate deployment zones under explicit reachability constraints. Experiments on a tethered UAV-UGV platform demonstrate reliable deployment-zone identification in hidden space scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14639
PDF
-
DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction
2026-03-14 13:58:56Jing Wang, Huimin Shi, Quan Zhou, Qibo Liu, Suofei Zhang, Huimin LuAbstract
The recent years have witnessed the remarkable development for open-vocabulary semantic segmentation (OVSS) using visual-language foundation models, yet still suffer from following fundamental challenges: (1) insufficient cross-modal communications between textual and visual spaces, and (2) significant computational costs from the interactions with massive number of categories. To address these issues, this paper describes a novel coarse-to-fine framework, called DCP-CLIP, for OVSS. Unlike prior efforts that mainly relied on pre-established category content and the inherent spatial-class interaction capability of CLIP, we dynamic constructing category-relevant textual features and explicitly models dual interactions between spatial image features and textual class semantics. Specifically, we first leverage CLIP's open-vocabulary recognition capability to identify semantic categories relevant to the image context, upon which we dynamically generate corresponding textual features to serve as initial textual guidance. Subsequently, we conduct a coarse segmentation by cross-modally integrating semantic information from textual guidance into the visual representations and achieve refined segmentation by integrating spatially enriched features from the encoder to recover fine-grained details and enhance spatial resolution. In final, we leverage spatial information from the segmentation side to refine category predictions for each mask, facilitating more precise semantic labeling. Experiments on multiple OVSS benchmarks demonstrate that DCP-CLIP outperforms existing methods by delivering both higher accuracy and greater efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13951
PDF
-
Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video
2026-03-14 12:00:55Yuting Tan, Xilong Cheng, Yunxiao Qin, Zhengnan Li, Jingjing ZhangAbstract
Humans develop visual intelligence through perceiving and interacting with their environment - a self-supervised learning process grounded in egocentric experience. Inspired by this, we ask how can artificial systems learn stable object representations from continuous, uncurated first-person videos without relying on manual annotations. This setting poses challenges of separating, recognizing, and persistently tracking objects amid clutter, occlusion, and ego-motion. We propose EgoViT, a unified vision Transformer framework designed to learn stable object representations from unlabeled egocentric video. EgoViT bootstraps this learning process by jointly discovering and stabilizing "proto-objects" through three synergistic mechanisms: (1) Proto-object Learning, which uses intra-frame distillation to form discriminative representations; (2) Depth Regularization, which grounds these representations in geometric structure; and (3) Teacher-Filtered Temporal Consistency, which enforces identity over time. This creates a virtuous cycle where initial object hypotheses are progressively refined into stable, persistent representations. The framework is trained end-to-end on unlabeled first-person videos and exhibits robustness to geometric priors of varied origin and quality. On standard benchmarks, EgoViT achieves +8.0% CorLoc improvement in unsupervised object discovery and +4.8% mIoU improvement in semantic segmentation, demonstrating its potential to lay a foundation for robust visual abstraction in embodied intelligence.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13912
PDF
-
Zero-Forgetting CISS via Dual-Phase Cognitive Cascades
2026-03-14 10:09:32Yuquan Lu, Yifu Guo, Zishan Xu, Siyu Zhang, Yu Huo, Siyue Chen, Siyan Wu, Chenghua Zhu, Ruixuan WangAbstract
Continual semantic segmentation (CSS) is a cornerstone task in computer vision that enables a large number of downstream applications, but faces the catastrophic forgetting challenge. In conventional class-incremental semantic segmentation (CISS) frameworks using Softmax-based classification heads, catastrophic forgetting originates from Catastrophic forgetting and task affiliation probability. We formulate these problems and provide a theoretical analysis to more deeply understand the limitations in existing CISS methods, particularly Strict Parameter Isolation (SPI). To address these challenges, we follow a dual-phase intuition from human annotators, and introduce Cognitive Cascade Segmentation (CogCaS), a novel dual-phase cascade formulation for CSS tasks in the CISS setting. By decoupling the task into class-existence detection and class-specific segmentation, CogCaS enables more effective continual learning, preserving previously learned knowledge while incorporating new classes. Using two benchmark datasets PASCAL VOC 2012 and ADE20K, we have shown significant improvements in a variety of challenging scenarios, particularly those with long sequence of incremental tasks, when compared to exsiting state-of-the-art methods. Our code will be made publicly available upon paper acceptance.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13874
PDF
-
Semantic Aware Feature Extraction for Enhanced 3D Reconstruction
2026-03-13 19:49:32Ronald Nap, Andy XiaoAbstract
Feature matching is a fundamental problem in computer vision with wide-ranging applications, including simultaneous localization and mapping (SLAM), image stitching, and 3D reconstruction. While recent advances in deep learning have improved keypoint detection and description, most approaches focus primarily on geometric attributes and often neglect higher-level semantic information. This work proposes a semantic-aware feature extraction framework that employs multi-task learning to jointly train keypoint detection, keypoint description, and semantic segmentation. The method is benchmarked against standard feature matching techniques and evaluated in the context of 3D reconstruction. To enhance feature correspondence, a deep matching module is integrated. The system is tested using input from a single monocular fisheye camera mounted on a vehicle and evaluated within a multi-floor parking structure. The proposed approach supports semantic 3D reconstruction with altitude estimation, capturing elevation changes and enabling multi-level mapping. Experimental results demonstrate that the method produces semantically annotated 3D point clouds with improved structural detail and elevation information, underscoring the effectiveness of joint training with semantic cues for more consistent feature matching and enhanced 3D reconstruction.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13556
PDF
-
AI Model Modulation with Logits Redistribution
2026-03-13 07:57:23Zihan Wang, Zhongkui Ma, Xinguo Feng, Zhiyang Mei, Ethan Ma, Derui Wang, Minhui Xue, Guangdong BaiAbstract
Large-scale models are typically adapted to meet the diverse requirements of model owners and users. However, maintaining multiple specialized versions of the model is inefficient. In response, we propose AIM, a novel model modulation paradigm that enables a single model to exhibit diverse behaviors to meet the specific end requirements. AIM enables two key modulation modes: utility and focus modulations. The former provides model owners with dynamic control over output quality to deliver varying utility levels, and the latter offers users precise control to shift model's focused input features. AIM introduces a logits redistribution strategy that operates in a training data-agnostic and retraining-free manner. We establish a formal foundation to ensure AIM's regulation capability, based on the statistical properties of logits ordering via joint probability distributions. Our evaluation confirms AIM's practicality and versatility for Al model modulation, with tasks spanning image classification, semantic segmentation and text generation, and prevalent architectures including ResNet, SegFormer and Llama.
Abstract (translated)
URL
https://arxiv.org/abs/2603.12755
PDF
-
CrossEarth-SAR: A SAR-Centric and Billion-Scale Geospatial Foundation Model for Domain Generalizable Semantic Segmentation
2026-03-12 14:52:35Ziqi Ye, Ziyang Gong, Ning Liao, Xiaoxing Hu, Di Wang, Hongruixuan Chen, Chen Huang, Yiguo He, Yuru Jia, Xiaoxing Wang, Haipeng Wang, Xue Yang, Junchi YanAbstract
Synthetic Aperture Radar (SAR) enables global, all-weather earth observation. However, owing to diverse imaging mechanisms, domain shifts across sensors and regions severely hinder its semantic generalization. To address this, we present CrossEarth-SAR, the first billion-scale SAR vision foundation model built upon a novel physics-guided sparse mixture-of-experts (MoE) architecture incorporating physical descriptors, explicitly designed for cross-domain semantic segmentation. To facilitate large-scale pre-training, we develop CrossEarth-SAR-200K, a weakly and fully supervised dataset that unifies public and private SAR imagery. We also introduce a benchmark suite comprising 22 sub-benchmarks across 8 distinct domain gaps, establishing the first unified standard for domain generalization semantic segmentation on SAR imagery. Extensive experiments demonstrate that CrossEarth-SAR achieves state-of-the-art results on 20 benchmarks, surpassing previous methods by over 10\% mIoU on some benchmarks under multi-gap transfer. All code, benchmark and datasets will be publicly available.
Abstract (translated)
合成孔径雷达（SAR）技术能够实现全球范围内的全天候地球观测。然而，由于成像机制的多样性及传感器与区域间的领域变化，这严重阻碍了其语义泛化能力。为解决这一问题，我们提出了CrossEarth-SAR，这是首个基于新颖物理引导稀疏专家混合架构（MoE）并融合物理描述符的大规模SAR视觉基础模型，该模型特别针对跨域语义分割设计。为了支持大规模预训练，我们开发了CrossEarth-SAR-200K数据集，这是一个将公共和私人SAR图像统一起来的弱监督与全监督混合的数据集。 此外，我们还引入了一套基准测试工具，包含涵盖8种不同领域差距共计22个子基准的评估系统，建立了首个针对SAR图像跨域泛化语义分割的标准。广泛的实验表明，在多间隙迁移条件下，CrossEarth-SAR在20个基准上实现了最新的性能表现，并且在某些基准上的mIoU指标比之前的最佳方法高出超过10%。 所有代码、基准和数据集都将公开提供使用。
URL
https://arxiv.org/abs/2603.12008
PDF
-
From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understanding
2026-03-10 17:51:12Wenzhao Xiang, Yue Wu, Hongyang Yu, Feng Gao, Fan Yang, Xilin ChenAbstract
Self-supervised visual pre-training methods face an inherent tension: contrastive learning (CL) captures global semantics but loses fine-grained detail, while masked image modeling (MIM) preserves local textures but suffers from "attention drift" due to semantically-agnostic random masking. We propose C2FMAE, a coarse-to-fine masked autoencoder that resolves this tension by explicitly learning hierarchical visual representations across three data granularities: semantic masks (scene-level), instance masks (object-level), and RGB images (pixel-level). Two synergistic innovations enforce a strict top-down learning principle. First, a cascaded decoder sequentially reconstructs from scene semantics to object instances to pixel details, establishing explicit cross-granularity dependencies that parallel decoders cannot capture. Second, a progressive masking curriculum dynamically shifts the training focus from semantic-guided to instance-guided and finally to random masking, creating a structured learning path from global context to local features. To support this framework, we construct a large-scale multi-granular dataset with high-quality pseudo-labels for all 1.28M ImageNet-1K images. Extensive experiments show that C2FMAE achieves significant performance gains on image classification, object detection, and semantic segmentation, validating the effectiveness of our hierarchical design in learning more robust and generalizable representations.
Abstract (translated)
自监督视觉预训练方法面临着固有的矛盾：对比学习（CL）能够捕捉全局语义信息，但会丢失细粒度的细节；而掩码图像建模（MIM）则能保留局部纹理特征，但由于采用无意义的随机掩码策略，会导致“注意力漂移”问题。为此，我们提出了一种名为C2FMAE（从粗到精的掩码自动编码器）的方法，通过显式学习三种数据粒度层次化的视觉表示来解决这一矛盾：语义掩码（场景级别）、实例掩码（对象级别）和RGB图像（像素级别）。为了严格贯彻自上而下的学习原则，C2FMAE引入了两个协同创新： 1. 级联解码器按顺序从场景语义重建至物体实例再到像素细节，建立了明确的跨粒度依赖关系，这是并行解码器所无法捕捉到的。 2. 动态掩码课程学习策略逐步将训练焦点由以语义指导为主转向以实例指导为主，最后转为随机掩码，从而构建了一个从全局上下文到局部特征的学习路径。 为了支持这一框架，我们还构建了大规模多粒度数据集，并为此数据集中所有128万张ImageNet-1K图像生成高质量的伪标签。广泛的实验结果表明，C2FMAE在图像分类、目标检测和语义分割任务上实现了显著性能提升，验证了我们的分层设计在学习更具鲁棒性和泛化能力表示的有效性。
URL
https://arxiv.org/abs/2603.09955
PDF
-
Grounding Synthetic Data Generation With Vision and Language Models
2026-03-10 13:03:53\"Umit Mert \c{C}a\u{g}lar, Alptekin TemizelAbstract
Deep learning models benefit from increasing data diversity and volume, motivating synthetic data augmentation to improve existing datasets. However, existing evaluation metrics for synthetic data typically calculate latent feature similarity, which is difficult to interpret and does not always correlate with the contribution to downstream tasks. We propose a vision-language grounded framework for interpretable synthetic data augmentation and evaluation in remote sensing. Our approach combines generative models, semantic segmentation and image captioning with vision and language models. Based on this framework, we introduce ARAS400k: A large-scale Remote sensing dataset Augmented with Synthetic data for segmentation and captioning, containing 100k real images and 300k synthetic images, each paired with segmentation maps and descriptions. ARAS400k enables the automated evaluation of synthetic data by analyzing semantic composition, minimizing caption redundancy, and verifying cross-modal consistency between visual structures and language descriptions. Experimental results indicate that while models trained exclusively on synthetic data reach competitive performance levels, those trained with augmented data (a combination of real and synthetic images) consistently outperform real-data baselines. Consequently, this work establishes a scalable benchmark for remote sensing tasks, specifically in semantic segmentation and image captioning. The dataset is available at this http URL and the code base at this http URL.
Abstract (translated)
深度学习模型从数据多样性和数量的增加中受益，这促使了合成数据增强以改善现有数据集。然而，现有的用于评估合成数据的方法通常计算潜在特征相似性，这种方法难以解释，并且并不总是与对下游任务的影响相关联。我们提出了一种基于视觉和语言的理解框架，旨在促进遥感领域的可解释的合成数据增强和评估。我们的方法结合了生成模型、语义分割以及图像描述（或称为图像配词）技术与视觉及语言模型。 基于这一框架，我们引入了ARAS400k：这是一个大型远程传感数据集，通过合成数据进行了扩充，用于语义分割和图像描述任务，该数据集中包含10万张真实图片和30万张合成图片，每一张图片都配有对应的语义分割图和描述文本。ARAS400k使得可以通过分析语义组成、减少描述的冗余性以及验证视觉结构与语言描述之间的跨模态一致性来进行自动化的合成数据评估。 实验结果显示，在仅使用合成数据训练模型可以达到竞争性的性能水平，但在结合真实图像和合成图像（即增强数据）进行训练时，这些模型始终优于基于真实数据的基线模型。因此，这项工作为遥感任务提供了可扩展的基准测试，特别是在语义分割和图像描述领域。 该数据集可在提供的链接获取，代码库也可在相应的网址上找到。
URL
https://arxiv.org/abs/2603.09625
PDF
-
SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation
2026-03-10 07:52:28Aodi Wu, Jianhong Zuo, Zeyuan Zhao, Xubo Luo, Ruisuo Wang, Xue WanAbstract
Autonomous space operations such as on-orbit servicing and active debris removal demand robust part-level semantic understanding and precise relative navigation of target spacecraft, yet collecting large-scale real data in orbit remains impractical due to cost and access constraints. Existing synthetic datasets, moreover, suffer from limited target diversity, single-modality sensing, and incomplete ground-truth annotations. We present \textbf{SpaceSense-Bench}, a large-scale multi-modal benchmark for spacecraft perception encompassing 136~satellite models with approximately 70~GB of data. Each frame provides time-synchronized 1024$\times$1024 RGB images, millimeter-precision depth maps, and 256-beam LiDAR point clouds, together with dense 7-class part-level semantic labels at both the pixel and point level as well as accurate 6-DoF pose ground truth. The dataset is generated through a high-fidelity space simulation built in Unreal Engine~5 and a fully automated pipeline covering data acquisition, multi-stage quality control, and conversion to mainstream formats. We benchmark five representative tasks (object detection, 2D semantic segmentation, RGB--LiDAR fusion-based 3D point cloud segmentation, monocular depth estimation, and orientation estimation) and identify two key findings: (i)~perceiving small-scale components (\emph{e.g.}, thrusters and omni-antennas) and generalizing to entirely unseen spacecraft in a zero-shot setting remain critical bottlenecks for current methods, and (ii)~scaling up the number of training satellites yields substantial performance gains on novel targets, underscoring the value of large-scale, diverse datasets for space perception research. The dataset, code, and toolkit are publicly available at this https URL.
Abstract (translated)
自主空间操作，如轨道服务和主动碎片清除，需要强大的部件级语义理解和目标航天器的精准相对导航。然而，由于成本和访问限制，在轨收集大规模真实数据仍然不切实际。现有的合成数据集还存在目标多样性有限、单一模态传感以及地面实况标注不完整的问题。我们提出了**SpaceSense-Bench**，这是一个涵盖136颗卫星模型的大型多模态基准库，包含大约70GB的数据。每个帧提供时间同步的1024×1024 RGB图像、毫米精度深度图和256束LiDAR点云，并附有像素级和点级密集的7类部件级语义标签以及精确的6自由度姿态地面实况标注。 该数据集通过在Unreal Engine 5中构建的高度逼真的空间模拟生成，并采用完全自动化的管道，涵盖数据采集、多阶段质量控制及转换为主流格式。我们针对五个代表性任务（物体检测、2D语义分割、基于RGB-LiDAR融合的3D点云分割、单目深度估计和姿态估计）进行了基准测试，并识别出两个关键发现：(i) 当前方法在感知小规模组件（例如，推进器和全向天线）以及零样本设置下泛化到完全未知的航天器方面仍然面临重大瓶颈；(ii) 增加训练卫星的数量会显著提高对新目标的表现，这凸显了大规模、多样化数据集对于空间感知研究的价值。该数据集、代码及工具包可在[公开网址]获取。
URL
https://arxiv.org/abs/2603.09320
PDF
-
RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation
2026-03-10 03:40:26Kunyu Tan, Mingjian LiangAbstract
RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light or dark environments. However, traditional approaches often overemphasize modality balance, resulting in limited robustness and severe performance degradation when sensor signals are partially missing. Recent advances such as cross-modal knowledge distillation and modality-adaptive fine-tuning attempt to enhance cross-modal interaction, but they typically decouple modality fusion and modality adaptation, requiring multi-stage training with frozen models or teacher-student frameworks. We present RTFDNet, a three-branch encoder-decoder that unifies fusion and decoupling for robust RGB-T segmentation. Synergistic Feature Fusion (SFF) performs channel-wise gated exchange and lightweight spatial attention to inject complementary cues. Cross-Modal Decouple Regularization (CMDR) isolates modality-specific components from the fused representation and supervises unimodal decoders via stop-gradient targets. Region Decouple Regularization (RDR) enforces class-selective prediction consistency in confident regions while blocking gradients to the fusion branch. This feedback loop strengthens unimodal paths without degrading the fused stream, enabling efficient standalone inference at test time. Extensive experiments demonstrate the effectiveness of RTFDNet, showing consistent performance across varying modality conditions. Our implementation will be released to facilitate further research. Our source code are publicly available at this https URL.
Abstract (translated)
RGB-Thermal（RGB-T）语义分割对于在低光或黑暗环境中运行的机器人系统至关重要。然而，传统方法往往过度强调模态平衡，在传感器信号部分缺失的情况下会导致有限的鲁棒性和严重的性能下降。最近的研究进展如跨模式知识蒸馏和模态自适应微调旨在增强跨模态互动，但它们通常将模态融合与模态适应解耦，需要使用冻结模型或教师-学生框架进行多阶段训练。 我们提出了一种名为RTFDNet的三分支编码器-解码器结构，它可以统一地处理融合和解耦以实现鲁棒的RGB-T分割。协同特征融合（Synergistic Feature Fusion, SFF）通过通道级门控交换和轻量级空间注意来注入互补线索。跨模态解耦正则化（Cross-Modal Decouple Regularization, CMDR）从融合表示中隔离出特定于某一种模态的组件，并通过停止梯度目标监督单模态解码器。区域解耦正则化（Region Decouple Regularization, RDR）在有信心的区域内强制执行类选择性预测一致性，同时阻止对融合分支的梯度传播。这一反馈回路增强了单模态路径而不降低融合流的质量，在测试时实现了高效的独立推理。 广泛实验表明了RTFDNet的有效性，并显示了其在不同模态条件下的一致性能表现。我们将会公开发布该实现以促进进一步的研究，源代码可在此URL获取（原文中提供的具体链接需替换为实际可用的地址）。
URL
https://arxiv.org/abs/2603.09149
PDF
-
Rotation Equivariant Mamba for Vision Tasks
2026-03-10 03:22:33Zhongchen Zhao, Qi Xie, Keyu Huang, Lei Zhang, Deyu Meng, Zongben XuAbstract
Rotation equivariance constitutes one of the most general and crucial structural priors for visual data, yet it remains notably absent from current Mamba-based vision architectures. Despite the success of Mamba in natural language processing and its growing adoption in computer vision, existing visual Mamba models fail to account for rotational symmetry in their design. This omission renders them inherently sensitive to image rotations, thereby constraining their robustness and cross-task generalization. To address this limitation, we propose to incorporate rotation symmetry, a universal and fundamental geometric prior in images, into Mamba-based architectures. Specifically, we introduce EQ-VMamba, the first rotation equivariant visual Mamba architecture for vision tasks. The core components of EQ-VMamba include a carefully designed rotation equivariant cross-scan strategy and group Mamba blocks. Moreover, we provide a rigorous theoretical analysis of the intrinsic equivariance error, demonstrating that the proposed architecture enforces end-to-end rotation equivariance throughout the network. Extensive experiments across multiple benchmarks - including high-level image classification task, mid-level semantic segmentation task, and low-level image super-resolution task - demonstrate that EQ-VMamba achieves superior or competitive performance compared to non-equivariant baselines, while requiring approximately 50% fewer parameters. These results indicate that embedding rotation equivariance not only effectively bolsters the robustness of visual Mamba models against rotation transformations, but also enhances overall performance with significantly improved parameter efficiency. Code is available at this https URL.
Abstract (translated)
旋转等变性构成了视觉数据中最通用和最关键的结构先验之一，然而它在当前基于Mamba的视觉架构中却显著缺席。尽管Mamba在自然语言处理领域取得了成功，并且逐渐被应用于计算机视觉，现有的视觉Mamba模型在其设计中未能考虑旋转对称性。这种遗漏使它们内在地容易受到图像旋转的影响，从而限制了其鲁棒性和跨任务泛化能力。 为了解决这一局限性，我们提议将旋转对称性——一种普遍而基本的几何先验，在基于Mamba的架构中融入其中。具体而言，我们提出了EQ-VMamba，这是首个针对视觉任务的旋转等变视觉Mamba架构。EQ-VMamba的核心组件包括精心设计的旋转等变交叉扫描策略和群组Mamba块。此外，我们还提供了关于内在等变误差的严格理论分析，表明所提出的架构在整个网络中执行端到端的旋转等变性。 在多个基准测试——包括高层次图像分类任务、中层次语义分割任务和低层次图像超分辨率任务上的广泛实验显示，EQ-VMamba相较于非等变基线模型，不仅实现了优越或具有竞争力的表现，还减少了大约50%的参数需求。这些结果表明，在视觉Mamba模型中嵌入旋转等变性不仅能有效增强其对旋转变换的鲁棒性，还能通过显著提高参数效率来提升整体性能。 代码可在此链接获取：[提供URL的地方]（请确保替换为实际可用的有效链接）。
URL
https://arxiv.org/abs/2603.09138
PDF
-
Weakly Supervised Teacher-Student Framework with Progressive Pseudo-mask Refinement for Gland Segmentation
2026-03-09 16:54:05Hikmat Khan, Wei Chen, Muhammad Khalid Khan NiaziAbstract
Background and objectives: Colorectal cancer histopathological grading depends on accurate segmentation of glandular structures. Current deep learning approaches rely on large scale pixel level annotations that are labor intensive and difficult to obtain in routine clinical practice. Weakly supervised semantic segmentation offers a promising alternative. However, class activation map based methods often produce incomplete pseudo masks that emphasize highly discriminative regions and fail to supervise unannotated glandular structures. We propose a weakly supervised teacher student framework that leverages sparse pathologist annotations and an Exponential Moving Average stabilized teacher network to generate refined pseudo masks. Methods: The framework integrates confidence based filtering, adaptive fusion of teacher predictions with limited ground truth, and curriculum guided refinement to progressively segment unannotated glandular regions. The method was evaluated on an institutional colorectal cancer cohort from The Ohio State University Wexner Medical Center consisting of 60 hematoxylin and eosin stained whole slide images and on public datasets including the Gland Segmentation dataset, TCGA COAD, TCGA READ, and SPIDER. Results: On the Gland Segmentation dataset the framework achieved a mean Intersection over Union of 80.10 and a mean Dice coefficient of 89.10. Cross cohort evaluation demonstrated robust generalization on TCGA COAD and TCGA READ without additional annotations, while reduced performance on SPIDER reflected domain shift. Conclusions: The proposed framework provides an annotation efficient and generalizable approach for gland segmentation in colorectal histopathology.
Abstract (translated)
背景与目标：结直肠癌的组织病理学分级依赖于腺体结构的精确分割。当前深度学习方法依赖于大规模像素级别的注释，这些注释在常规临床实践中耗时且难以获取。弱监督语义分割提供了一种有前景的替代方案。然而，基于类激活图的方法往往会产生不完整的伪掩模，过分强调具有高度区分性的区域，并且无法对未标注的腺体结构进行监督。我们提出了一种利用稀疏病理学家注释和指数移动平均稳定的教师网络生成精炼伪掩模的弱监督师生框架。 方法：该框架结合了基于置信度过滤、教师预测与有限真实标签自适应融合以及课程引导细化，逐步分割未标注的腺体区域。研究在来自俄亥俄州立大学韦克斯纳医学中心的一个机构结直肠癌队列上进行了评估，包括60张苏木精和伊红染色的全切片图像，并且还在公开数据集（如Gland Segmentation 数据集、TCGA COAD、TCGA READ 和 SPIDER）上进行了测试。 结果：在Gland Segmentation 数据集中，该框架实现了平均交并比为80.10%和平均Dice系数为89.10%。跨队列评估显示，在没有额外注释的情况下对TCGA COAD 和 TCGA READ 的泛化能力稳健，而SPIDER上的性能下降反映了域偏移。 结论：所提出的框架提供了一种注解效率高且具有广泛适用性的方法，用于结直肠组织病理学中的腺体分割。
URL
https://arxiv.org/abs/2603.08605
PDF
-
Real-Time Monocular Scene Analysis for UAV in Outdoor Environments
2026-03-09 14:08:50Yara AlaaEldinAbstract
In this thesis, we leverage monocular cameras on aerial robots to predict depth and semantic maps in low-altitude unstructured environments. We propose a joint deep-learning architecture, named Co-SemDepth, that can perform the two tasks accurately and rapidly, and validate its effectiveness on a variety of datasets. The training of neural networks requires an abundance of annotated data, and in the UAV field, the availability of such data is limited. We introduce a new synthetic dataset in this thesis, TopAir that contains images captured with a nadir view in outdoor environments at different altitudes, helping to fill the gap. While using synthetic data for the training is convenient, it raises issues when shifting to the real domain for testing. We conduct an extensive analytical study to assess the effect of several factors on the synthetic-to-real generalization. Co-SemDepth and TaskPrompter models are used for comparison in this study. The results reveal a superior generalization performance for Co-SemDepth in depth estimation and for TaskPrompter in semantic segmentation. Also, our analysis allows us to determine which training datasets lead to a better generalization. Moreover, to help attenuate the gap between the synthetic and real domains, image style transfer techniques are explored on aerial images to convert from the synthetic to the realistic style. Cycle-GAN and Diffusion models are employed. The results reveal that diffusion models are better in the synthetic to real style transfer. In the end, we focus on the marine domain and address its challenges. Co-SemDepth is trained on a collected synthetic marine data, called MidSea, and tested on both synthetic and real data. The results reveal good generalization performance of Co-SemDepth when tested on real data from the SMD dataset while further enhancement is needed on the MIT dataset.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13368
PDF
-
VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion
2026-03-09 10:27:08Zaidao Han, Risa Higashita, Jiang LiuAbstract
Camera-based 3D Semantic Scene Completion (SSC) is a critical task for autonomous driving and robotic scene understanding. It aims to infer a complete 3D volumetric representation of both semantics and geometry from a single image. Existing methods typically focus on end-to-end 2D-to-3D feature lifting and voxel completion. However, they often overlook the interference between high-confidence visible-region perception and low-confidence occluded-region reasoning caused by single-image input, which can lead to feature dilution and error propagation. To address these challenges, we introduce an offline Visible Region Label Extraction (VRLE) strategy that explicitly separates and extracts voxel-level supervision for visible regions from dense 3D ground truth. This strategy purifies the supervisory space for two complementary sub-tasks: visible-region perception and occluded-region reasoning. Building on this idea, we propose the Visible-Occluded Interactive Completion Network (VOIC), a novel dual-decoder framework that explicitly decouples SSC into visible-region semantic perception and occluded-region scene completion. VOIC first constructs a base 3D voxel representation by fusing image features with depth-derived occupancy. The visible decoder focuses on generating high-fidelity geometric and semantic priors, while the occlusion decoder leverages these priors together with cross-modal interaction to perform coherent global scene reasoning. Extensive experiments on the SemanticKITTI and SSCBench-KITTI360 benchmarks demonstrate that VOIC outperforms existing monocular SSC methods in both geometric completion and semantic segmentation accuracy, achieving state-of-the-art performance.
Abstract (translated)
基于相机的三维语义场景完成（SSC）是自主驾驶和机器人场景理解中的关键任务，其目标是从单张图像中推断出完整的3D体积表示，涵盖几何和语义信息。现有的方法通常集中于端到端的2D至3D特征提升及体素补全。然而，这些方法往往忽视了由单一图像输入所导致的高置信度可视区域感知与低置信度被遮挡区域推理之间的干扰问题，这些问题可能导致特征稀释和误差传播。 为了解决这些挑战，我们提出了一种离线可见区域标签提取（VRLE）策略，该策略明确地将密集3D真实标签中的体素级监督分离并抽取给可视区域。这一策略通过精炼两种互补子任务的指导空间来提高准确性和鲁棒性：可视区域感知和遮挡区域推理。 在此基础上，我们提出了可见-遮挡交互完成网络（VOIC），这是一种新颖的双解码器框架，它明确地将SSC拆分为两个部分：可视区域语义感知与被遮挡区域场景补全。VOIC首先通过融合图像特征与深度衍生的占用信息来构建基础3D体素表示。可见区域解码器专注于生成高保真度的几何和语义先验，而遮挡区域解码器则利用这些先验及跨模态交互进行一致的整体场景推理。 在SemanticKITTI和SSCBench-KITTI360基准测试中的广泛实验表明，VOIC优于现有的单目SSC方法，在几何补全和语义分割精度方面均达到了最先进的性能。
URL
https://arxiv.org/abs/2512.18954
PDF
-
Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance
2026-03-08 10:09:26Guodong Sun, Junjie Liu, Gaoyang Zhang, Bo Wu, Yang ZhangAbstract
Scene understanding plays a critical role in enabling intelligence and autonomy in robotic systems. Traditional approaches often face challenges, including occlusions, ambiguous boundaries, and the inability to adapt attention based on task-specific requirements and sample variations. To address these limitations, this paper presents an efficient RGB-D scene understanding model that performs a range of tasks, including semantic segmentation, instance segmentation, orientation estimation, panoptic segmentation, and scene classification. The proposed model incorporates an enhanced fusion encoder, which effectively leverages redundant information from both RGB and depth inputs. For semantic segmentation, we introduce normalized focus channel layers and a context feature interaction layer, designed to mitigate issues such as shallow feature misguidance and insufficient local-global feature representation. The instance segmentation task benefits from a non-bottleneck 1D structure, which achieves superior contour representation with fewer parameters. Additionally, we propose a multi-task adaptive loss function that dynamically adjusts the learning strategy for different tasks based on scene variations. Extensive experiments on the NYUv2, SUN RGB-D, and Cityscapes datasets demonstrate that our approach outperforms existing methods in both segmentation accuracy and processing speed.
Abstract (translated)
场景理解在增强机器人系统的智能和自主性方面扮演着关键角色。传统方法经常面临遮挡、边界模糊以及无法根据任务特定需求和样本变化调整注意力等问题的挑战。为了克服这些限制，本文提出了一种高效的RGB-D场景理解模型，该模型能够执行一系列任务，包括语义分割、实例分割、姿态估计、全景分割和场景分类。所提出的模型包含了一个增强融合编码器，可以有效利用来自RGB和深度输入数据的冗余信息。 对于语义分割任务，我们引入了标准化关注通道层和上下文特征交互层，旨在解决浅层特征误导以及局部-全局特征表示不足等问题。在实例分割任务中，我们的模型采用了一种非瓶颈1D结构，在减少参数的同时实现了更优的轮廓表示。此外，我们还提出了一种多任务自适应损失函数，可以根据场景变化动态调整不同的学习策略。 通过在NYUv2、SUN RGB-D和Cityscapes数据集上的大量实验验证，我们的方法不仅在分割准确性上超越了现有技术，而且在处理速度方面也表现出色。
URL
https://arxiv.org/abs/2603.07570
PDF
-
SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing
2026-03-08 04:55:41Xiaokang Zhang, Bo Li, Chufeng Zhou, Weikang Yu, Lefei ZhangAbstract
Pretraining and fine-tuning have emerged as a new paradigm in remote sensing image interpretation. Among them, Masked Autoencoder (MAE)-based pretraining stands out for its strong capability to learn general feature representations via reconstructing masked image regions. However, applying MAE to multispectral remote sensing images remains challenging due to complex backgrounds, indistinct targets, and the lack of semantic guidance during masking, which hinders the learning of underlying structures and meaningful spatial-spectral features. To address this, we propose a simple yet effective approach, Spectral Index-Guided MAE (SIGMAE), for multispectral image pretraining. The core idea is to incorporate domain-specific spectral indices as prior knowledge to guide dynamic token masking toward informative regions. SIGMAE introduces Semantic Saliency-Guided Dynamic Token Masking (SSDTM), a curriculum-style strategy that quantifies each patch's semantic richness and internal heterogeneity to adaptively select the most informative tokens during training. By prioritizing semantically salient regions and progressively increasing sample difficulty, SSDTM enhances spectrally rich and structurally aware representation learning, mitigates overfitting, and reduces redundant computation compared with random masking. Extensive experiments on five widely used datasets covering various downstream tasks, including scene classification, semantic segmentation, object extraction and change detection, demonstrate that SIGMAE outperforms other pretrained geospatial foundation models. Moreover, it exhibits strong spatial-spectral reconstruction capability, even with a 90% mask ratio, and improves complex target recognition under limited labeled data. The source codes and model weights will be released at this https URL.
Abstract (translated)
预训练和微调已成为遥感图像解释的新范式。其中，基于遮罩自动编码器（MAE）的预训练因其通过重构被遮罩的图像区域来学习通用特征表示的能力而脱颖而出。然而，由于复杂的背景、不清晰的目标以及在遮罩过程中缺乏语义指导，将MAE应用于多光谱遥感图像仍然具有挑战性，这阻碍了底层结构和有意义的空间-光谱特征的学习。为了解决这个问题，我们提出了一种简单而有效的方法——基于光谱指数的MAE（SIGMAE），用于多光谱图像预训练。核心思想是将领域特定的光谱指数作为先验知识，以指导动态标记遮罩向信息丰富的区域进行导向。 SIGMAE引入了语义显著性引导的动态令牌遮罩（SSDTM）策略，这是一种类似课程的学习方式，它量化每个补丁的语义丰富性和内部异质性，并在训练过程中自适应地选择最富有信息性的令牌。通过优先考虑语义显著区域并逐步增加样本难度，SSDTM增强了光谱丰富的和结构感知表示学习，减轻了过拟合问题，并与随机遮罩相比减少了冗余计算。 在五个广泛使用的数据集上进行了大量实验，这些数据集涵盖了包括场景分类、语义分割、目标提取和变化检测在内的各种下游任务。结果表明，SIGMAE优于其他预训练的地理空间基础模型。此外，即使在90%的遮罩比例下，它也展示了强大的光谱-空间重建能力，并且在标签有限的情况下提高了复杂目标识别的能力。 源代码和模型权重将在以下链接发布：[提供URL]
URL
https://arxiv.org/abs/2603.07463
PDF
