Semantic_Segmentation
-
Local Precise Refinement: A Dual-Gated Mixture-of-Experts for Enhancing Foundation Model Generalization against Spectral Shifts
2026-03-08 04:53:05Xi Chen, Maojun Zhang, Yu Liu, Shen YanAbstract
Domain Generalization Semantic Segmentation (DGSS) in spectral remote sensing is severely challenged by spectral shifts across diverse acquisition conditions, which cause significant performance degradation for models deployed in unseen domains. While Parameter-Efficient Fine-Tuning (PEFT) on foundation models is a promising direction, existing methods employ global, homogeneous adjustments. This "one-size-fits-all" tuning struggles with the spatial heterogeneity of land cover, causing semantic confusion. We argue that the key to robust DGSS lies not in a single global adaptation, but in performing fine-grained, spatially-adaptive refinement of a foundation model's features. To achieve this, we propose SpectralMoE, a novel PEFT framework for DGSS. It operationalizes this principle by utilizing a Mixture-of-Experts (MoE) architecture to perform local precise refinement on the foundation model's features, incorporating depth features estimated from selected RGB bands of the spectral remote sensing imagery to guide the fine-tuning process. Specifically, SpectralMoE employs a dual-gated MoE architecture that independently routes visual and depth features to top-k selected experts for specialized refinement, enabling modality-specific adjustments. A subsequent cross-attention mechanism then judiciously fuses the refined structural cues into the visual stream, mitigating semantic ambiguities caused by spectral variations. Extensive experiments show that SpectralMoE sets a new state-of-the-art on multiple DGSS benchmarks across hyperspectral, multispectral, and RGB remote sensing imagery.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13352
PDF
-
Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation
2026-03-06 15:25:20Jonas Ernst, Wolfgang Boettcher, Lukas Hoyer, Jan Eric Lenssen, Bernt SchieleAbstract
We present Rewis3d, a framework that leverages recent advances in feed-forward 3D reconstruction to significantly improve weakly supervised semantic segmentation on 2D images. Obtaining dense, pixel-level annotations remains a costly bottleneck for training segmentation models. Alleviating this issue, sparse annotations offer an efficient weakly-supervised alternative. However, they still incur a performance gap. To address this, we introduce a novel approach that leverages 3D scene reconstruction as an auxiliary supervisory signal. Our key insight is that 3D geometric structure recovered from 2D videos provides strong cues that can propagate sparse annotations across entire scenes. Specifically, a dual student-teacher architecture enforces semantic consistency between 2D images and reconstructed 3D point clouds, using state-of-the-art feed-forward reconstruction to generate reliable geometric supervision. Extensive experiments demonstrate that Rewis3d achieves state-of-the-art performance in sparse supervision, outperforming existing approaches by 2-7% without requiring additional labels or inference overhead.
Abstract (translated)
我们介绍了一个名为Rewis3d的框架，该框架利用近期在前馈式三维重建方面的进展，显著提升了二维图像上弱监督语义分割的效果。获得密集、像素级别的注释仍然是训练分割模型的一个昂贵瓶颈问题。为了解决这个问题，稀疏标注提供了一种有效的弱监督替代方案。然而，这种做法仍然会导致性能差距。为此，我们引入了一个新颖的方法，该方法利用三维场景重建作为辅助监督信号。 我们的关键见解是，从二维视频中恢复出的三维几何结构可以提供强大的线索，从而将稀疏注释传播到整个场景中。具体来说，一个双学生的教师架构强制执行二维图像和重构的三维点云之间的语义一致性，并使用最先进的前馈式重建来生成可靠的几何监督。 广泛的实验表明，Rewis3d在稀疏监督下达到了最先进的性能，相比现有方法提高了2-7%，并且不需要额外的标签或推理开销。
URL
https://arxiv.org/abs/2603.06374
PDF
-
P-SLCR: Unsupervised Point Cloud Semantic Segmentation via Prototypes Structure Learning and Consistent Reasoning
2026-03-06 14:23:10Lixin Zhan, Jie Jiang, Tianjian Zhou, Yukun Du, Yan Zheng, Xuehu DuanAbstract
Current semantic segmentation approaches for point cloud scenes heavily rely on manual labeling, while research on unsupervised semantic segmentation methods specifically for raw point clouds is still in its early stages. Unsupervised point cloud learning poses significant challenges due to the absence of annotation information and the lack of pre-training. The development of effective strategies is crucial in this context. In this paper, we propose a novel prototype library-driven unsupervised point cloud semantic segmentation strategy that utilizes Structure Learning and Consistent Reasoning (P-SLCR). First, we propose a Consistent Structure Learning to establish structural feature learning between consistent points and the library of consistent prototypes by selecting high-quality features. Second, we propose a Semantic Relation Consistent Reasoning that constructs a prototype inter-relation matrix between consistent and ambiguous prototype libraries separately. This process ensures the preservation of semantic consistency by imposing constraints on consistent and ambiguous prototype libraries through the prototype inter-relation matrix. Finally, our method was extensively evaluated on the S3DIS, SemanticKITTI, and Scannet datasets, achieving the best performance compared to unsupervised methods. Specifically, the mIoU of 47.1% is achieved for Area-5 of the S3DIS dataset, surpassing the classical fully supervised method PointNet by 2.5%.
Abstract (translated)
目前针对点云场景的语义分割方法严重依赖于人工标注，而专门用于原始点云数据的无监督语义分割方法的研究仍处于初级阶段。由于缺乏注释信息和预训练数据，无监督点云学习面临着重大挑战。在这种背景下，开发有效的策略至关重要。 本文提出了一种新颖的方法：原型库驱动的无监督点云语义分割策略（P-SLCR），该方法结合了结构学习和一致性推理。首先，我们提出了一个一致结构学习模块，通过选择高质量特征，在一致点与一致原型库之间建立结构性特征学习。其次，我们提出了一致性关系推理模块，它分别构建两个独立的原型互相关矩阵：一致原型库和模棱两可原型库之间的矩阵。这一过程确保了语义一致性，通过对一致和模棱两可原型库进行约束来维持原型间的相互关联矩阵。 最后，我们的方法在S3DIS、SemanticKITTI和ScanNet数据集上进行了广泛的评估，在无监督方法中取得了最佳性能。具体而言，在S3DIS数据集的Area-5部分实现了47.1%的mIoU指标，这一成绩甚至超过了经典的完全有监督方法PointNet 2.5个百分点。
URL
https://arxiv.org/abs/2603.06321
PDF
-
Making Training-Free Diffusion Segmentors Scale with the Generative Power
2026-03-06 11:35:37Benyuan Meng, Qianqian Xu, Zitai Wang, Xiaochun Cao, Longtao Huang, Qingming HuangAbstract
As powerful generative models, text-to-image diffusion models have recently been explored for discriminative tasks. A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training, leading to what training-free diffusion segmentors. These methods typically rely on cross-attention maps from the model's attention layers, which are assumed to capture semantic relationships between image pixels and text tokens. Ideally, such approaches should benefit from more powerful diffusion models, i.e., stronger generative capability should lead to better segmentation. However, we observe that existing methods often fail to scale accordingly. To understand this issue, we identify two underlying gaps: (i) cross-attention is computed across multiple heads and layers, but there exists a discrepancy between these individual attention maps and a unified global representation. (ii) Even when a global map is available, it does not directly translate to accurate semantic correlation for segmentation, due to score imbalances among different text tokens. To bridge these gaps, we propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability. We evaluate our approach on standard semantic segmentation benchmarks and further integrate it into a generative technique, demonstrating both improved performance broad applicability. Codes are at this https URL.
Abstract (translated)
作为强大的生成模型，文本到图像的扩散模型最近被探索用于判别任务。一项研究重点是将预训练的扩散模型适应于语义分割任务而不进行进一步的训练，从而产生无需训练的扩散分割器（training-free diffusion segmentors）。这些方法通常依赖于来自模型注意层的交叉注意力图，假设它们捕获了图像像素和文本标记之间的语义关系。理想情况下，这样的方法应该受益于更强大的扩散模型，即更强的生成能力应导致更好的分割效果。然而，我们观察到现有方法往往无法相应地扩展。为了理解这一问题，我们确定了两个潜在差距：(i) 交叉注意力在多个头和层之间进行计算，但这些单独的注意图与统一的整体表示之间存在差异。(ii) 即使可以获取整体映射，在分割过程中它也无法直接转化为准确的语义相关性，由于不同文本标记之间的评分不平衡。为了弥合这两个差距，我们提出了两种技术：自动聚合（auto aggregation）和逐像素重缩放（per-pixel rescaling），这些技术结合使用可以使无训练分割更好地利用生成能力。我们在标准语义分割基准上评估了我们的方法，并将其进一步集成到一种生成技术中，展示了改进的性能和广泛的应用性。代码可在以下链接找到：[请在此处插入实际URL]。
URL
https://arxiv.org/abs/2603.06178
PDF
-
JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas
2026-03-06 11:22:14Sandeep Inuganti, Hideaki Kanayama, Kanta Shimizu, Mahdi Chamseddine, Soichiro Yokota, Didier Stricker, Jason RambachAbstract
Semantic segmentation across visual modalities such as 3D point clouds and panoramic images remains a challenging task, primarily due to the scarcity of annotated data and the limited adaptability of fixed-label models. In this paper, we present JOPP-3D, an open-vocabulary semantic segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding. We convert RGB-D panoramic images into their corresponding tangential perspective images and 3D point clouds, then use these modalities to extract and align foundational vision-language features. This allows natural language querying to generate semantic masks on both input modalities. Experimental evaluation on the Stanford-2D-3D-s and ToF-360 datasets demonstrates the capability of JOPP-3D to produce coherent and semantically meaningful segmentations across panoramic and 3D domains. Our proposed method achieves a significant improvement compared to the SOTA in open and closed vocabulary 2D and 3D semantic segmentation.
Abstract (translated)
在视觉模态（如三维点云和全景图像）上进行语义分割仍然是一个具有挑战性的任务，主要原因是标注数据的稀缺以及固定标签模型适应能力有限。本文介绍了 JOPP-3D，这是一种开放式词汇语义分割框架，通过共同利用全景图和点云数据来实现基于语言的场景理解。我们把 RGB-D 全景图像转换为其相应的切线视角图像和 3D 点云，然后使用这些模态提取并对齐基础视觉-语言特征。这使得自然语言查询能够在这两种输入模式上生成语义蒙版。在斯坦福2D-3D-s 和 ToF-360 数据集上的实验评估证明了 JOPP-3D 能够在全景和 3D 域中生成一致且具有语义意义的分割结果。我们提出的方法在开放性和封闭性词汇的 2D 和 3D 语义分割方面相比现有最佳方法取得了显著的进步。
URL
https://arxiv.org/abs/2603.06168
PDF
-
Generalizable Multiscale Segmentation of Heterogeneous Map Collections
2026-03-05 10:40:01Remi PetitpierreAbstract
Historical map collections are highly diverse in style, scale, and geographic focus, often consisting of many single-sheet documents. Yet most work in map recognition focuses on specialist models tailored to homogeneous map series. In contrast, this article aims to develop generalizable semantic segmentation models and ontology. First, we introduce Semap, a new open benchmark dataset comprising 1,439 manually annotated patches designed to reflect the variety of historical map documents. Second, we present a segmentation framework that combines procedural data synthesis with multiscale integration to improve robustness and transferability. This framework achieves state-of-the-art performance on both the HCMSSD and Semap datasets, showing that a diversity-driven approach to map recognition is not only viable but also beneficial. The results indicate that segmentation performance remains largely stable across map collections, scales, geographic regions, and publication contexts. By proposing benchmark datasets and methods for the generic segmentation of historical maps, this work opens the way to integrating the long tail of cartographic archives to historical geographic studies.
Abstract (translated)
历史地图收藏在风格、比例尺和地理焦点方面具有高度多样性，通常由许多单张文档组成。然而，大多数地图识别工作集中在针对同质化地图系列的专用模型上。与此相反，本文旨在开发通用化的语义分割模型和本体论。首先，我们介绍了Semap，这是一个新的开放基准数据集，包含1,439个手动注释的补丁，旨在反映历史地图文档的多样性。其次，我们提出了一种结合程序化数据合成与多尺度集成的分割框架，以提高鲁棒性和迁移性。该框架在HCMSSD和Semap数据集上均取得了最先进的性能，表明多样性的驱动方法对于地图识别不仅可行而且有益。结果显示，在不同的地图集合、比例尺、地理区域及出版背景下，分割表现基本保持稳定。通过提出针对历史地图通用分割的基准数据集和方法，这项工作为将长尾制图档案整合到历史地理研究中开辟了道路。
URL
https://arxiv.org/abs/2603.05037
PDF
-
Semantic Bridging Domains: Pseudo-Source as Test-Time Connector
2026-03-04 08:50:14Xizhong Yang, Huiming Wang, Ning Xu, Mofei SongAbstract
Distribution shifts between training and testing data are a critical bottleneck limiting the practical utility of models, especially in real-world test-time scenarios. To adapt models when the source domain is unknown and the target domain is unlabeled, previous works constructed pseudo-source domains via data generation and translation, then aligned the target domain with them. However, significant discrepancies exist between the pseudo-source and the original source domain, leading to potential divergence when correcting the target directly. From this perspective, we propose a Stepwise Semantic Alignment (SSA) method, viewing the pseudo-source as a semantic bridge connecting the source and target, rather than a direct substitute for the source. Specifically, we leverage easily accessible universal semantics to rectify the semantic features of the pseudo-source, and then align the target domain using the corrected pseudo-source semantics. Additionally, we introduce a Hierarchical Feature Aggregation (HFA) module and a Confidence-Aware Complementary Learning (CACL) strategy to enhance the semantic quality of the SSA process in the absence of source and ground truth of target domains. We evaluated our approach on tasks like semantic segmentation and image classification, achieving a 5.2% performance boost on GTA2Cityscapes over the state-of-the-art.
Abstract (translated)
训练数据和测试数据之间的分布变化是限制模型实际应用的关键瓶颈，尤其是在现实世界的测试场景中。为了在源域未知且目标域未标记的情况下适应模型，以前的研究通过生成和转换构建了伪源域，并将目标域与它们对齐。然而，伪源域与原始源域之间存在显著差异，这可能导致直接纠正目标时的潜在偏差。从这个角度来看，我们提出了逐步语义对齐（SSA）方法，将伪源视作连接源域和目标域的语义桥梁，而不是直接替代品。具体来说，我们利用易于获取的通用语义来修正伪源的语义特征，并使用修正后的伪源语义对齐目标域。此外，我们引入了一个层次特征聚合（HFA）模块和一种基于信心感知互补学习（CACL）策略，在没有来源数据和目标域真实标签的情况下提升SSA过程中的语义质量。我们在诸如语义分割和图像分类等任务上评估了我们的方法，在GTA2Cityscapes上的表现比现有最佳方法高出5.2%。
URL
https://arxiv.org/abs/2603.03844
PDF
-
TinyIceNet: Low-Power SAR Sea Ice Segmentation for On-Board FPGA Inference
2026-03-03 15:21:00Mhd Rashed Al Koutayni, Mohamed Selim, Gerd Reis, Alain Pagani, Didier StrickerAbstract
Accurate sea ice mapping is essential for safe maritime navigation in polar regions, where rapidly changing ice conditions require timely and reliable information. While Sentinel-1 Synthetic Aperture Radar (SAR) provides high-resolution, all-weather observations of sea ice, conventional ground-based processing is limited by downlink bandwidth, latency, and energy costs associated with transmitting large volumes of raw data. On-board processing, enabled by dedicated inference chips integrated directly within the satellite payload, offers a transformative alternative by generating actionable sea ice products in orbit. In this context, we present TinyIceNet, a compact semantic segmentation network co-designed for on-board Stage of Development (SOD) mapping from dual-polarized Sentinel-1 SAR imagery under strict hardware and power constraints. Trained on the AI4Arctic dataset, TinyIceNet combines SAR-aware architectural simplifications with low-precision quantization to balance accuracy and efficiency. The model is synthesized using High-Level Synthesis and deployed on a Xilinx Zynq UltraScale+ FPGA platform, demonstrating near-real-time inference with significantly reduced energy consumption. Experimental results show that TinyIceNet achieves 75.216% F1 score on SOD segmentation while reducing energy consumption by 2x compared to full-precision GPU baselines, underscoring the potential of chip-level hardware-algorithm co-design for future spaceborne and edge AI systems.
Abstract (translated)
准确的海冰测绘对于极地地区的海上安全导航至关重要，尤其是在快速变化的冰况下，需要及时和可靠的信息。虽然Sentinel-1合成孔径雷达（SAR）提供了高分辨率、全天候的海洋冰层观测数据，但传统的地面处理受限于下行链路带宽、延迟以及传输大量原始数据的能量成本问题。而通过在卫星载荷中直接集成专用推理芯片实现的空间处理提供了一种变革性的替代方案，在轨道上生成可操作的海冰产品。 在此背景下，我们介绍了TinyIceNet，这是一种专门为双极化Sentinel-1 SAR图像设计的小型语义分割网络，用于在严格的硬件和功率约束条件下进行发展阶段（SOD）映射。通过使用AI4Arctic数据集训练，TinyIceNet结合了雷达感知的架构简化与低精度量化技术来平衡准确性和效率。该模型采用高层次综合方法合成，并部署于Xilinx Zynq UltraScale+ FPGA平台上，在显著降低能耗的同时展示了接近实时推理的能力。实验结果表明，TinyIceNet在SOD分割中实现了75.216%的F1得分，与全精度GPU基准相比将能源消耗降低了两倍，突显了芯片级硬件-算法协同设计对于未来空间和边缘AI系统潜力的重要性。 该研究强调了专用推理芯片和硬件优化模型对提高太空任务效率的关键作用，并为未来的极地观测提供了重要的技术基础。
URL
https://arxiv.org/abs/2603.03075
PDF
-
Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers
2026-03-03 12:20:42Youngjun Jun, Seil Kang, Woojung Han, Seong Jae HwangAbstract
Video Diffusion Transformers (DiTs) have been synthesizing high-quality video with high fidelity from given text descriptions involving motion. However, understanding how Video DiTs convert motion words into video remains insufficient. Furthermore, while prior studies on interpretable saliency maps primarily target objects, motion-related behavior in Video DiTs remains largely unexplored. In this paper, we investigate concrete motion features that specify when and which object moves for a given motion concept. First, to spatially localize, we introduce GramCol, which adaptively produces per-frame saliency maps for any text concept, including both motion and non-motion. Second, we propose a motion-feature selection algorithm to obtain an Interpretable Motion-Attentive Map (IMAP) that localizes motion spatially and temporally. Our method discovers concept saliency maps without the need for any gradient calculation or parameter update. Experimentally, our method shows outstanding localization capability on the motion localization task and zero-shot video semantic segmentation, providing interpretable and clearer saliency maps for both motion and non-motion concepts.
Abstract (translated)
视频扩散变换器（DiTs）已经能够从给定的包含运动描述的文本中合成高质量且忠实度高的视频。然而，关于Video DiT如何将运动词汇转化为视频的理解还很不充分。此外，之前的研究主要集中在可解释性的注意力图谱上，这些研究大多针对的是物体本身，而与运动相关的特性在Video DiTs中的表现则很少被探讨。 本文旨在探究对于给定的运动概念而言，在特定时间和哪一对象进行移动的具体运动特征。首先，为了空间定位，我们引入了GramCol技术，它可以为包括运动和非运动在内的任何文本概念自适应地生成每帧注意力图谱。其次，我们提出了一种选择性运动特征算法，以获得可解释的运动注意图（IMAP），该方法可以对运动进行时空定位。 我们的方法能够在不需要计算梯度或参数更新的情况下发现概念性的注意力图谱。实验表明，我们的方法在运动定位任务和零样本视频语义分割上展示了卓越的空间定位能力，并提供了更清晰、更具可解释性的运动和非运动概念的注意力图谱。
URL
https://arxiv.org/abs/2603.02919
PDF
-
DREAM: Where Visual Understanding Meets Text-to-Image Generation
2026-03-03 06:54:19Chao Li, Tianhong Li, Sai Vidyaranya Nuthalapati, Hong-You Chen, Satya Narayan Shukla, Yonghuan Yang, Jun Xiao, Xiangjun Fan, Aashu Singh, Dina Katabi, Shlok Kumar MishraAbstract
Unifying visual representation learning and text-to-image (T2I) generation within a single model remains a central challenge in multimodal learning. We introduce DREAM, a unified framework that jointly optimizes discriminative and generative objectives, while learning strong visual representations. DREAM is built on two key techniques: During training, Masking Warmup, a progressive masking schedule, begins with minimal masking to establish the contrastive alignment necessary for representation learning, then gradually transitions to full masking for stable generative training. At inference, DREAM employs Semantically Aligned Decoding to align partially masked image candidates with the target text and select the best one for further decoding, improving text-image fidelity (+6.3%) without external rerankers. Trained solely on CC12M, DREAM achieves 72.7% ImageNet linear-probing accuracy (+1.1% over CLIP) and an FID of 4.25 (+6.2% over FLUID), with consistent gains in few-shot classification, semantic segmentation, and depth estimation. These results demonstrate that discriminative and generative objectives can be synergistic, allowing unified multimodal models that excel at both visual understanding and generation.
Abstract (translated)
统一视觉表示学习和文本到图像（T2I）生成模型的单一框架仍然是多模态学习中的一个核心挑战。我们引入了DREAM，这是一个统一的框架，它同时优化判别性和生成性目标，并在此过程中学习强大的视觉表示。DREAM基于两种关键技术构建：在训练阶段，遮罩预热（Masking Warmup），一种渐进式的遮罩调度策略，从最小化遮罩开始建立对比对齐以支持表示学习，然后逐渐过渡到完全遮罩以实现稳定的生成性训练；在推理阶段，DREAM采用语义对齐解码技术来将部分遮罩的图像候选与目标文本进行对齐，并选择最佳选项继续解码，从而提高图文一致性（+6.3%），而无需外部重排序器。仅使用CC12M数据集进行训练，DREAM达到了72.7%的ImageNet线性探测精度（比CLIP高1.1%）和FID为4.25（比FLUID高6.2%），并且在少量样本分类、语义分割和深度估计方面表现出一致的优势。这些结果表明，判别性和生成性的目标可以协同工作，使得统一的多模态模型能够在视觉理解和生成两方面都表现优异。
URL
https://arxiv.org/abs/2603.02667
PDF
-
CAWM-Mamba: A unified model for infrared-visible image fusion and compound adverse weather restoration
2026-03-03 03:27:05Huichun Liu, Xiaosong Li, Zhuangfan Huang, Tao Ye, Yang Liu, Haishu TanAbstract
Multimodal Image Fusion (MMIF) integrates complementary information from various modalities to produce clearer and more informative fused images. MMIF under adverse weather is particularly crucial in autonomous driving and UAV monitoring applications. However, existing adverse weather fusion methods generally only tackle single types of degradation such as haze, rain, or snow, and fail when multiple degradations coexist (e.g., haze+rain, rain+snow). To address this challenge, we propose Compound Adverse Weather Mamba (CAWM-Mamba), the first end-to-end framework that jointly performs image fusion and compound weather restoration with unified shared weights. Our network contains three key components: (1) a Weather-Aware Preprocess Module (WAPM) to enhance degraded visible features and extracts global weather embeddings; (2) a Cross-modal Feature Interaction Module (CFIM) to facilitate the alignment of heterogeneous modalities and exchange of complementary features across modalities; and (3) a Wavelet Space State Block (WSSB) that leverages wavelet-domain decomposition to decouple multi-frequency degradations. WSSB includes Freq-SSM, a module that models anisotropic high-frequency degradation without redundancy, and a unified degradation representation mechanism to further improve generalization across complex compound weather conditions. Extensive experiments on the AWMM-100K benchmark and three standard fusion datasets demonstrate that CAWM-Mamba consistently outperforms state-of-the-art methods in both compound and single-weather scenarios. In addition, our fusion results excel in downstream tasks covering semantic segmentation and object detection, confirming the practical value in real-world adverse weather perception. The source code will be available at this https URL.
Abstract (translated)
多模态图像融合（MMIF）整合了来自各种模式的互补信息，以生成更清晰、更具信息量的融合图像。在恶劣天气条件下进行的MMIF对于自动驾驶和无人机监控应用尤为关键。然而，现有的恶劣天气融合方法通常只处理单一类型的退化情况，如雾霾、雨或雪，并且当多种退化同时存在时（例如雾霾+雨，雨+雪）则无法应对。为了解决这一挑战，我们提出了复合恶劣天气蟒蛇（CAWM-Mamba），这是第一个端到端框架，它能够联合执行图像融合和复合天气恢复，采用统一的共享权重。我们的网络包含三个关键组件： 1. **气象感知预处理模块**（WAPM）：该模块旨在增强退化的可见特征，并提取全局气象嵌入。 2. **跨模态特性交互模块**（CFIM）：它促进了异构模式的对齐和跨模式之间的互补特性的交换。 3. **小波空间状态块**（WSSB）：该块利用小波域分解来分离多频退化。WSSB包括Freq-SSM，这是一个模拟各向异性高频退化的模块，并且没有冗余；以及一个统一的降解表示机制，以进一步提升在复杂复合天气条件下的泛化能力。 通过AWMM-100K基准测试和三个标准融合数据集进行的大量实验表明，在复合天气场景和单一天气场景中，CAWM-Mamba始终优于现有的最先进技术。此外，我们的融合结果在下游任务（如语义分割和目标检测）中的表现也非常出色，证实了它在实际恶劣天气感知应用中的实用价值。 源代码可在[此处](https://此链接将由您提供)获取。
URL
https://arxiv.org/abs/2603.02560
PDF
-
Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation
2026-03-03 03:18:12Chonghua Lv, Dong Zhao, Shuang Wang, Dou Quan, Ning Huyan, Nicu Sebe, Zhun ZhongAbstract
Knowledge distillation (KD) has been widely applied in semantic segmentation to compress large models, but conventional approaches primarily preserve in-domain accuracy while neglecting out-of-domain generalization, which is essential under distribution shifts. This limitation becomes more severe with the emergence of vision foundation models (VFMs): although VFMs exhibit strong robustness on unseen data, distilling them with conventional KD often compromises this ability. We propose Generalizable Knowledge Distillation (GKD), a multi-stage framework that explicitly enhances generalization. GKD decouples representation learning from task learning. In the first stage, the student acquires domain-agnostic representations through selective feature distillation, and in the second stage, these representations are frozen for task adaptation, thereby mitigating overfitting to visible domains. To further support transfer, we introduce a query-based soft distillation mechanism, where student features act as queries to teacher representations to selectively retrieve transferable spatial knowledge from VFMs. Extensive experiments on five domain generalization benchmarks demonstrate that GKD consistently outperforms existing KD methods, achieving average gains of +1.9% in foundation-to-foundation (F2F) and +10.6% in foundation-to-local (F2L) distillation. The code will be available at this https URL.
Abstract (translated)
知识蒸馏(KD)已被广泛应用于语义分割领域，用于压缩大型模型。然而，传统的做法主要保持了内部数据集的准确性，而忽略了外部泛化的关键性，在分布变化的情况下这一点尤为重要。这一局限在视觉基础模型(VFMs)出现后变得更为严重：尽管VFMs在未见数据上表现出强大的鲁棒性，但使用传统KD方法往往削弱了这种能力。我们提出了可泛化知识蒸馏(GKD)，这是一种多阶段框架，专门针对提高泛化的明确目标。GKD将表示学习与任务学习解耦。在第一阶段，学生模型通过选择性特征蒸馏获得不受领域影响的表示；在第二阶段，这些表示被冻结以进行任务适应，从而减轻对可见领域的过度拟合问题。为了进一步支持迁移，我们引入了基于查询的软蒸馏机制，在该机制中，学生特性作为查询来检索VFMs中的可迁移的空间知识。 在五个领域泛化基准上的大量实验表明，GKD始终优于现有的KD方法，在基础到基础(F2F)和基础到本地(F2L)蒸馏方面分别实现了平均1.9%和10.6%的改进。代码将在提供的链接中提供。
URL
https://arxiv.org/abs/2603.02554
PDF
-
SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data
2026-03-03 01:28:21Lekang Wen, Liang Liao, Jing Xiao, Mi WangAbstract
Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation. However, practical systems often encounter missing modalities due to sensor failures or incomplete coverage, termed Incomplete Multimodal Semantic Segmentation (IMSS). IMSS faces three key challenges: (1) multimodal imbalance, where dominant modalities suppress fragile ones; (2) intra-class variation in scale, shape, and orientation across modalities; and (3) cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses. Existing methods rely on contrastive learning or joint optimization, which risk over-alignment, discarding modality-specific cues or imbalanced training, favoring robust modalities, while largely overlooking intra-class variation and cross-modal heterogeneity. To address these limitations, we propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance. SGMA introduces two complementary plug-and-play modules: (1) Semantic-Guided Fusion (SGF) module extracts multi-scale, class-wise semantic prototypes that capture consistent categorical representations across modalities, estimates per-modality robustness based on prototype-feature alignment, and performs adaptive fusion weighted by robustness scores to mitigate intra-class variation and cross-modal heterogeneity; (2) Modality-Aware Sampling (MAS) module leverages robustness estimations from SGF to dynamically reweight training samples, prioritizing challenging samples from fragile modalities to address modality imbalance. Extensive experiments across multiple datasets and backbones demonstrate that SGMA consistently outperforms state-of-the-art methods, with particularly significant improvements in fragile modalities.
Abstract (translated)
多模态语义分割通过整合来自不同传感器的互补信息，用于遥感地球观测。然而，在实际系统中，由于传感器故障或覆盖不完整，经常会遇到缺失模式的情况，这种情况被称为不完整的多模态语义分割（IMSS）。IMSS面临三大挑战：(1) 多模态不平衡，其中主导模式会压制脆弱模式；(2) 在不同模式下的同一类别对象在大小、形状和方向上的变化；(3) 跨模态异质性导致冲突的线索产生不一致的语义响应。现有的方法主要依赖于对比学习或联合优化，这些方法有过度对齐的风险，可能会忽略特定模态的信息或者由于训练不平衡而偏向稳健模式，同时很大程度上忽视了类别内部变化和跨模态差异。为解决这些问题，我们提出了基于语义引导的模态感知（SGMA）框架，该框架确保多模态学习平衡的同时减少类内变异性并通过语义指导来协调跨模态不一致性。 SGMA引入了两个互补的即插即用模块： (1) 语义引导融合(SGF)模块：提取跨尺度、类别级别的语义原型以捕捉不同模式下的连贯分类表示，基于原型-特征对齐估计每个模式的稳健性，并根据稳健性分数执行自适应融合以缓解类内变异性及跨模态异质性； (2) 模态感知采样(MAS)模块：利用SGF中的稳健性评估动态重新加权训练样本，优先考虑来自脆弱模态的挑战性样本，解决模态不平衡问题。 通过多个数据集和骨干网络进行广泛的实验表明，SGMA在各种方法中始终表现出色，并且对于脆弱模式改进尤为显著。
URL
https://arxiv.org/abs/2603.02505
PDF
-
Advancing Earth Observation Through Machine Learning: A TorchGeo Tutorial
2026-03-02 20:56:08Caleb Robinson, Nils Lehmann, Adam J. Stewart, Burak Ekim, Heng Fang, Isaac A. Corley, Mauricio CordeiroAbstract
Earth observation machine learning pipelines differ fundamentally from standard computer vision workflows. Imagery is typically delivered as large, georeferenced scenes, labels may be raster masks or vector geometries in distinct coordinate reference systems, and both training and evaluation often require spatially aware sampling and splitting strategies. TorchGeo is a PyTorch-based domain library that provides datasets, samplers, transforms and pre-trained models with the goal of making it easy to use geospatial data in machine learning pipelines. In this paper, we introduce a tutorial that demonstrates 1.) the core TorchGeo abstractions through code examples, and 2.) an end-to-end case study on multispectral water segmentation from Sentinel-2 imagery using the Earth Surface Water dataset. This demonstrates how to train a semantic segmentation model using TorchGeo datasets, apply the model to a Sentinel-2 scene over Rio de Janeiro, Brazil, and save the resulting predictions as a GeoTIFF for further geospatial analysis. The tutorial code itself is distributed as two Python notebooks: this https URL and this https URL.
Abstract (translated)
地球观测机器学习管道与标准计算机视觉工作流程在根本上有所不同。图像通常以大型、地理参考的场景形式提供，标签可能是不同的坐标参考系统中的栅格掩码或向量几何图形，并且训练和评估过程常常需要空间感知采样和分割策略。TorchGeo 是一个基于 PyTorch 的领域库，它提供了数据集、采样器、转换以及预训练模型，旨在简化在机器学习管道中使用地理空间数据的过程。 在这篇论文中，我们介绍了一个教程，该教程通过代码示例展示了1.) TorchGeo的核心抽象，并2.) 使用Sentinel-2影像进行多光谱水域分割的端到端案例研究。本案例基于地球表面水体数据集（Earth Surface Water dataset），演示了如何使用TorchGeo的数据集训练语义分割模型，将该模型应用于巴西里约热内卢地区的Sentinel-2场景，并保存预测结果为GeoTIFF文件以供进一步的地理空间分析。 教程代码本身以两个Python笔记本的形式分发：[此链接](https://this.https.URL) 和 [此链接](https://this.https.URL)。
URL
https://arxiv.org/abs/2603.02386
PDF
-
Downstream Task Inspired Underwater Image Enhancement: A Perception-Aware Study from Dataset Construction to Network Design
2026-03-02 11:50:09Bosen Lin, Feng Gao, Yanwei Yu, Junyu Dong, Qian DuAbstract
In real underwater environments, downstream image recognition tasks such as semantic segmentation and object detection often face challenges posed by problems like blurring and color inconsistencies. Underwater image enhancement (UIE) has emerged as a promising preprocessing approach, aiming to improve the recognizability of targets in underwater images. However, most existing UIE methods mainly focus on enhancing images for human visual perception, frequently failing to reconstruct high-frequency details that are critical for task-specific recognition. To address this issue, we propose a Downstream Task-Inspired Underwater Image Enhancement (DTI-UIE) framework, which leverages human visual perception model to enhance images effectively for underwater vision tasks. Specifically, we design an efficient two-branch network with task-aware attention module for feature mixing. The network benefits from a multi-stage training framework and a task-driven perceptual loss. Additionally, inspired by human perception, we automatically construct a Task-Inspired UIE Dataset (TI-UIED) using various task-specific networks. Experimental results demonstrate that DTI-UIE significantly improves task performance by generating preprocessed images that are beneficial for downstream tasks such as semantic segmentation, object detection, and instance segmentation. The codes are publicly available at this https URL.
Abstract (translated)
在真实的水下环境中，下游图像识别任务（如语义分割和目标检测）常常会遇到诸如模糊和颜色不一致等问题所带来的挑战。水下图像增强（UIE）作为一种前景预处理方法应运而生，旨在提高水下图像中目标的可辨识性。然而，大多数现有的UIE方法主要集中在改善人类视觉感知效果上，通常无法恢复对特定任务识别至关重要的高频细节信息。为了解决这一问题，我们提出了一种受下游任务启发的水下图像增强（DTI-UIE）框架，该框架利用了人类视觉感知模型来有效提高水下视觉任务中的图像质量。具体而言，我们设计了一个高效的双分支网络，并配备了具备任务意识注意模块的功能混合结构。此外，该网络受益于一个多阶段训练框架以及一种由任务驱动的感知损失函数。另外，受人类感知启发，我们利用各种特定任务的网络自动构建了“任务启发式水下图像增强数据集”（TI-UIED）。实验结果表明，DTI-UIE通过生成有利于下游任务如语义分割、目标检测和实例分割等任务处理的预处理图像，显著提升了这些任务的表现。相关代码可以在以下链接获得：[此URL]。
URL
https://arxiv.org/abs/2603.01767
PDF
-
Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing
2026-03-02 07:05:37Zijin Yin, Bing Li, Kongming Liang, Hao Sun, Zhongjiang He, Zhanyu Ma, Jun GuoAbstract
Semantic segmentation takes pivotal roles in various applications such as autonomous driving and medical image analysis. When deploying segmentation models in practice, it is critical to test their behaviors in varied and complex scenes in advance. In this paper, we construct an automatic data generation pipeline Gen4Seg to stress-test semantic segmentation models by generating various challenging samples with different attribute changes. Beyond previous evaluation paradigms focusing solely on global weather and style transfer, we investigate variations in both appearance and geometry attributes at the object and image level. These include object color, material, size, position, as well as image-level variations such as weather and style. To achieve this, we propose to edit visual attributes of existing real images with precise control of structural information, empowered by diffusion models. In this way, the existing segmentation labels can be reused for the edited images, which greatly reduces the labor costs. Using our pipeline, we construct two new benchmarks, Pascal-EA and COCO-EA. We benchmark a wide variety of semantic segmentation models, spanning from closed-set models to open-vocabulary large models. We have several key findings: 1) advanced open-vocabulary models do not exhibit greater robustness compared to closed-set methods under geometric variations; 2) data augmentation techniques, such as CutOut and CutMix, are limited in enhancing robustness against appearance variations; 3) our pipeline can also be employed as a data augmentation tool and improve both in-distribution and out-of-distribution performances. Our work suggests the potential of generative models as effective tools for automatically analyzing segmentation models, and we hope our findings will assist practitioners and researchers in developing more robust and reliable segmentation models.
Abstract (translated)
语义分割在自动驾驶和医学图像分析等多种应用中扮演着关键角色。当将分割模型部署到实际场景时，提前测试它们在各种复杂环境中的表现是至关重要的。在这篇论文中，我们构建了一个自动数据生成管道Gen4Seg，通过生成具有不同属性变化的各种挑战性样本来压力测试语义分割模型。与以往只关注全局天气和风格转换的评估范式不同，我们研究了对象级和图像级别上外观和几何属性的变化情况。这些变化包括物体的颜色、材质、尺寸、位置以及图像级别的天气和风格等。为了实现这一目标，我们提出利用扩散模型对现有真实图像的视觉属性进行编辑，并且能够精确控制结构信息。这样可以重用原有的分割标签来编辑后的图片，大大减少了人工劳动成本。 使用我们的管道，我们构建了两个新的基准测试集：Pascal-EA和COCO-EA。我们将各种语义分割模型进行了广泛的评估，这些模型涵盖了从封闭集合模型到开放词汇大模型的不同类型。我们有以下几个主要发现： 1. 在几何变化的情况下，先进的开放词汇模型并不比封闭集合方法表现出更强的鲁棒性。 2. 数据增强技术（如CutOut和CutMix）在对抗外观变化时有限于提升模型的鲁棒性。 3. 我们的管道也可以用作数据增强工具，并且能够改善分布内和分布外的表现。 我们的工作展示了生成模型作为自动分析分割模型的有效工具的巨大潜力，希望我们的研究发现能帮助从业者和研究人员开发出更加稳健可靠的分割模型。
URL
https://arxiv.org/abs/2603.01535
PDF
-
Open-Vocabulary vs Supervised Learning Methods for Post-Disaster Visual Scene Understanding
2026-03-01 23:50:08Anna Michailidou, Georgios Angelidis, Vasileios Argyriou, Panagiotis Sarigiannidis, Georgios Th. PapadopoulosAbstract
Aerial imagery is critical for large-scale post-disaster damage assessment. Automated interpretation remains challenging due to clutter, visual variability, and strong cross-event domain shift, while supervised approaches still rely on costly, task-specific annotations with limited coverage across disaster types and regions. Recent open-vocabulary and foundation vision models offer an appealing alternative, by reducing dependence on fixed label sets and extensive task-specific annotations. Instead, they leverage large-scale pretraining and vision-language representations. These properties are particularly relevant for post-disaster domains, where visual concepts are ambiguous and data availability is constrained. In this work, we present a comparative evaluation of supervised learning and open-vocabulary vision models for post-disaster scene understanding, focusing on semantic segmentation and object detection across multiple datasets, including FloodNet+, RescueNet, DFire, and LADD. We examine performance trends, failure modes, and practical trade-offs between different learning paradigms, providing insight into their applicability for real-world disaster response. The most notable remark across all evaluated benchmarks is that supervised training remains the most reliable approach (i.e., when the label space is fixed and annotations are available), especially for small objects and fine boundary delineation in cluttered scenes.
Abstract (translated)
航空影像对于大规模灾后损害评估至关重要。由于杂乱的环境、视觉变化性和强烈的事件间域偏移，自动解读仍面临挑战，而监督方法仍然依赖于成本高昂的任务特定标注，并且这些标注在灾难类型和地区方面的覆盖率有限。最近开放词汇和基础视觉模型提供了一种有吸引力的选择，它们减少了对固定标签集和大量任务特定注释的依赖，而是利用大规模预训练和视觉语言表示。对于灾后领域来说，这种特性尤为重要，因为这里的视觉概念模糊且数据可用性受限。 在这项工作中，我们展示了监督学习与开放词汇视觉模型在灾后场景理解方面的比较评估，重点在于多个数据集上的语义分割和目标检测，包括FloodNet+、RescueNet、DFire和LADD。我们研究了性能趋势、失败模式以及不同学习范式之间的实际权衡，提供了对它们在现实世界灾害响应中的适用性的见解。 所有评估基准中最值得注意的一点是，在标签空间固定且有注释可用的情况下，监督训练仍然是最可靠的方法（特别是对于小对象和杂乱场景中精细边界划分）。
URL
https://arxiv.org/abs/2603.01324
PDF
-
CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling
2026-03-01 17:57:43Li Jin, Weikai Chen, Yujie Wang, Yingda Yin, Zeyu Hu, Runze Zhang, Keyang Luo, Shengju Qian, Xin Wang, Xueying QinAbstract
Open-world promptable 3D semantic segmentation remains brittle as semantics are inferred in the input sensor coordinates. Yet, humans, in contrast, interpret parts via functional roles in a canonical space -- wings extend laterally, handles protrude to the side, and legs support from below. Psychophysical evidence shows that we mentally rotate objects into canonical frames to reveal these roles. To fill this gap, we propose \methodName{}, which attains canonical space perception by inducing a latent canonical reference frame learned directly from data. By construction, we create a unified canonical dataset through LLM-guided intra- and cross-category alignment, exposing canonical spatial regularities across 200 categories. By induction, we realize canonicality inside the model through a dual-branch architecture with canonical map anchoring and canonical box calibration, collapsing pose variation and symmetry into a stable canonical embedding. This shift from input pose space to canonical embedding yields far more stable and transferable part semantics. Experimental results show that \methodName{} establishes new state of the art in open-world promptable 3D segmentation.
Abstract (translated)
URL
https://arxiv.org/abs/2603.01205
PDF
-
Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation
2026-03-01 12:33:14Wangkai Li, Zhaoyang Li, Yuwen Pan, Rui Sun, Yujia Chen, Tianzhu ZhangAbstract
Adverse weather conditions significantly degrade the performance of LiDAR point cloud semantic segmentation networks by introducing large distribution shifts. Existing augmentation-based methods attempt to enhance robustness by simulating weather interference during training. However, they struggle to fully exploit the potential of augmentations due to the trade-off between minor and aggressive augmentations. To address this, we propose A3Point, an adaptive augmentation-aware latent learning framework that effectively utilizes a diverse range of augmentations while mitigating the semantic shift, which refers to the change in the semantic meaning caused by augmentations. A3Point consists of two key components: semantic confusion prior (SCP) latent learning, which captures the model's inherent semantic confusion information, and semantic shift region (SSR) localization, which decouples semantic confusion and semantic shift, enabling adaptive optimization strategies for different disturbance levels. Extensive experiments on multiple standard generalized LiDAR segmentation benchmarks under adverse weather demonstrate the effectiveness of our method, setting new state-of-the-art results.
Abstract (translated)
恶劣的天气条件会显著降低激光雷达点云语义分割网络的表现，通过引入大的分布偏移。现有的基于数据增强的方法试图通过在训练过程中模拟天气干扰来提高模型的鲁棒性。然而，由于轻微和激进的数据增强之间的权衡，它们难以充分利用增强技术的潜力。为了解决这个问题，我们提出了A3Point，这是一种自适应增广感知隐式学习框架，能够有效利用多样化的增强方法的同时减少由这些增强引起的语义偏移（即数据增强造成的语义含义变化）。A3Point主要包含两个关键组成部分：语义混淆先验（SCP）隐式学习，用于捕捉模型内在的语义模糊信息；以及语义偏移区域（SSR）定位，它将语义混淆与语义偏移解耦，从而能够根据不同干扰程度实施自适应优化策略。在多个标准广义激光雷达分割基准测试中的恶劣天气条件下进行的广泛实验展示了我们方法的有效性，并且我们的方法为这一任务设定了新的最先进水平（SOTA）结果。
URL
https://arxiv.org/abs/2603.01074
PDF
-
MME: Mixture of Mesh Experts with Random Walk Transformer Gating
2026-02-28 22:13:00Amir Belder, Ayellet TalAbstract
In recent years, various methods have been proposed for mesh analysis, each offering distinct advantages and often excelling on different object classes. We present a novel Mixture of Experts (MoE) framework designed to harness the complementary strengths of these diverse approaches. We propose a new gate architecture that encourages each expert to specialise in the classes it excels in. Our design is guided by two key ideas: (1) random walks over the mesh surface effectively capture the regions that individual experts attend to, and (2) an attention mechanism that enables the gate to focus on the areas most informative for each expert's decision-making. To further enhance performance, we introduce a dynamic loss balancing scheme that adjusts a trade-off between diversity and similarity losses throughout the training, where diversity prompts expert specialization, and similarity enables knowledge sharing among the experts. Our framework achieves state-of-the-art results in mesh classification, retrieval, and semantic segmentation tasks. Our code is available at: this https URL.
Abstract (translated)
近年来，人们提出了多种网格分析方法，每种方法都具有独特的优势，并且在不同的物体类别上表现出色。我们提出了一种新颖的专家混合（MoE）框架，旨在利用这些不同方法的优势。我们设计了一个新的门控架构，鼓励每个专家专注于其擅长的对象类别。我们的设计基于两个关键理念：(1) 网格表面上的随机游走可以有效地捕捉各个专家关注的区域；(2) 一种注意力机制能够使门控聚焦于对每位专家决策最有信息量的区域。为了进一步提高性能，我们引入了一种动态损失平衡方案，在整个训练过程中调整多样性和相似性损失之间的权衡，其中多样性推动了专家的专业化，而相似性则促进了专家间的知识共享。我们的框架在网格分类、检索和语义分割任务中达到了最先进的成果。我们的代码可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2603.00828
PDF
