Semantic_Segmentation
-
Cross-Stage Attention Propagation for Efficient Semantic Segmentation
2026-04-07 04:55:56Beoungwoo KangAbstract
Recent lightweight semantic segmentation methods have made significant progress by combining compact backbones with efficient decoder heads. However, most multi-scale decoders compute attention independently at each feature scale, introducing substantial redundancy since the resulting attention distributions across scales are strongly correlated. We propose Cross-Stage Attention Propagation (CSAP), a decoder framework that computes attention at the deepest feature scale and propagates the resulting attention maps to shallower stages, bypassing query-key computation at those stages entirely. This design preserves multi-scale contextual reasoning while substantially reducing the decoder's computational cost. CSAP-Tiny achieves 42.9% mIoU on ADE20K with only 5.5 GFLOPs, 80.5% on Cityscapes with 21.5 GFLOPs, and 40.9% on COCO-Stuff 164K with 5.5 GFLOPs, surpassing SegNeXt-Tiny by +1.8% on ADE20K while requiring 16.8% fewer floating-point operations.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05431
PDF
-
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
2026-04-06 06:48:32Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen, Jinyin Chen, Isao EchizenAbstract
Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.
Abstract (translated)
基于大规模网络图文对训练的视觉语言模型（VLMs）在多种视觉任务上展现出令人瞩目的零样本迁移能力。然而，超越标准基准的全面独立评估对于理解其鲁棒性、局限性和实际应用价值至关重要。本研究提出了一种针对自然对抗场景下多样化下游任务的VLM系统评估框架，该框架在既往评估工作中被忽视。我们在一系列精心构建的对抗数据集（包括文字攻击、ImageNet-A及自然语言诱导的对抗样本）上，对CLIP、鲁棒CLIP、BLIP2和SigLIP2等主流VLMs进行了广泛评估。通过测量所选VLMs在零样本图像分类、语义分割及视觉问答任务中的自然对抗性能，分析发现：鲁棒CLIP模型可能放大自然对抗脆弱性，而CLIP模型对自然语言诱导的对抗样本性能显著下降。此外，我们提供了可解释性分析以识别失效模式。希望本研究能启发未来关于鲁棒公平的多模态模式识别研究。
URL
https://arxiv.org/abs/2604.04473
PDF
-
Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso
2026-04-05 04:07:59Fei Wang, Yutong Zhang, Xiong WangAbstract
Learning interpretable multimodal representations inherently relies on uncovering the conditional dependencies between heterogeneous features. However, sparse graph estimation techniques, such as Graphical Lasso (GLasso), to visual-linguistic domains is severely bottlenecked by high-dimensional noise, modality misalignment, and the confounding of shared versus category-specific topologies. In this paper, we propose Cross-Modal Graphical Lasso (CM-GLasso) that overcomes these fundamental limitations. By coupling a novel text-visualization strategy with a unified vision-language encoder, we strictly align multimodal features into a shared latent space. We introduce a cross-attention distillation mechanism that condenses high-dimensional patches into explicit semantic nodes, naturally extracting spatial-aware cross-modal priors. Furthermore, we unify tailored GLasso estimation and Common-Specific Structure Learning (CSSL) into a joint objective optimized via the Alternating Direction Method of Multiplier (ADMM). This formulation guarantees the simultaneous disentanglement of invariant and class-specific precision matrices without multi-step error accumulation. Extensive experiments across eight benchmarks covering both natural and medical domains demonstrate that CM-GLasso establishes a new state-of-the-art in generative classification and dense semantic segmentation tasks.
Abstract (translated)
学习可解释的多模态表示本质上依赖于挖掘异构特征间的条件依赖关系。然而，将稀疏图估计技术（如图形套索GLasso）应用于视觉-语言领域时，会受到高维噪声、模态错位以及共享拓扑与类别特定拓扑混淆的严重制约。本文提出跨模态图形套索（CM-GLasso）以克服这些根本性限制。通过将新颖的文本-可视化策略与统一视觉-语言编码器相结合，我们将多模态特征严格对齐至共享潜在空间。引入的跨注意力蒸馏机制能将高维图像块压缩为显式语义节点，自然提取空间感知的跨模态先验。此外，我们将定制化图形套索估计与共同-特定结构学习（CSSL）统一为联合目标函数，并通过乘子交替方向法（ADMM）进行优化。该 formulation 保证了不变与类别特定精度矩阵的同步解耦，且避免了多步骤误差累积。在涵盖自然与医学领域的八个基准数据集上的大量实验表明，CM-GLasso 在生成式分类与密集语义分割任务中建立了新的最先进水平。
URL
https://arxiv.org/abs/2604.03953
PDF
-
CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation
2026-04-03 10:32:02Zelin Zhang, Kedi Li, Huiqi Liang, Tao Zhang, Chuanzhi XuAbstract
Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that either use modality-specific adaptations or rely on loosely coupled interactions, thereby limiting flexibility and resulting in less effective cross-modal coordination. Moreover, these methods often struggle to balance efficient information exchange with preserving the unique characteristics of each modality across different modality combinations. To address these challenges, we propose CrossWeaver, a simple yet effective multimodal fusion framework for arbitrary-modality semantic segmentation. Its core is a Modality Interaction Block (MIB), which enables selective and reliability-aware cross-modal interaction within the encoder, while a lightweight Seam-Aligned Fusion (SAF) module further aggregates the enhanced features. Extensive experiments on multiple multimodal semantic segmentation benchmarks demonstrate that our framework achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.
Abstract (translated)
多模态语义分割在利用不同感知模态间的互补信息方面展现出巨大潜力。然而，现有方法通常依赖于精心设计的融合策略，这些策略要么采用模态特定适配，要么依赖松散耦合的交互，从而限制了灵活性并导致跨模态协调效果不佳。此外，这些方法往往难以在高效信息交换与保持不同模态组合中各模态独特特性之间取得平衡。为应对这些挑战，我们提出了CrossWeaver——一个简单而有效的任意模态语义分割多模态融合框架。其核心是模态交互块（MIB），它能够在编码器内部实现选择性且感知可靠性的跨模态交互，同时轻量级的接缝对齐融合（SAF）模块进一步聚合增强后的特征。在多个多模态语义分割基准数据集上的大量实验表明，我们的框架以极少的额外参数实现了最先进的性能，并能强泛化至未见过的模态组合。
URL
https://arxiv.org/abs/2604.02948
PDF
-
PaveBench: A Versatile Benchmark for Pavement Distress Perception and Interactive Vision-Language Analysis
2026-04-03 07:15:41Dexiang Li, Zhenning Che, Haijun Zhang, Dongliang Zhou, Zhao Zhang, Yahong HanAbstract
Pavement condition assessment is essential for road safety and maintenance. Existing research has made significant progress. However, most studies focus on conventional computer vision tasks such as classification, detection, and segmentation. In real-world applications, pavement inspection requires more than visual recognition. It also requires quantitative analysis, explanation, and interactive decision support. Current datasets are limited. They focus on unimodal perception. They lack support for multi-turn interaction and fact-grounded reasoning. They also do not connect perception with vision-language analysis. To address these limitations, we introduce PaveBench, a large-scale benchmark for pavement distress perception and interactive vision-language analysis on real-world highway inspection images. PaveBench supports four core tasks: classification, object detection, semantic segmentation, and vision-language question answering. It provides unified task definitions and evaluation protocols. On the visual side, PaveBench provides large-scale annotations and includes a curated hard-distractor subset for robustness evaluation. It contains a large collection of real-world pavement images. On the multimodal side, we introduce PaveVQA, a real-image question answering (QA) dataset that supports single-turn, multi-turn, and expert-corrected interactions. It covers recognition, localization, quantitative estimation, and maintenance reasoning. We evaluate several state-of-the-art methods and provide a detailed analysis. We also present a simple and effective agent-augmented visual question answering framework that integrates domain-specific models as tools alongside vision-language models. The dataset is available at: this https URL.
Abstract (translated)
路面状况评估对于道路安全与养护至关重要。现有研究已取得显著进展，但多数工作聚焦于分类、检测与分割等传统计算机视觉任务。在实际应用中，路面检测不仅需要视觉识别，还需定量分析、解释说明与交互式决策支持。当前数据集存在局限：侧重单模态感知，缺乏多轮交互与事实验证推理支持，且未实现感知与视觉-语言分析的结合。为应对这些挑战，我们提出PaveBench——首个面向真实高速公路巡检图像的大规模路面病害感知与交互式视觉-语言分析基准。该基准支持分类、目标检测、语义分割与视觉-语言问答四大核心任务，并提供统一的任务定义与评估协议。在视觉层面，PaveBench提供大规模标注数据，并包含精心筛选的困难干扰子集以评估模型鲁棒性；在 multimodal 层面，我们引入PaveVQA——首个支持单轮、多轮及专家修正交互的真实图像问答数据集，涵盖识别、定位、定量估算与养护推理等维度。我们评估了多项先进方法并展开详细分析，同时提出一种简单有效的智能体增强型视觉问答框架，可将领域专用模型作为工具与视觉-语言模型协同使用。数据集已公开：this https URL。
URL
https://arxiv.org/abs/2604.02804
PDF
-
Environment-Aware Channel Prediction for Vehicular Communications: A Multimodal Visual Feature Fusion Framework
2026-04-02 13:31:17Xuejian Zhang, Ruisi He, Minseok Kim, Inocent Calist, Mi Yang, Ziyi QiAbstract
The deep integration of communication with intelligence and sensing, as a defining vision of 6G, renders environment-aware channel prediction a key enabling technology. As a representative 6G application, vehicular communications require accurate and forward-looking channel prediction under stringent reliability, latency, and adaptability demands. Traditional empirical and deterministic models remain limited in balancing accuracy, generalization, and deployability, while the growing availability of onboard and roadside sensing devices offers a promising source of environmental priors. This paper proposes an environment-aware channel prediction framework based on multimodal visual feature fusion. Using GPS data and vehicle-side panoramic RGB images, together with semantic segmentation and depth estimation, the framework extracts semantic, depth, and position features through a three-branch architecture and performs adaptive multimodal fusion via a squeeze-excitation attention gating module. For 360-dimensional angular power spectrum (APS) prediction, a dedicated regression head and a composite multi-constraint loss are further designed. As a result, joint prediction of path loss (PL), delay spread (DS), azimuth spread of arrival (ASA), azimuth spread of departure (ASD), and APS is achieved. Experiments on a synchronized urban V2I measurement dataset yield the best root mean square error (RMSE) of 3.26 dB for PL, RMSEs of 37.66 ns, 5.05 degrees, and 5.08 degrees for DS, ASA, and ASD, respectively, and mean/median APS cosine similarities of 0.9342/0.9571, demonstrating strong accuracy, generalization, and practical potential for intelligent channel prediction in 6G vehicular communications.
Abstract (translated)
通信与智能感知的深度融合作为6G的标志性愿景，使得环境感知信道预测成为一项关键使能技术。作为6G的典型应用，车联网通信在严苛的可靠性、低时延和自适应需求下，要求实现精准且具有前瞻性的信道预测。传统经验模型与确定性模型在精度、泛化能力与部署可行性之间难以兼顾，而车载与路侧感知设备的日益普及则为环境先验信息提供了极具潜力的数据来源。本文提出一种基于多模态视觉特征融合的环境感知信道预测框架。该框架利用GPS数据与车辆全景RGB图像，结合语义分割与深度估计，通过三分支架构提取语义、深度与位置特征，并借助挤压激励注意力门控模块实现自适应多模态融合。针对360维角度功率谱（APS）预测，进一步设计了专用回归头与复合多约束损失函数，从而实现了路径损耗（PL）、时延扩展（DS）、到达方位扩展（ASA）、离开方位扩展（ASD）及APS的联合预测。在同步采集的城市车对基础设施（V2I）测量数据集上的实验表明，PL预测取得了3.26 dB的最佳均方根误差（RMSE），DS、ASA、ASD的RMSE分别达到37.66纳秒、5.05度与5.08度，APS预测的平均/中值余弦相似度达0.9342/0.9571，验证了该方法在6G车联网通信智能信道预测中具备高精度、强泛化能力及实际应用潜力。
URL
https://arxiv.org/abs/2604.02396
PDF
-
Decouple and Rectify: Semantics-Preserving Structural Enhancement for Open-Vocabulary Remote Sensing Segmentation
2026-04-02 13:15:05Jie Feng, Fengze Li, Junpeng Zhang, Siyu Chen, Yuping Liang, Junying Chen, Ronghua ShangAbstract
Open-vocabulary semantic segmentation in the remote sensing (RS) field requires both language-aligned recognition and fine-grained spatial delineation. Although CLIP offers robust semantic generalization, its global-aligned visual representations inherently struggle to capture structural details. Recent methods attempt to compensate for this by introducing RS-pretrained DINO features. However, these methods treat CLIP representations as a monolithic semantic space and cannot localize where structural enhancement is required, failing to effectively delineate boundaries while risking the disruption of CLIP's semantic integrity. To address this limitation, we propose DR-Seg, a novel decouple-and-rectify framework in this paper. Our method is motivated by the key observation that CLIP feature channels exhibit distinct functional heterogeneity rather than forming a uniform semantic space. Building on this insight, DR-Seg decouples CLIP features into semantics-dominated and structure-dominated subspaces, enabling targeted structural enhancement by DINO without distorting language-aligned semantics. Subsequently, a prior-driven graph rectification module injects high-fidelity structural priors under DINO guidance to form a refined branch, while an uncertainty-guided adaptive fusion module dynamically integrates this refined branch with the original CLIP branch for final prediction. Comprehensive experiments across eight benchmarks demonstrate that DR-Seg establishes a new state-of-the-art.
Abstract (translated)
URL
https://arxiv.org/abs/2604.02010
PDF
-
Semantic Segmentation of Textured Non-manifold 3D Meshes using Transformers
2026-04-02 09:50:10Mohammadreza Heidarianbaei, Max Mehltretter, Franz RottensteinerAbstract
Textured 3D meshes jointly represent geometry, topology, and appearance, yet their irregular structure poses significant challenges for deep-learning-based semantic segmentation. While a few recent methods operate directly on meshes without imposing geometric constraints, they typically overlook the rich textural information also provided by such meshes. We introduce a texture-aware transformer that learns directly from raw pixels associated with each mesh face, coupled with a new hierarchical learning scheme for multi-scale feature aggregation. A texture branch summarizes all face-level pixels into a learnable token, which is fused with geometrical descriptors and processed by a stack of Two-Stage Transformer Blocks (TSTB), which allow for both a local and a global information flow. We evaluate our model on the Semantic Urban Meshes (SUM) benchmark and a newly curated cultural-heritage dataset comprising textured roof tiles with triangle-level annotations for damage types. Our method achieves 81.9\% mF1 and 94.3\% OA on SUM and 49.7\% mF1 and 72.8\% OA on the new dataset, substantially outperforming existing approaches.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01836
PDF
-
Mitigating the ID-OOD Tradeoff in Open-Set Test-Time Adaptation
2026-04-02 04:00:57Wenjie Zhao, Jia Li, Xin Dong, Yapeng Tian, Yu Xiang, Yunhui GuoAbstract
Open-set test-time adaptation (OSTTA) addresses the challenge of adapting models to new environments where out-of-distribution (OOD) samples coexist with in-distribution (ID) samples affected by distribution shifts. In such settings, covariate shift-for example, changes in weather conditions such as snow-can alter ID samples, reducing model reliability. Consequently, models must not only correctly classify covariate-shifted ID (csID) samples but also effectively reject covariate-shifted OOD (csOOD) samples. Entropy minimization is a common strategy in test-time adaptation to maintain ID performance under distribution shifts, while entropy maximization is widely applied to enhance OOD detection. Several studies have sought to combine these objectives to tackle the challenges of OSTTA. However, the intrinsic conflict between entropy minimization and maximization inevitably leads to a trade-off between csID classification and csOOD detection. In this paper, we first analyze the limitations of entropy maximization in OSTTA and then introduce an angular loss to regulate feature norm magnitudes, along with a feature-norm loss to suppress csOOD logits, thereby improving OOD detection. These objectives form ROSETTA, a $\underline{r}$obust $\underline{o}$pen-$\underline{se}$t $\underline{t}$est-$\underline{t}$ime $\underline{a}$daptation. Our method achieves strong OOD detection while maintaining high ID classification performance on CIFAR-10-C, CIFAR-100-C, Tiny-ImageNet-C and ImageNet-C. Furthermore, experiments on the Cityscapes validate the method's effectiveness in real-world semantic segmentation, and results on the HAC dataset demonstrate its applicability across different open-set TTA setups.
Abstract (translated)
URL
https://arxiv.org/abs/2604.01589
PDF
-
Prototype-Based Low Altitude UAV Semantic Segmentation
2026-04-02 02:50:05Da Zhang, Gao Junyu, Zhao ZhiyuanAbstract
Semantic segmentation of low-altitude UAV imagery presents unique challenges due to extreme scale variations, complex object boundaries, and limited computational resources on edge devices. Existing transformer-based segmentation methods achieve remarkable performance but incur high computational overhead, while lightweight approaches struggle to capture fine-grained details in high-resolution aerial scenes. To address these limitations, we propose PBSeg, an efficient prototype-based segmentation framework tailored for UAV applications. PBSeg introduces a novel prototype-based cross-attention (PBCA) that exploits feature redundancy to reduce computational complexity while maintaining segmentation quality. The framework incorporates an efficient multi-scale feature extraction module that combines deformable convolutions (DConv) with context-aware modulation (CAM) to capture both local details and global semantics. Experiments on two challenging UAV datasets demonstrate the effectiveness of the proposed approach. PBSeg achieves 71.86\% mIoU on UAVid and 80.92\% mIoU on UDD6, establishing competitive performance while maintaining computational efficiency. Code is available at this https URL.
Abstract (translated)
低空无人机图像的语义分割面临独特挑战，包括极端的尺度变化、复杂的物体边界以及边缘设备上有限的计算资源。现有基于Transformer的分割方法虽性能卓越，但计算开销较高；而轻量级方法则难以捕捉高分辨率航拍场景中的细粒度细节。为应对这些限制，我们提出了PBSeg——一个专为无人机应用设计的高效原型分割框架。PBSeg引入了新颖的基于原型的交叉注意力机制（PBCA），通过利用特征冗余来降低计算复杂度，同时保持分割质量。该框架集成了高效的多尺度特征提取模块，结合可变形卷积（DConv）与上下文感知调制（CAM），以同时捕获局部细节与全局语义。在两个具有挑战性的无人机数据集上的实验验证了该方法的有效性：PBSeg在UAVid上达到71.86%的平均交并比（mIoU），在UDD6上达到80.92% mIoU，在保持计算效率的同时实现了具有竞争力的性能。代码已公开于此链接。
URL
https://arxiv.org/abs/2604.01550
PDF
-
IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentation
2026-04-01 20:19:52Nermin Samet, Gilles Puy, Renaud MarletAbstract
This paper presents a new method for the zero-shot open-vocabulary semantic segmentation (OVSS) of 3D automotive lidar data. To circumvent the recognized image-text modality gap that is intrinsic to approaches based on Vision Language Models (VLMs) such as CLIP, our method relies instead on image generation from text, to create prototype images. Given a 3D network distilled from a 2D Vision Foundation Model (VFM), we then label a point cloud by matching 3D point features with 2D image features of these prototypes. Our method is state-of-the-art for OVSS on nuScenes and SemanticKITTI. Code, pre-trained models, and generated images are available at this https URL.
Abstract (translated)
本文提出了一种针对3D车载激光雷达数据的零样本开放词汇语义分割（OVSS）新方法。为规避基于视觉语言模型（如CLIP）方法中固有的图像-文本模态差异问题，本方法转而依赖从文本生成图像来创建原型图像。给定一个从2D视觉基础模型（VFM）蒸馏得到的3D网络，我们通过将3D点特征与这些原型的2D图像特征进行匹配，从而为点云标注标签。该方法在nuScenes和SemanticKITTI数据集上的OVSS任务中达到最先进水平。代码、预训练模型及生成图像可通过此链接获取。
URL
https://arxiv.org/abs/2604.01361
PDF
-
Excite, Attend and Segment : Domain-Agnostic Fine-Grained Mask Discovery with Feature Calibration and Self-Supervised Upsampling
2026-03-31 22:03:15Deepank Singh, Anurag Nihal, Vedhus HoskereAbstract
Unsupervised segmentation approaches have increasingly leveraged foundation models (FM) to improve salient object discovery. However, these methods often falter in scenes with complex, multi-component morphologies, where fine-grained structural detail is indispensable. Many state-of-the-art unsupervised segmentation pipelines rely on mask discovery approaches that utilize coarse, patch-level representations. These coarse representations inherently suppress the fine-grained detail required to resolve such complex morphologies. To overcome this limitation, we propose Excite, Attend and Segment (EASe), an unsupervised domain-agnostic semantic segmentation framework for easy fine-grained mask discovery across challenging real-world scenes. EASe utilizes novel Semantic-Aware Upsampling with Channel Excitation (SAUCE) to excite low-resolution FM feature channels for selective calibration and attends across spatially-encoded image and FM features to recover full-resolution semantic representations. Finally, EASe segments the aggregated features into multi-granularity masks using a novel training-free Cue-Attentive Feature Aggregator (CAFE) which leverages SAUCE attention scores as a semantic grouping signal. EASe, together with SAUCE and CAFE, operate directly at pixel-level feature representations to enable accurate fine-grained dense semantic mask discovery. Our evaluation demonstrates superior performance of EASe over previous state-of-the-arts (SOTAs) across major standard benchmarks and diverse datasets with complex morphologies. Code is available at this https URL
Abstract (translated)
无监督分割方法日益借助基础模型（FM）来提升显著目标发现能力。然而，这些方法在结构复杂、多组分的形态场景中往往难以应对，而此类场景恰恰需要精细的结构细节。许多先进的无监督分割流程依赖基于粗略图块级表征的掩码发现方法，这类粗粒度表征本质上抑制了解决复杂形态所需的细粒度细节。为克服此限制，我们提出了Excite, Attend and Segment（EASe）——一个无监督、领域无关的语义分割框架，旨在实现挑战性真实场景中的便捷细粒度掩码发现。EASe采用新颖的语义感知通道激励上采样（SAUCE）技术，激发低分辨率FM特征通道以实现选择性校准，并跨空间编码的图像与FM特征进行注意力聚合，从而恢复全分辨率语义表征。最终，EASe通过新型免训练的线索感知特征聚合器（CAFE）将聚合特征分割为多粒度掩码，该模块利用SAUCE的注意力分数作为语义分组信号。EASe与SAUCE、CAFE共同直接在像素级特征表征上运作，实现精确的细粒度密集语义掩码发现。我们的评估表明，在主要标准基准及多样化的复杂形态数据集上，EASe性能优于此前先进方法（SOTAs）。代码已公开于此网址。
URL
https://arxiv.org/abs/2604.00276
PDF
-
Polyhedral Unmixing: Bridging Semantic Segmentation with Hyperspectral Unmixing via Polyhedral-Cone Partitioning
2026-03-31 08:46:09Antoine Bottenmuller (CMM, PSL, STIM), Etienne Decenci\`ere (CMM, PSL, STIM), Petr Dokl\'adal (CMM, PSL, STIM)Abstract
Semantic segmentation and hyperspectral unmixing are two central problems in spectral image analysis. The former assigns each pixel a discrete label corresponding to its material class, whereas the latter estimates pure material spectra, called endmembers, and, for each pixel, a vector representing material abundances in the observed scene. Despite their complementarity, these two problems are usually addressed independently. This paper aims to bridge these two lines of work by formally showing that, under the linear mixing model, pixel classification by dominant materials induces polyhedral-cone regions in the spectral space. We leverage this fundamental property to propose a direct segmentation-to-unmixing pipeline that performs blind hyperspectral unmixing from any semantic segmentation by constructing a polyhedral-cone partition of the space that best fits the labeled pixels. Signed distances from pixels to the estimated regions are then computed, linearly transformed via a change of basis in the distance space, and projected onto the probability simplex, yielding an initial abundance estimate. This estimate is used to extract endmembers and recover final abundances via matrix pseudo-inversion. Because the segmentation method can be freely chosen, the user gains explicit control over the unmixing process, while the rest of the pipeline remains essentially deterministic and lightweight. Beyond improving interpretability, experiments on three real datasets demonstrate the effectiveness of the proposed approach when associated with appropriate clustering algorithms, and show consistent improvements over recent deep and non-deep state-of-the-art methods. The code is available at: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.29438
PDF
-
ConInfer: Context-Aware Inference for Training-Free Open-Vocabulary Remote Sensing Segmentation
2026-03-31 05:12:02Wenyang Chen, Zhanxuan Hu, Yaping Zhang, Hailong Ning, Yonghang TaiAbstract
Training-free open-vocabulary remote sensing segmentation (OVRSS), empowered by vision-language models, has emerged as a promising paradigm for achieving category-agnostic semantic understanding in remote sensing imagery. Existing approaches mainly focus on enhancing feature representations or mitigating modality discrepancies to improve patch-level prediction accuracy. However, such independent prediction schemes are fundamentally misaligned with the intrinsic characteristics of remote sensing data. In real-world applications, remote sensing scenes are typically large-scale and exhibit strong spatial as well as semantic correlations, making isolated patch-wise predictions insufficient for accurate segmentation. To address this limitation, we propose ConInfer, a context-aware inference framework for OVRSS that performs joint prediction across multiple spatial units while explicitly modeling their inter-unit semantic dependencies. By incorporating global contextual cues, our method significantly enhances segmentation consistency, robustness, and generalization in complex remote sensing environments. Extensive experiments on multiple benchmark datasets demonstrate that our approach consistently surpasses state-of-the-art per-pixel VLM-based baselines such as SegEarth-OV, achieving average improvements of 2.80% and 6.13% on open-vocabulary semantic segmentation and object extraction tasks, respectively. The implementation code is available at: this https URL
Abstract (translated)
无需训练的开放词汇遥感分割（OVRSS）得益于视觉语言模型，已成为实现遥感影像类别无关语义理解的有前景范式。现有方法主要聚焦于增强特征表示或缓解模态差异以提升图像块级预测精度。然而此类独立预测方案与遥感数据的内在特性存在根本错配——真实场景中遥感影像通常规模庞大且呈现强烈的空间与语义关联性，孤立的图像块预测难以实现精确分割。为突破此局限，我们提出ConInfer，一种面向OVRSS的上下文感知推理框架，通过跨多空间单元联合预测并显式建模单元间语义依赖。借助全局上下文信息的融入，本方法在复杂遥感环境中显著提升了分割一致性、鲁棒性与泛化能力。在多个基准数据集上的大量实验表明，本方法持续超越SegEarth-OV等先进逐像素视觉语言模型基线，在开放词汇语义分割与目标提取任务中分别实现平均2.80%与6.13%的性能提升。实现代码已公开于：此超链接
URL
https://arxiv.org/abs/2603.29271
PDF
-
Detection of Adversarial Attacks in Robotic Perception
2026-03-30 15:41:49Ziad Sharawy, Mohammad Nakshbandiand, Sorin Mihai GrigorescuAbstract
Deep Neural Networks (DNNs) achieve strong performance in semantic segmentation for robotic perception but remain vulnerable to adversarial attacks, threatening safety-critical applications. While robustness has been studied for image classification, semantic segmentation in robotic contexts requires specialized architectures and detection strategies.
Abstract (translated)
深度神经网络（DNNs）在机器人感知的语义分割任务中表现出色，但仍容易受到对抗攻击的威胁，这对安全关键型应用构成风险。尽管图像分类领域的鲁棒性研究已较为深入，但机器人场景下的语义分割需要专用架构和针对性的检测策略。
URL
https://arxiv.org/abs/2603.28594
PDF
-
Unified Restoration-Perception Learning: Maritime Infrared-Visible Image Fusion and Segmentation
2026-03-30 13:26:05Weichao Cai, Weiliang Huang, Biao Xue, Chao Huang, Fei Yuan, Bob ZhangAbstract
Marine scene understanding and segmentation plays a vital role in maritime monitoring and navigation safety. However, prevalent factors like fog and strong reflections in maritime environments cause severe image degradation, significantly compromising the stability of semantic perception. Existing restoration and enhancement methods typically target specific degradations or focus solely on visual quality, lacking end-to-end collaborative mechanisms that simultaneously improve structural recovery and semantic effectiveness. Moreover, publicly available infrared-visible datasets are predominantly collected from urban scenes, failing to capture the authentic characteristics of coupled degradations in marine environments. To address these challenges, the Infrared-Visible Maritime Ship Dataset (IVMSD) is proposed to cover various maritime scenarios under diverse weather and illumination conditions. Building upon this dataset, a Multi-task Complementary Learning Framework (MCLF) is proposed to collaboratively perform image restoration, multimodal fusion, and semantic segmentation within a unified architecture. The framework includes a Frequency-Spatial Enhancement Complementary (FSEC) module for degradation suppression and structural enhancement, a Semantic-Visual Consistency Attention (SVCA) module for semantic-consistent guidance, and a cross-modality guided attention mechanism for selective fusion. Experimental results on IVMSD demonstrate that the proposed method achieves state-of-the-art segmentation performance, significantly enhancing robustness and perceptual quality under complex maritime conditions.
Abstract (translated)
海洋场景理解与分割在海上监测和航行安全中起着至关重要的作用。然而，海上环境中常见的雾和强反射等因素会导致图像严重退化，显著影响语义感知的稳定性。现有的复原与增强方法通常针对特定退化类型，或仅关注视觉质量，缺乏能同时提升结构恢复与语义效果端到端的协同机制。此外，公开的红外-可见光数据集多采集自城市场景，未能真实反映海洋环境中耦合退化的特性。为应对这些挑战，本文提出红外-可见光海洋船舶数据集（IVMSD），涵盖不同天气与光照条件下的多种海上场景。基于该数据集，进一步提出多任务互补学习框架（MCLF），在统一架构中协同完成图像复原、多模态融合与语义分割任务。该框架包含用于退化抑制与结构增强的频空增强互补模块（FSEC）、用于语义一致引导的语义视觉一致性注意力模块（SVCA），以及用于选择性融合的跨模态引导注意力机制。在IVMSD上的实验结果表明，所提方法实现了最先进的分割性能，显著提升了复杂海况下的鲁棒性与感知质量。
URL
https://arxiv.org/abs/2603.28414
PDF
-
RecycleLoRA: Rank-Revealing QR-Based Dual-LoRA Subspace Adaptation for Domain Generalized Semantic Segmentation
2026-03-30 08:05:39Chanseul Cho, Seokju Yun, Jeaseong Jeon, Seungjae Moon, Youngmin RoAbstract
Domain Generalized Semantic Segmentation (DGSS) aims to maintain robust performance across unseen target domains. Vision Foundation Models (VFMs) offer rich multi-domain knowledge that can enhance generalization. However, strategies for actively exploiting the rich subspace structures within VFMs remain under-explored, with many existing methods focusing primarily on preserving pre-trained knowledge. Furthermore, their LoRA components often suffer from limited representational diversity and inefficient parameter utilization. We propose RecycleLoRA, which addresses both challenges by employing Rank-Revealing QR Decomposition (RRQR) to systematically exploit VFM's subspace structures and enhance LoRA's representational richness. Our main adapter leverages minor subspace directions identified by RRQR to learn diverse and independent features, achieving competitive performance even when used alone. We further introduce a sub adapter that carefully refines major directions with minimal adjustments, providing complementary improvements to the main adapter's strong baseline performance. This design enables the dual adapters to learn distinct representations without requiring additional regularization losses. Our systematic exploitation of pre-trained subspace structures through RRQR-based initialization leads to superior domain generalization performance. RecycleLoRA achieves state-of-the-art performance on both synthetic-to-real generalization and real-to-real generalization tasks without complex architectures or additional inference latency.
Abstract (translated)
域泛化语义分割（DGSS）旨在使模型在未见过的目标域中保持稳健性能。视觉基础模型（VFMs）提供了丰富的多领域知识以增强泛化能力。然而，主动利用VFMs中丰富子空间结构的策略尚未得到充分探索，现有方法多侧重于保留预训练知识。此外，其LoRA组件常面临表征多样性有限与参数利用效率低的问题。我们提出RecycleLoRA，通过采用秩揭示QR分解（RRQR）系统性地挖掘VFM的子空间结构并增强LoRA的表征丰富度，从而同时解决上述挑战。主适配器利用RRQR识别的次要子空间方向学习多样且独立的特征，即使单独使用也能达到具有竞争力的性能。我们进一步引入子适配器，以最小调整精细优化主要方向，为主适配器的强基线性能提供互补提升。该设计使双适配器无需额外正则化损失即可学习差异化表征。通过基于RRQR的初始化系统利用预训练子空间结构，我们实现了更优的域泛化性能。RecycleLoRA在合成到真实泛化与真实到真实泛化任务上均达到最先进水平，且无需复杂架构或增加推理延迟。
URL
https://arxiv.org/abs/2603.28142
PDF
-
Transcription and Recognition of Italian Parliamentary Speeches Using Vision-Language Models
2026-03-30 07:06:49Luigi Curini, Alfio Ferrara, Giovanni Pagano, Sergio PicasciaAbstract
Parliamentary proceedings represent a rich yet challenging resource for computational analysis, particularly when preserved only as scanned historical documents. Existing efforts to transcribe Italian parliamentary speeches have relied on traditional Optical Character Recognition pipelines, resulting in transcription errors and limited semantic annotation. In this paper, we propose a pipeline based on Vision-Language Models for the automatic transcription, semantic segmentation, and entity linking of Italian parliamentary speeches. The pipeline employs a specialised OCR model to extract text while preserving reading order, followed by a large-scale Vision-Language Model that performs transcription refinement, element classification, and speaker identification by jointly reasoning over visual layout and textual content. Extracted speakers are then linked to the Chamber of Deputies knowledge base through SPARQL queries and a multi-strategy fuzzy matching procedure. Evaluation against an established benchmark demonstrates substantial improvements both in transcription quality and speaker tagging.
Abstract (translated)
URL
https://arxiv.org/abs/2603.28103
PDF
-
SegRGB-X: General RGB-X Semantic Segmentation Model
2026-03-30 04:32:11Jiong Liu, Yingjie Xu, Xingcheng Zhou, Rui Song, Walter Zimmer, Alois Knoll, Hu CaoAbstract
Semantic segmentation across arbitrary sensor modalities faces significant challenges due to diverse sensor characteristics, and the traditional configurations for this task result in redundant development efforts. We address these challenges by introducing a universal arbitrary-modal semantic segmentation framework that unifies segmentation across multiple modalities. Our approach features three key innovations: (1) the Modality-aware CLIP (MA-CLIP), which provides modality-specific scene understanding guidance through LoRA fine-tuning; (2) Modality-aligned Embeddings for capturing fine-grained features; and (3) the Domain-specific Refinement Module (DSRM) for dynamic feature adjustment. Evaluated on five diverse datasets with different complementary modalities (event, thermal, depth, polarization, and light field), our model surpasses specialized multi-modal methods and achieves state-of-the-art performance with a mIoU of 65.03%. The codes will be released upon acceptance.
Abstract (translated)
由于传感器特性多样，任意传感器模态的语义分割面临显著挑战，且该任务传统配置导致重复性开发工作。我们通过引入通用任意模态语义分割框架来解决这些挑战，该框架统一了多模态间的分割任务。我们的方法包含三项关键创新：(1) 模态感知CLIP（MA-CLIP），通过LoRA微调提供模态特定的场景理解指导；(2) 模态对齐嵌入，用于捕捉细粒度特征；(3) 特定领域优化模块（DSRM），用于动态特征调整。在五个具有不同互补模态（事件、热成像、深度、偏振和光场）的多样化数据集上评估，我们的模型超越了专用多模态方法，并以65.03%的mIoU达到最先进性能。代码将在论文接收后公开。
URL
https://arxiv.org/abs/2603.28023
PDF
-
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
2026-03-30 03:33:10Jiachen Li, Hongyun Wang, Jinyu Xu, Wenbo Jiang, Yanchun Ma, Yongjian Liu, Qing Xie, Bolong ZhengAbstract
Referring image segmentation aims to localize and segment a target object in an image based on a free-form referring expression. The core challenge lies in effectively bridging linguistic descriptions with object-level visual representations, especially when referring expressions involve detailed attributes and complex inter-object relationships. Existing methods either rely on cross-modal alignment or employ Semantic Segmentation Prompts, but they often lack explicit reasoning mechanisms for grounding language descriptions to target regions in the image. To address these limitations, we propose PPCR, a Progressive Prompt-guided Cross-modal Reasoning framework for referring image segmentation. PPCR explicitly structures the reasoning process as a Semantic Understanding-Spatial Grounding-Instance Segmentation pipeline. Specifically, PPCR first employs multimodal large language models (MLLMs) to generate Semantic Segmentation Prompt that capture key semantic cues of the target object. Based on this semantic context, Spatial Segmentation Prompt are further generated to reason about object location and spatial extent, enabling a progressive transition from semantic understanding to spatial grounding. The Semantic and Spatial Segmentation prompts are then jointly integrated into the segmentation module to guide accurate target localization and segmentation. Extensive experiments on standard referring image segmentation benchmarks demonstrate that PPCR consistently outperforms existing methods. The code will be publicly released to facilitate reproducibility.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27993
PDF
