Semantic_Segmentation
-
A Unified Spatial Alignment Framework for Highly Transferable Transformation-Based Attacks on Spatially Structured Tasks
2026-03-26 09:31:46Jiaming Liang, Chi-Man PunAbstract
Transformation-based adversarial attacks (TAAs) demonstrate strong transferability when deceiving classification models. However, existing TAAs often perform unsatisfactorily or even fail when applied to structured tasks such as semantic segmentation and object detection. Encouragingly, recent studies that categorize transformations into non-spatial and spatial transformations inspire us to address this challenge. We find that for non-structured tasks, labels are spatially non-structured, and thus TAAs are not required to adjust labels when applying spatial transformations. In contrast, for structured tasks, labels are spatially structured, and failing to transform labels synchronously with inputs can cause spatial misalignment and yield erroneous gradients. To address these issues, we propose a novel unified Spatial Alignment Framework (SAF) for highly transferable TAAs on spatially structured tasks, where the TAAs spatially transform labels synchronously with the input using the proposed Spatial Alignment (SA) algorithm. Extensive experiments demonstrate the crucial role of our SAF for TAAs on structured tasks. Specifically, in non-targeted attacks, our SAF degrades the average mIoU on Cityscapes from 24.50 to 11.34, and on Kvasir-SEG from 49.91 to 31.80, while reducing the average mAP of COCO from 17.89 to 5.25.
Abstract (translated)
基于变换的对抗攻击（TAAs）在欺骗分类模型时展现出强迁移性。然而，当应用于语义分割和目标检测等结构化任务时，现有TAAs往往表现不佳甚至失效。令人鼓舞的是，近期研究将变换分为非空间变换与空间变换的分类方式启发了我们应对这一挑战。我们发现：对于非结构化任务，其标签在空间上呈非结构化分布，因此TAAs在应用空间变换时无需调整标签；而对于结构化任务，标签具有空间结构性，若不同步变换标签与输入会导致空间错位并产生错误梯度。针对这些问题，我们提出了一种新颖的统一空间对齐框架（SAF），用于在空间结构化任务中实现高迁移性TAAs，其中TAAs通过所提出的空间对齐（SA）算法同步变换标签与输入。大量实验验证了SAF对结构化任务中TAAs的关键作用。具体而言，在非目标攻击中，SAF使Cityscapes数据集上的平均mIoU从24.50降至11.34，Kvasir-SEG数据集上从49.91降至31.80，同时将COCO数据集的平均mAP从17.89降至5.25。
URL
https://arxiv.org/abs/2603.25230
PDF
-
Denoise and Align: Towards Source-Free UDA for Robust Panoramic Semantic Segmentation
2026-03-26 07:54:43Yaowen Chang, Zhen Cao, Xu Zheng, Xiaoxin Mi, Zhen DongAbstract
Panoramic semantic segmentation is pivotal for comprehensive 360° scene understanding in critical applications like autonomous driving and virtual reality. However, progress in this domain is constrained by two key challenges: the severe geometric distortions inherent in panoramic projections and the prohibitive cost of dense annotation. While Unsupervised Domain Adaptation (UDA) from label-rich pinhole-camera datasets offers a viable alternative, many real-world tasks impose a stricter source-free (SFUDA) constraint where source data is inaccessible for privacy or proprietary reasons. This constraint significantly amplifies the core problems of domain shift, leading to unreliable pseudo-labels and dramatic performance degradation, particularly for minority classes. To overcome these limitations, we propose the DAPASS framework. DAPASS introduces two synergistic modules to robustly transfer knowledge without source data. First, our Panoramic Confidence-Guided Denoising (PCGD) module generates high-fidelity, class-balanced pseudo-labels by enforcing perturbation consistency and incorporating neighborhood-level confidence to filter noise. Second, a Contextual Resolution Adversarial Module (CRAM) explicitly addresses scale variance and distortion by adversarially aligning fine-grained details from high-resolution crops with global semantics from low-resolution contexts. DAPASS achieves state-of-the-art performances on outdoor (Cityscapes-to-DensePASS) and indoor (Stanford2D3D) benchmarks, yielding 55.04% (+2.05%) and 70.38% (+1.54%) mIoU, respectively.
Abstract (translated)
全景语义分割对于自动驾驶和虚拟现实等关键应用中的全方位360°场景理解至关重要。然而，该领域的发展受限于两个核心挑战：全景投影固有的严重几何畸变，以及密集标注的高昂成本。尽管从标注丰富的针孔相机数据集进行无监督域适应（UDA）提供了一种可行方案，但许多实际任务施加了更严格的源自由（SFUDA）约束——因隐私或专有原因无法访问源数据。这一约束显著加剧了域偏移的核心问题，导致伪标签不可靠且性能急剧下降，尤其对少数类影响显著。为克服这些限制，我们提出了DAPASS框架。该框架引入两个协同模块以在无源数据条件下稳健迁移知识：首先，全景置信度引导去噪（PCGD）模块通过强制扰动一致性并引入邻域级置信度过滤噪声，生成高保真、类别平衡的伪标签；其次，上下文分辨率对抗模块（CRAM）通过对抗性对齐高分辨率裁剪的细粒度细节与低分辨率上下文的全局语义，显式解决尺度变异和畸变问题。DAPASS在户外（Cityscapes-to-DensePASS）和室内（Stanford2D3D）基准测试中均达到最先进性能，分别取得55.04%（+2.05%）和70.38%（+1.54%）的mIoU。
URL
https://arxiv.org/abs/2603.25131
PDF
-
Heuristic Self-Paced Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions
2026-03-25 14:01:55Shiqin Wang, Haoyang Chen, Huaizhou Huang, Yinkan He, Dongfang Sun, Xiaoqing Chen, Xingyu Liu, Zheng Wang, Kaiyan ZhaoAbstract
The learning order of semantic classes significantly impacts unsupervised domain adaptation for semantic segmentation, especially under adverse weather conditions. Most existing curricula rely on handcrafted heuristics (e.g., fixed uncertainty metrics) and follow a static schedule, which fails to adapt to a model's evolving, high-dimensional training dynamics, leading to category bias. Inspired by Reinforcement Learning, we cast curriculum learning as a sequential decision problem and propose an autonomous class scheduler. This scheduler consists of two components: (i) a high-dimensional state encoder that maps the model's training status into a latent space and distills key features indicative of progress, and (ii) a category-fair policy-gradient objective that ensures balanced improvement across classes. Coupled with mixed source-target supervision, the learned class rankings direct the network's focus to the most informative classes at each stage, enabling more adaptive and dynamic learning. It is worth noting that our method achieves state-of-the-art performance on three widely used benchmarks (e.g., ACDC, Dark Zurich, and Nighttime Driving) and shows generalization ability in synthetic-to-real semantic segmentation.
Abstract (translated)
语义类别的学习顺序显著影响无监督领域适应在语义分割中的表现，尤其在恶劣天气条件下。大多数现有课程依赖手工设计的启发式方法（如固定的不确定性指标）并遵循静态计划，这无法适应模型不断变化的高维训练动态，导致类别偏差。受强化学习启发，我们将课程学习构建为序列决策问题，并提出一种自主类别调度器。该调度器包含两个组件：(i) 高维状态编码器，将模型训练状态映射到潜在空间并提炼指示进展的关键特征；(ii) 类别公平的策略梯度目标，确保各类别间的平衡提升。结合混合源-目标监督，学习到的类别排序在每个阶段引导网络关注最具信息量的类别，实现更自适应和动态的学习。值得注意的是，我们的方法在三个广泛使用的基准数据集（如ACDC、Dark Zurich和Nighttime Driving）上达到最先进性能，并在合成到真实的语义分割中展现出泛化能力。
URL
https://arxiv.org/abs/2603.24322
PDF
-
RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation
2026-03-25 13:33:18Kai Zhu, Zhenyu Cui, Zehua Zang, Jiahuan ZhouAbstract
Recently, state space models have demonstrated efficient video segmentation through linear-complexity state space compression. However, Video Semantic Segmentation (VSS) requires pixel-level spatiotemporal modeling capabilities to maintain temporal consistency in segmentation of semantic objects. While state space models can preserve common semantic information during state space compression, the fixed-size state space inevitably forgets specific information, which limits the models' capability for pixel-level segmentation. To tackle the above issue, we proposed a Refining Specifics State Space Model approach (RS-SSM) for video semantic segmentation, which performs complementary refining of forgotten spatiotemporal specifics. Specifically, a Channel-wise Amplitude Perceptron (CwAP) is designed to extract and align the distribution characteristics of specific information in the state space. Besides, a Forgetting Gate Information Refiner (FGIR) is proposed to adaptively invert and refine the forgetting gate matrix in the state space model based on the specific information distribution. Consequently, our RS-SSM leverages the inverted forgetting gate to complementarily refine the specific information forgotten during state space compression, thereby enhancing the model's capability for spatiotemporal pixel-level segmentation. Extensive experiments on four VSS benchmarks demonstrate that our RS-SSM achieves state-of-the-art performance while maintaining high computational efficiency. The code is available at this https URL.
Abstract (translated)
最近，状态空间模型通过线性复杂度的状态空间压缩，展现了高效的视频分割能力。然而，视频语义分割（VSS）需要像素级的时空建模能力，以保持语义对象分割的时间一致性。虽然状态空间模型能在压缩过程中保留通用语义信息，但固定大小的状态空间不可避免地会遗忘具体信息，这限制了模型在像素级分割上的能力。针对上述问题，我们提出了一种用于视频语义分割的细化细节状态空间模型方法（RS-SSM），该方法对遗忘的时空细节进行互补细化。具体而言，设计了通道幅度感知机（CwAP）来提取和对齐状态空间中具体信息的分布特征。此外，提出了遗忘门信息细化器（FGIR），基于具体信息分布自适应反转并细化状态空间模型中的遗忘门矩阵。因此，我们的RS-SSM利用反转后的遗忘门，对状态空间压缩过程中被遗忘的具体信息进行互补细化，从而增强模型的时空像素级分割能力。在四个VSS基准测试上的大量实验表明，我们的RS-SSM在保持高计算效率的同时实现了最先进的性能。代码可在以下链接获取：https URL。
URL
https://arxiv.org/abs/2603.24295
PDF
-
InstanceRSR: Real-World Super-Resolution via Instance-Aware Representation Alignment
2026-03-25 12:27:58Zixin Guo, Kai Zhao, Luyan ZhangAbstract
Existing real-world super-resolution (RSR) methods based on generative priors have achieved remarkable progress in producing high-quality and globally consistent reconstructions. However, they often struggle to recover fine-grained details of diverse object instances in complex real-world scenes. This limitation primarily arises because commonly adopted denoising losses (e.g., MSE) inherently favor global consistency while neglecting instance-level perception and restoration. To address this issue, we propose InstanceRSR, a novel RSR framework that jointly models semantic information and introduces instance-level feature alignment. Specifically, we employ low-resolution (LR) images as global consistency guidance while jointly modeling image data and semantic segmentation maps to enforce semantic relevance during sampling. Moreover, we design an instance representation learning module to align the diffusion latent space with the instance latent space, enabling instance-aware feature alignment, and further incorporate a scale alignment mechanism to enhance fine-grained perception and detail recovery. Benefiting from these designs, our approach not only generates photorealistic details but also preserves semantic consistency at the instance level. Extensive experiments on multiple real-world benchmarks demonstrate that InstanceRSR significantly outperforms existing methods in both quantitative metrics and visual quality, achieving new state-of-the-art (SOTA) performance.
Abstract (translated)
现有基于生成先验的真实世界超分辨率（RSR）方法在生成高质量、全局一致的图像重建方面取得了显著进展。然而，它们在复杂真实场景中恢复多样目标实例的细粒度细节方面往往存在不足。这一局限性主要源于常用的去噪损失函数（如MSE）本质上倾向于全局一致性，而忽略了实例级的感知与修复。针对该问题，我们提出了InstanceRSR，一种新型RSR框架，通过联合建模语义信息并引入实例级特征对齐来解决问题。具体而言，我们采用低分辨率（LR）图像作为全局一致性指导，同时联合建模图像数据与语义分割图，以在采样过程中强化语义相关性。此外，我们设计了实例表征学习模块，将扩散隐空间与实例隐空间进行对齐，实现实例感知的特征对齐，并进一步引入尺度对齐机制以增强细粒度感知与细节恢复能力。得益于上述设计，我们的方法不仅能生成照片般真实的细节，还能在实例层面保持语义一致性。在多个真实场景基准数据集上的大量实验表明，InstanceRSR在定量指标与视觉质量上均显著优于现有方法，达到了新的最先进（SOTA）性能。
URL
https://arxiv.org/abs/2603.24240
PDF
-
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
2026-03-24 10:10:12ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun, Gilhan Park, WonJun Moon, Jae-Pil HeoAbstract
A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at this https URL.
Abstract (translated)
近期无需训练的开放词汇语义分割方法普遍采用滑动窗口推理策略，以克服CLIP模型处理高分辨率图像的局限。然而该方法引入新挑战：各窗口独立处理导致窗口间出现语义不一致。针对此问题，我们提出全局-局部对齐CLIP（GLA-CLIP）框架，促进窗口间全面信息交互。GLA-CLIP将键值令牌扩展至整合所有窗口的上下文线索，而非局限于单个窗口内的令牌。进一步观察发现窗口偏差现象：由于查询特征通过内窗口块交互生成，缺乏本地上下文之外的语义依据，导致外窗口令牌更少被关注。为缓解此问题，我们引入代理锚点——通过聚合所有窗口中与给定查询高度相似的令牌构建，为测量内/外窗口块相似性提供统一语义参考。此外提出动态归一化方案：通过动态缩放和阈值化注意力图，依据物体尺度调整注意力强度，以应对小目标场景。GLA-CLIP还可赋能现有方法并扩大其感受野。大量实验验证GLA-CLIP能有效提升无需训练的开放词汇语义分割性能。代码见此https链接。
URL
https://arxiv.org/abs/2603.23030
PDF
-
Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion
2026-03-23 22:52:00Abu Noman Md Sakib, OFM Riaz Rahman Aranya, Kevin Desai, Zijie ZhangAbstract
Attribution maps for semantic segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays within the target region. These questions require a dedicated evaluation protocol. We introduce a reproducible benchmark that tests intervention-based faithfulness, off-target leakage, perturbation robustness, and runtime on Pascal VOC and SBD across three pretrained backbones. To further demonstrate the benchmark, we propose Dual-Evidence Attribution (DEA), a lightweight correction that fuses gradient evidence with region-level intervention signals through agreement-weighted fusion. DEA increases emphasis where both sources agree and retains causal support when gradient responses are unstable. Across all completed runs, DEA consistently improves deletion-based faithfulness over gradient-only baselines and preserves strong robustness, at the cost of additional compute from intervention passes. The benchmark exposes a faithfulness-stability tradeoff among attribution families that is entirely hidden under visual evaluation, providing a foundation for principled method selection in segmentation explainability. Code is available at this https URL.
Abstract (translated)
语义分割的归因图几乎总是通过视觉合理性来评判。然而，看起来令人信服并不能保证被高亮的像素真正驱动模型的预测，也不能保证归因的贡献度局限于目标区域。这些问题需要专门的评估协议。我们引入了一个可复现的基准，该基准在Pascal VOC和SBD数据集上，跨三种预训练主干网络，测试了基于干预的忠实性、目标外信息泄露、扰动鲁棒性和运行时间。为了进一步展示该基准，我们提出了双证据归因（DEA），这是一种轻量级修正方法，通过基于一致性的加权融合，将梯度证据与区域级干预信号相结合。DEA在两者一致的区域增强强调，并在梯度响应不稳定时保留因果支持。在所有完成的测试中，DEA相较于仅使用梯度的基线，持续提升了基于删除的忠实性，并保持了强鲁棒性，代价是干预过程带来的额外计算开销。该基准揭示了归因方法家族内部的忠实性-稳定性权衡，而这种权衡在视觉评估下完全被隐藏，为分割可解释性中的原则性方法选择提供了基础。代码可在该https URL获取。
URL
https://arxiv.org/abs/2603.22624
PDF
-
CanViT: Toward Active-Vision Foundation Models
2026-03-23 21:05:21Yoha\"i-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh KrishnaAbstract
Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines. As a result, Active-Vision Foundation Models (AVFMs) have remained unexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve low-latency sequential inference and scalability to large scenes. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes -- an order of magnitude more than previous active models -- and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 19.5x fewer inference FLOPs and no fine-tuning, as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B reaches 81.2% top-1 accuracy with frozen teacher probes. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work closes the wide gap between passive and active vision on semantic segmentation and demonstrates the potential of AVFMs as a new research axis.
Abstract (translated)
主动视觉通过顺序、局部的注视点，有望实现高效且符合生物机理的感知，但一直缺乏可扩展的通用架构与预训练流程，导致主动视觉基础模型（AVFM）尚未被探索。本文提出首个任务与策略无关的AVFM——CanViT。其采用场景相对位置编码（scene-relative RoPE），将视网膜拓扑视觉Transformer主干与空间拓扑的全场景潜在工作空间（画布）相绑定。通过新型非对称交叉注意力机制——画布注意力（Canvas Attention），实现了与该高容量工作记忆的高效交互。通过解耦思考（主干层级）与记忆（画布层级），并移除画布端的自注意力与全连接层，实现了低延迟的顺序推理与对大规模场景的扩展性。我们提出了一种无标签的主动视觉预训练方案：策略无关的从被动到主动的稠密潜在蒸馏——即从随机位置、缩放级别与长度的低分辨率注视点序列中，重建全场景的DINOv3嵌入特征。我们在1320万张ImageNet-21k图像（比以往主动模型多一个数量级）及10亿个随机注视点上，使用单张H100显卡在166小时内完成了CanViT-B的随机初始化预训练。在ADE20K分割任务中，冻结的CanViT-B仅凭单次低分辨率注视即达到38.5% mIoU，不仅以19.5倍更少的推理浮点运算次数（无需微调）超越了最佳主动模型的27.6%，还超越了其浮点运算或输入匹配的DINOv3教师模型。增加注视次数后，CanViT-B可达45.9%的ADE20K mIoU。在ImageNet-1k分类任务中，CanViT-B使用冻结的教师探针达到81.2%的top-1准确率。CanViT可泛化至更长推理轨迹、更大场景与新策略。本研究弥合了被动与主动视觉在语义分割上的显著差距，并展示了AVFM作为新研究方向的潜力。
URL
https://arxiv.org/abs/2603.22570
PDF
-
UrbanVGGT: Scalable Sidewalk Width Estimation from Street View Images
2026-03-23 19:52:18Kaizhen Tan, Fan ZhangAbstract
Sidewalk width is an important indicator of pedestrian accessibility, comfort, and network quality, yet large-scale width data remain scarce in most cities. Existing approaches typically rely on costly field surveys, high-resolution overhead imagery, or simplified geometric assumptions that limit scalability or introduce systematic error. To address this gap, we present UrbanVGGT, a measurement pipeline for estimating metric sidewalk width from a single street-view image. The method combines semantic segmentation, feed-forward 3D reconstruction, adaptive ground-plane fitting, camera-height-based scale calibration, and directional width measurement on the recovered plane. On a ground-truth benchmark from Washington, D.C., UrbanVGGT achieves a mean absolute error of 0.252 m, with 95.5% of estimates within 0.50 m of the reference width. Ablation experiments show that metric scale calibration is the most critical component, and controlled comparisons with alternative geometry backbones support the effectiveness of the overall design. As a feasibility demonstration, we further apply the pipeline to three cities and generate SV-SideWidth, a prototype sidewalk-width dataset covering 527 OpenStreetMap street segments. The results indicate that street-view imagery can support scalable generation of candidate sidewalk-width attributes, while broader cross-city validation and local ground-truth auditing remain necessary before deployment as authoritative planning data.
Abstract (translated)
人行道宽度是衡量行人可达性、舒适度及路网质量的重要指标，但大多数城市仍缺乏大规模宽度数据。现有方法通常依赖成本高昂的实地勘测、高分辨率航拍影像，或基于简化几何假设的估算，这些方法要么难以规模化，要么会引入系统性误差。为填补这一空白，本研究提出 UrbanVGGT——一个从单张街景图像估算指标人行道宽度的测量流程。该方法融合了语义分割、前馈式三维重建、自适应地平面拟合、基于相机高度的尺度标定，以及在恢复平面上的方向性宽度测量。在华盛顿特区的真实值基准测试中，UrbanVGGT 的平均绝对误差为 0.252 米，95.5% 的估算值与参考宽度的偏差在 0.50 米以内。消融实验表明，指标尺度标定是最关键组件；与替代几何主干网络的受控对比也验证了整体设计的有效性。作为可行性演示，我们进一步将该流程应用于三座城市，生成了覆盖 527 个 OpenStreetMap 街道路段的原型数据集 SV-SideWidth。结果表明，街景影像能够支持可扩展的人行道宽度候选属性生成，但在将其作为权威规划数据部署前，仍需进行更广泛的多城市验证和本地真实值核查。
URL
https://arxiv.org/abs/2603.22531
PDF
-
Spatially-Aware Evaluation Framework for Aerial LiDAR Point Cloud Semantic Segmentation: Distance-Based Metrics on Challenging Regions
2026-03-23 18:00:37Alex Salvatierra, Jos\'e Antonio Sanz, Christian Guti\'errez, Mikel GalarAbstract
Semantic segmentation metrics for 3D point clouds, such as mean Intersection over Union (mIoU) and Overall Accuracy (OA), present two key limitations in the context of aerial LiDAR data. First, they treat all misclassifications equally regardless of their spatial context, overlooking cases where the geometric severity of errors directly impacts the quality of derived geospatial products such as Digital Terrain Models. Second, they are often dominated by the large proportion of easily classified points, which can mask meaningful differences between models and under-represent performance in challenging regions. To address these limitations, we propose a novel evaluation framework for comparing semantic segmentation models through two complementary approaches. First, we introduce distance-based metrics that account for the spatial deviation between each misclassified point and the nearest ground-truth point of the predicted class, capturing the geometric severity of errors. Second, we propose a focused evaluation on a common subset of hard points, defined as the points misclassified by at least one of the evaluated models, thereby reducing the bias introduced by easily classified points and better revealing differences in model performance in challenging regions. We validate our framework by comparing three state-of-the-art deep learning models on three aerial LiDAR datasets. Results demonstrate that the proposed metrics provide complementary information to traditional measures, revealing spatial error patterns that are critical for Earth Observation applications but invisible to conventional evaluation approaches. The proposed framework enables more informed model selection for scenarios where spatial consistency is critical.
Abstract (translated)
针对航空LiDAR数据的3D点云语义分割评估指标（如平均交并比mIoU和总体准确率OA）存在两个关键局限性。首先，这些指标对所有误分类情况一视同仁， irrespective of their spatial context，忽略了误差的几何严重性直接影响数字地形模型等衍生地理空间产品质量的情况。其次，由于大量易分类点占据主导，可能掩盖模型间的实质性差异，并低估困难区域的性能表现。 为应对这些局限，我们提出一种新型评估框架，通过两种互补方法比较语义分割模型：其一，引入基于距离的指标，通过计算每个误分类点与预测类别最近真实点之间的空间偏差，量化误差的几何严重性；其二，聚焦于困难点的共性子集（定义为至少被一个评估模型误分类的点），从而减少易分类点带来的偏差，更准确揭示模型在困难区域的性能差异。 我们在三个航空LiDAR数据集上对比三种先进深度学习模型验证该框架。结果表明，所提指标能为传统度量提供补充信息，揭示对地球观测应用至关重要但传统评估方法无法捕捉的空间误差模式。该框架可在空间一致性要求严苛的场景中支持更明智的模型选择。
URL
https://arxiv.org/abs/2603.22420
PDF
-
Riverine Land Cover Mapping through Semantic Segmentation of Multispectral Point Clouds
2026-03-23 17:27:17Sopitta Thurachen, Josef Taher, Matti Lehtom\"aki, Leena Matikainen, Linnea Bl{\aa}field, Mikel Calle Navarro, Antero Kukko, Tomi Westerlund, Harri KaartinenAbstract
Accurate land cover mapping in riverine environments is essential for effective river management, ecological understanding, and geomorphic change monitoring. This study explores the use of Point Transformer v2 (PTv2), an advanced deep neural network architecture designed for point cloud data, for land cover mapping through semantic segmentation of multispectral LiDAR data in real-world riverine environments. We utilize the geometric and spectral information from the 3-channel LiDAR point cloud to map land cover classes, including sand, gravel, low vegetation, high vegetation, forest floor, and water. The PTv2 model was trained and evaluated on point cloud data from the Oulanka river in northern Finland using both geometry and spectral features. To improve the model's generalization in new riverine environments, we additionally investigate multi-dataset training that adds sparsely annotated data from an additional river dataset. Results demonstrated that using the full-feature configuration resulted in performance with a mean Intersection over Union (mIoU) of 0.950, significantly outperforming the geometry baseline. Other ablation studies revealed that intensity and reflectance features were the key for accurate land cover mapping. The multi-dataset training experiment showed improved generalization performance, suggesting potential for developing more robust models despite limited high-quality annotated data. Our work demonstrates the potential of applying transformer-based architectures to multispectral point clouds in riverine environments. The approach offers new capabilities for monitoring sediment transport and other river management applications.
Abstract (translated)
在河流环境中进行精确的土地覆盖制图，对于有效的河流管理、生态理解以及地貌变化监测至关重要。本研究探索了使用专为点云数据设计的先进深度神经网络架构——Point Transformer v2（PTv2），通过真实河流环境中多光谱LiDAR数据的语义分割来实现土地覆盖制图。我们利用3通道LiDAR点云的几何与光谱信息，对包括沙地、砾石、低植被、高植被、林下层和水体在内的土地覆盖类别进行制图。该PTv2模型基于芬兰北部奥兰卡河的几何与光谱特征点云数据进行了训练与评估。为提升模型在新河流环境中的泛化能力，本研究进一步探究了通过添加另一河流数据集中稀疏标注数据的多数据集训练方法。结果表明，采用全特征配置的模型性能达到平均交并比（mIoU）0.950，显著优于仅基于几何特征的基线模型。其他消融实验揭示，强度与反射率特征是实现精确土地覆盖制图的关键。多数据集训练实验显示出改进的泛化性能，表明即使在高质量标注数据有限的情况下，也有潜力开发出更稳健的模型。我们的工作证明了基于Transformer的架构应用于河流环境多光谱点云的潜力，该方法为监测泥沙输运及其他河流管理应用提供了新的能力。
URL
https://arxiv.org/abs/2603.22230
PDF
-
Benchmarking Deep Learning Models for Aerial LiDAR Point Cloud Semantic Segmentation under Real Acquisition Conditions: A Case Study in Navarre
2026-03-23 17:26:41Alex Salvatierra, Jos\'e Antonio Sanz, Christian Guti\'errez, Mikel GalarAbstract
Recent advances in deep learning have significantly improved 3D semantic segmentation, but most models focus on indoor or terrestrial datasets. Their behavior under real aerial acquisition conditions remains insufficiently explored, and although a few studies have addressed similar scenarios, they differ in dataset design, acquisition conditions, and model selection. To address this gap, we conduct an experimental benchmark evaluating several state-of-the-art architectures on a large-scale aerial LiDAR dataset acquired under operational flight conditions in Navarre, Spain, covering heterogeneous urban, rural, and industrial landscapes. This study compares four representative deep learning models, including KPConv, RandLA-Net, Superpoint Transformer, and Point Transformer V3, across five semantic classes commonly found in airborne surveys, such as ground, vegetation, buildings, and vehicles, highlighting the inherent challenges of class imbalance and geometric variability in aerial data. Results show that all tested models achieve high overall accuracy exceeding 93%, with KPConv attaining the highest mean IoU (78.51%) through consistent performance across classes, particularly on challenging and underrepresented categories. Point Transformer V3 demonstrates superior performance on the underrepresented vehicle class (75.11% IoU), while Superpoint Transformer and RandLA-Net trade off segmentation robustness for computational efficiency.
Abstract (translated)
深度学习的最新进展显著提升了3D语义分割性能，但多数模型聚焦于室内或地面数据集。其在真实航拍采集条件下的表现尚未得到充分探索，尽管少数研究涉及类似场景，但在数据集设计、采集条件和模型选择上存在差异。为填补这一空白，本研究通过实验基准测试，在西班牙纳瓦拉地区 operational flight 条件下获取的大规模航拍激光雷达数据集上，评估了多个先进架构。该数据集覆盖异质性城市、乡村及工业景观。研究对比了四种代表性深度学习模型（包括KPConv、RandLA-Net、Superpoint Transformer和Point Transformer V3），针对航测中常见的五类语义目标（如地面、植被、建筑物和车辆）进行分析，凸显了航拍数据中固有的类别不平衡与几何变异挑战。结果表明，所有测试模型整体准确率均超过93%，其中KPConv通过跨类别的稳定表现达到最高平均交并比（78.51%），尤其在具有挑战性的低占比类别上表现突出。Point Transformer V3在低占比的车辆类别上展现最优性能（交并比75.11%），而Superpoint Transformer与RandLA-Net则在分割鲁棒性与计算效率间取得权衡。
URL
https://arxiv.org/abs/2603.22229
PDF
-
CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation
2026-03-23 04:40:35Mohammad Eslami, Dhanvinkumar Ganeshkumar, Saber Kazeminasab, Michael G. Morley, Michael V. Boland, Michael M. Lin, John B. Miller, David S. Friedman, Nazlee Zebardast, Lucia Sobrin, Tobias ElzeAbstract
We present CataractSAM-2, a domain-adapted extension of Meta's Segment Anything Model 2, designed for real-time semantic segmentation of cataract ophthalmic surgery videos with high accuracy. Positioned at the intersection of computer vision and medical robotics, CataractSAM-2 enables precise intraoperative perception crucial for robotic-assisted and computer-guided surgical systems. Furthermore, to alleviate the burden of manual labeling, we introduce an interactive annotation framework that combines sparse prompts with video-based mask propagation. This tool significantly reduces annotation time and facilitates the scalable creation of high-quality ground-truth masks, accelerating dataset development for ocular anterior segment surgeries. We also demonstrate the model's strong zero-shot generalization to glaucoma trabeculectomy procedures, confirming its cross-procedural utility and potential for broader surgical applications. The trained model and annotation toolkit are released as open-source resources, establishing CataractSAM-2 as a foundation for expanding anterior ophthalmic surgical datasets and advancing real-time AI-driven solutions in medical robotics, as well as surgical video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21566
PDF
-
PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation
2026-03-23 03:40:47Gensheng Pei, Xiruo Jiang, Xinhao Cai, Tao Chen, Yazhou Yao, Byeungwoo JeonAbstract
Training-free open-vocabulary semantic segmentation (OVSS) promises rapid adaptation to new label sets without retraining. Yet, many methods rely on heavy post-processing or handle text and vision in isolation, leaving cross-modal geometry underutilized. Others introduce auxiliary vision backbones or multi-model pipelines, which increase complexity and latency while compromising design simplicity. We present PEARL, \textbf{\underline{P}}rocrust\textbf{\underline{e}}s \textbf{\underline{a}}lignment with text-awa\textbf{\underline{r}}e \textbf{\underline{L}}aplacian propagation, a compact two-step inference that follows an align-then-propagate principle. The Procrustes alignment step performs an orthogonal projection inside the last self-attention block, rotating keys toward the query subspace via a stable polar iteration. The text-aware Laplacian propagation then refines per-pixel logits on a small grid through a confidence-weighted, text-guided graph solve: text provides both a data-trust signal and neighbor gating, while image gradients preserve boundaries. In this work, our method is fully training-free, plug-and-play, and uses only fixed constants, adding minimal latency with a small per-head projection and a few conjugate-gradient steps. Our approach, PEARL, sets a new state-of-the-art in training-free OVSS without extra data or auxiliary backbones across standard benchmarks, achieving superior performance under both with-background and without-background protocols.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21528
PDF
-
LiFR-Seg: Anytime High-Frame-Rate Segmentation via Event-Guided Propagation
2026-03-22 08:13:58Xiaoshan Wu, Xiaoyang Lyu, Yifei Yu, Bo Wang, Zhongrui Wang, Xiaojuan QiAbstract
Dense semantic segmentation in dynamic environments is fundamentally limited by the low-frame-rate (LFR) nature of standard cameras, which creates critical perceptual gaps between frames. To solve this, we introduce Anytime Interframe Semantic Segmentation: a new task for predicting segmentation at any arbitrary time using only a single past RGB frame and a stream of asynchronous event data. This task presents a core challenge: how to robustly propagate dense semantic features using a motion field derived from sparse and often noisy event data, all while mitigating feature degradation in highly dynamic scenes. We propose LiFR-Seg, a novel framework that directly addresses these challenges by propagating deep semantic features through time. The core of our method is an uncertainty-aware warping process, guided by an event-driven motion field and its learned, explicit confidence. A temporal memory attention module further ensures coherence in dynamic scenarios. We validate our method on the DSEC dataset and a new high-frequency synthetic benchmark (SHF-DSEC) we contribute. Remarkably, our LFR system achieves performance (73.82% mIoU on DSEC) that is statistically indistinguishable from an HFR upper-bound (within 0.09%) that has full access to the target frame. This work presents a new, efficient paradigm for achieving robust, high-frame-rate perception with low-frame-rate hardware. Project Page: this https URL Code: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21115
PDF
-
CTFS : Collaborative Teacher Framework for Forward-Looking Sonar Image Semantic Segmentation with Extremely Limited Labels
2026-03-22 05:57:36Ping Guo, Chengzhou Li, Guanchen Meng, Qi Jia, Jinyuan Liu, Zhu Liu, Yu Liu, Zhongxuan Luo, Xin FanAbstract
As one of the most important underwater sensing technologies, forward-looking sonar exhibits unique imaging characteristics. Sonar images are often affected by severe speckle noise, low texture contrast, acoustic shadows, and geometric distortions. These factors make it difficult for traditional teacher-student frameworks to achieve satisfactory performance in sonar semantic segmentation tasks under extremely limited labeled data conditions. To address this issue, we propose a Collaborative Teacher Semantic Segmentation Framework for forward-looking sonar images. This framework introduces a multi-teacher collaborative mechanism composed of one general teacher and multiple sonar-specific teachers. By adopting a multi-teacher alternating guidance strategy, the student model can learn general semantic representations while simultaneously capturing the unique characteristics of sonar images, thereby achieving more comprehensive and robust feature modeling. Considering the challenges of sonar images, which can lead teachers to generate a large number of noisy pseudo-labels, we further design a cross-teacher reliability assessment mechanism. This mechanism dynamically quantifies the reliability of pseudo-labels by evaluating the consistency and stability of predictions across multiple views and multiple teachers, thereby mitigating the negative impact caused by noisy pseudo-labels. Notably, on the FLSMD dataset, when only 2% of the data is labeled, our method achieves a 5.08% improvement in mIoU compared to other state-of-the-art approaches.
Abstract (translated)
作为最重要的水下感知技术之一，前视声呐具有独特的成像特性。声呐图像常受严重散斑噪声、低纹理对比度、声影及几何畸变的影响。这些因素导致传统师生框架在标注数据极度有限的条件下难以在声呐语义分割任务中取得理想性能。针对这一问题，我们提出了一种面向前视声呐图像的协同教师语义分割框架。该框架引入由一位通用教师与多位声呐专用教师构成的多教师协同机制，通过采用多教师交替引导策略，使学生模型既能学习通用语义表征，又能同时捕捉声呐图像的独特特征，从而实现更全面稳健的特征建模。考虑到声呐图像的挑战可能导致教师生成大量噪声伪标签，我们进一步设计了跨教师可靠性评估机制。该机制通过评估多视图及多教师间预测的一致性与稳定性，动态量化伪标签的可靠性，从而缓解噪声伪标签带来的负面影响。值得注意的是，在FLSMD数据集上，当仅使用2%标注数据时，本方法在mIoU指标上相较于其他先进方法取得了5.08%的性能提升。
URL
https://arxiv.org/abs/2603.21071
PDF
-
Ordinal Semantic Segmentation Applied to Medical and Odontological Images
2026-03-21 16:05:48Mariana D\'oria Prata Lima, Gilson Antonio Giraldi, Jaime S. CardosoAbstract
Semantic segmentation consists of assigning a semantic label to each pixel according to predefined classes. This process facilitates the understanding of object appearance and spatial relationships, playing an important role in the global interpretation of image content. Although modern deep learning approaches achieve high accuracy, they often ignore ordinal relationships among classes, which may encode important domain knowledge for scene interpretation. In this work, loss functions that incorporate ordinal relationships into deep neural networks are investigated to promote greater semantic consistency in semantic segmentation tasks. These loss functions are categorized as unimodal, quasi-unimodal, and spatial. Unimodal losses constrain the predicted probability distribution according to the class ordering, while quasi-unimodal losses relax this constraint by allowing small variations while preserving ordinal coherence. Spatial losses penalize semantic inconsistencies between neighboring pixels, encouraging smoother transitions in the image space. In particular, this study adapts loss functions originally proposed for ordinal classification to ordinal semantic segmentation. Among them, the Expanded Mean Squared Error (EXP_MSE), the Quasi-Unimodal Loss (QUL), and the spatial Contact Surface Loss using Signal Distance Function (CSSDF) are investigated. These approaches have shown promising results in medical imaging, improving robustness, generalization, and anatomical consistency.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26736
PDF
-
Lean Learning Beyond Clouds: Efficient Discrepancy-Conditioned Optical-SAR Fusion for Semantic Segmentation
2026-03-21 13:14:40Chenxing Meng, Wuzhou Quan, Yingjie Cai, Liqun Cao, Liyan Zhang, Mingqiang WeiAbstract
Cloud occlusion severely degrades the semantic integrity of optical remote sensing imagery. While incorporating Synthetic Aperture Radar (SAR) provides complementary observations, achieving efficient global modeling and reliable cross-modal fusion under cloud interference remains challenging. Existing methods rely on dense global attention to capture long-range dependencies, yet such aggregation indiscriminately propagates cloud-induced noise. Improving robustness typically entails enlarging model capacity, which further increases computational overhead. Given the large-scale and high-resolution nature of remote sensing applications, such computational demands hinder practical deployment, leading to an efficiency-reliability trade-off. To address this dilemma, we propose EDC, an efficiency-oriented and discrepancy-conditioned optical-SAR semantic segmentation framework. A tri-stream encoder with Carrier Tokens enables compact global context modeling with reduced complexity. To prevent noise contamination, we introduce a Discrepancy-Conditioned Hybrid Fusion (DCHF) mechanism that selectively suppresses unreliable regions during global aggregation. In addition, an auxiliary cloud removal branch with teacher-guided distillation enhances semantic consistency under occlusion. Extensive experiments demonstrate that EDC achieves superior accuracy and efficiency, improving mIoU by 0.56\% and 0.88\% on M3M-CR and WHU-OPT-SAR, respectively, while reducing the number of parameters by 46.7\% and accelerating inference by 1.98$\times$. Our implementation is available at this https URL.
Abstract (translated)
云层遮挡会严重破坏光学遥感图像的语义完整性。尽管引入合成孔径雷达（SAR）可提供互补观测，但在云干扰下实现高效的全局建模与可靠的跨模态融合仍具挑战性。现有方法依赖密集的全局注意力来捕获长程依赖，但这种聚合会无差别地传播云层引起的噪声。提升鲁棒性通常需要扩大模型容量，这进一步增加了计算开销。鉴于遥感应用的大规模与高分辨率特性，此类计算需求阻碍了实际部署，导致效率与可靠性之间的权衡。为应对这一困境，我们提出EDC——一个面向效率与差异条件的光学-SAR语义分割框架。采用带载体令牌的三流编码器，以更低复杂度实现紧凑的全局上下文建模。为防止噪声污染，我们引入差异条件混合融合（DCHF）机制，在全局聚合过程中选择性地抑制不可靠区域。此外，配备教师引导蒸馏的辅助去云分支增强了遮挡下的语义一致性。大量实验表明，EDC实现了更优的准确性与效率，在M3M-CR和WHU-OPT-SAR数据集上分别将mIoU提升0.56%和0.88%，同时减少46.7%的参数数量并将推理速度提升1.98倍。我们的实现代码已公开于此https URL。
URL
https://arxiv.org/abs/2603.20811
PDF
-
OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation
2026-03-21 12:07:42Aarush Aggarwal, Akshat Tomar, Amritanshu Tiwari, Sargam GoyalAbstract
Robust semantic segmentation is crucial for safe autonomous driving, yet deployed models remain vulnerable to black-box adversarial attacks when target weights are unknown. Most existing approaches either craft image-wide perturbations or optimize patches for a single architecture, which limits their practicality and transferability. We introduce OmniPatch, a training framework for learning a universal adversarial patch that generalizes across images and both ViT and CNN architectures without requiring access to target model parameters.
Abstract (translated)
鲁棒的语义分割对于安全自动驾驶至关重要，但现有部署模型在目标权重未知时仍易受黑盒对抗攻击。大多数现有方法要么生成全图像扰动，要么针对单一架构优化补丁，这限制了其实用性和迁移性。我们提出OmniPatch，一种用于学习通用对抗补丁的训练框架，该补丁可跨图像泛化，并同时适用于ViT和CNN架构，且无需访问目标模型参数。
URL
https://arxiv.org/abs/2603.20777
PDF
-
An Open Source Computer Vision and Machine Learning Framework for Affordable Life Science Robotic Automation
2026-03-20 19:51:28Zachary Logan, Andrew Dudash, Daniel Negr\'onAbstract
We present an open-source robotic framework that integrates computer vision and machine learning based inverse kinematics to enable low-cost laboratory automation tasks such as colony picking and liquid handling. The system uses a custom trained U-net model for semantic segmentation of microbial cultures, combined with Mixture Density Network for predicating joint angles of a simple 5-DOF robot arm. We evaluated the framework using a modified robot arm, upgraded with a custom liquid handling end-effector. Experimental results demonstrate the framework's feasibility for precise, repeatable operations, with mean positional error below 1 mm and joint angle prediction errors below 4 degrees and colony detection capabilities with IoU score of 0.537 and Dice coefficient of 0.596.
Abstract (translated)
我们提出一个开源机器人框架，该框架整合了计算机视觉与基于机器学习的逆运动学方法，以实现菌落挑取和液体处理等低成本实验室自动化任务。该系统采用自定义训练的U-Net模型对微生物培养物进行语义分割，并结合混合密度网络预测简单5自由度机械臂的关节角度。我们使用改装后的机械臂（升级了定制液体处理末端执行器）对该框架进行了评估。实验结果表明，该框架能够实现精确、可重复的操作，平均位置误差低于1毫米，关节角度预测误差低于4度，菌落检测的IoU分数为0.537，Dice系数为0.596。
URL
https://arxiv.org/abs/2603.20465
PDF
