Semantic_Segmentation
-
Data-Centric Benchmark for Label Noise Estimation and Ranking in Remote Sensing Image Segmentation
2026-02-28 11:46:56Keiller Nogueira, Codrut-Andrei Diaconu, D\'avid Kerekes, Jakob Gawlikowski, C\'edric L\'eonard, Nassim Ait Ali Braham, June Moh Goo, Zichao Zeng, Zhipeng Liu, Pallavi Jain, Andrea Nascetti, Ronny H\"anschAbstract
High-quality pixel-level annotations are essential for the semantic segmentation of remote sensing imagery. However, such labels are expensive to obtain and often affected by noise due to the labor-intensive and time-consuming nature of pixel-wise annotation, which makes it challenging for human annotators to label every pixel accurately. Annotation errors can significantly degrade the performance and robustness of modern segmentation models, motivating the need for reliable mechanisms to identify and quantify noisy training samples. This paper introduces a novel Data-Centric benchmark, together with a novel, publicly available dataset and two techniques for identifying, quantifying, and ranking training samples according to their level of label noise in remote sensing semantic segmentation. Such proposed methods leverage complementary strategies based on model uncertainty, prediction consistency, and representation analysis, and consistently outperform established baselines across a range of experimental settings. The outcomes of this work are publicly available at this https URL.
Abstract (translated)
高质量的像素级标注对于遥感图像语义分割至关重要。然而，由于逐像素标注工作量大且耗时长，这样的标签获取成本高昂，并且容易受到噪声影响，使得人类标注者难以准确地标记每个像素。注释错误会显著降低现代分割模型的表现和鲁棒性，这促使了对可靠机制的需求来识别和量化带有噪音的训练样本。 本文介绍了一种新的数据为中心的基准测试，以及一个新颖、公开可用的数据集和两种技术，用于根据其标签噪声程度来识别、量化和排序遥感语义分割中的训练样本。这些方法利用基于模型不确定性、预测一致性和表示分析的互补策略，并在各种实验设置中均优于现有基线表现。 该研究的结果可在以下网址获取：[此 HTTPS URL]（请将括号内的文本替换为实际链接地址）。
URL
https://arxiv.org/abs/2603.00604
PDF
-
TP-Spikformer: Token Pruned Spiking Transformer
2026-02-28 07:53:26Wenjie Wei, Xiaolong Zhou, Malu Zhang, Ammar Belatreche, Qian Sun, Yimeng Shan, Dehao Zhang, Zijian Zhou, Zeyu Ma, Yang Yang, Haizhou LiAbstract
Spiking neural networks (SNNs) offer an energy-efficient alternative to traditional neural networks due to their event-driven computing paradigm. However, recent advancements in spiking transformers have focused on improving accuracy with large-scale architectures, which require significant computational resources and limit deployment on resource-constrained devices. In this paper, we propose a simple yet effective token pruning method for spiking transformers, termed TP-Spikformer, that reduces storage and computational overhead while maintaining competitive performance. Specifically, we first introduce a heuristic spatiotemporal information-retaining criterion that comprehensively evaluates tokens' importance, assigning higher scores to informative tokens for retention and lower scores to uninformative ones for pruning. Based on this criterion, we propose an information-retaining token pruning framework that employs a block-level early stopping strategy for uninformative tokens, instead of removing them outright. This also helps preserve more information during token pruning. We demonstrate the effectiveness, efficiency and scalability of TP-Spikformer through extensive experiments across diverse architectures, including Spikformer, QKFormer and Spike-driven Transformer V1 and V3, and a range of tasks such as image classification, object detection, semantic segmentation and event-based object tracking. Particularly, TP-Spikformer performs well in a training-free manner. These results reveal its potential as an efficient and practical solution for deploying SNNs in real-world applications with limited computational resources.
Abstract (translated)
脉冲神经网络（SNN）由于其事件驱动的计算范式，为传统的神经网络提供了一种节能的替代方案。然而，最近在脉冲变压器方面的进展主要集中在通过大规模架构来提高准确性上，这需要大量的计算资源，并限制了这些模型在资源受限设备上的部署。在这篇论文中，我们提出了一种简单而有效的脉冲变压器标记剪枝方法，名为TP-Spikformer，该方法减少了存储和计算开销，同时保持了竞争力的性能。 具体来说，首先我们引入了一个基于时空信息保留的标准来评估标记的重要性，给重要的标记分配较高的分数以保留，对不太重要的标记则分配较低的分数进行裁减。在此标准基础上，我们提出了一种信息保留标记剪枝框架，采用块级早期停止策略，而不是直接去除那些不太重要的标记，这也有助于在标记裁剪过程中保存更多信息。 通过广泛的实验验证了TP-Spikformer的有效性、效率和可扩展性，这些实验涵盖了多种架构（如Spikformer、QKFormer以及Spike驱动的Transformer V1 和V3），以及包括图像分类、目标检测、语义分割及基于事件的目标跟踪等多种任务。特别值得注意的是，TP-Spikformer在无需训练的情况下也能表现出色。 这些结果揭示了其作为部署资源受限的真实世界应用中的SNN的一个高效且实用解决方案的巨大潜力。
URL
https://arxiv.org/abs/2603.00527
PDF
-
Explainable Continuous-Time Mask Refinement with Local Self-Similarity Priors for Medical Image Segmentation
2026-02-28 04:31:49Rajdeep Chatterjee, Sudip Chakrabarty, Trishaani AcharjeeAbstract
Accurate semantic segmentation of foot ulcers is essential for automated wound monitoring, yet boundary delineation remains challenging due to tissue heterogeneity and poor contrast with surrounding skin. To overcome the limitations of standard intensity-based networks, we present LSS-LTCNet:an ante-hoc explainable framework synergizing deterministic structural priors with continuous-time neural dynamics. Our architecture departs from traditional black-box models by employing a Local Self-Similarity (LSS) mechanism that extracts dense, illumination-invariant texture descriptors to explicitly disentangle necrotic tissue from background artifacts. To enforce topological precision, we introduce a Liquid Time-Constant (LTC) refinement module that treats boundary evolution as an ODEgoverned dynamic system, iteratively refining masks over continuous time-steps. Comprehensive evaluation on the MICCAI FUSeg dataset demonstrates that LSS-LTCNet achieves state-of-the-art boundary alignment, securing a peak Dice score of 86.96% and an exceptional 95th percentile Hausdorff Distance (HD95) of 8.91 pixels. Requiring merely 25.70M parameters, the model significantly outperforms heavier U-Net and transformer baselines in efficiency. By providing inherent visual audit trails alongside high-fidelity predictions, LSS-LTCNet offers a robust and transparent solution for computer-aided diagnosis in mobile healthcare (mHealth) settings.
Abstract (translated)
足部溃疡的精确语义分割对于自动化伤口监测至关重要，但由于组织异质性和与周围皮肤对比度差，边界划分仍然具有挑战性。为了克服基于强度的标准网络的局限性，我们提出了LSS-LTCNet：一种事前可解释框架，该框架结合了确定性的结构先验和连续时间神经动力学。我们的架构通过采用局部自相似（Local Self-Similarity, LSS）机制来提取密集且不受光照影响的纹理描述符而不同于传统的黑盒模型，这种机制能够明确区分坏死组织与背景伪影。为了确保拓扑精度，我们引入了一个液态时间常数（Liquid Time-Constant, LTC）细化模块，该模块将边界演化视为一个受ODE（常微分方程）支配的动力系统，并在连续的时间步长中迭代地优化掩模。 在MICCAI FUSeg数据集上的全面评估表明，LSS-LTCNet实现了最先进的边界对齐效果，达到了86.96%的峰值Dice分数和出色的第95百分位Hausdorff距离（HD95）为8.91像素。该模型仅需25.70M参数，在效率上显著优于更重的U-Net和变压器基线。通过提供固有的视觉审核轨迹以及高保真的预测，LSS-LTCNet为移动医疗环境中的计算机辅助诊断提供了可靠且透明的解决方案。
URL
https://arxiv.org/abs/2603.00459
PDF
-
Unsupervised Semantic Segmentation in Synchrotron Computed Tomography with Self-Correcting Pseudo Labels
2026-02-27 23:15:41Austin Yunker, Peter Kenesei, Hemant Sharma, Jun-Sang Park, Antonino Miceli, Rajkumar KettimuthuAbstract
X-ray computed tomography (CT) is a widely used imaging technique that provides detailed examinations into the internal structure of an object with synchrotron CT (SR-CT) enabling improved data quality by using higher energy, monochromatic X-rays. While SR-CT allows for improved resolution, time-resolved experimentation, and reduced imaging artifacts, it also produces significantly larger datasets than conventional CT. Accurate and efficient evaluation of these datasets is a critical component of these workflows; yet is often done manually representing a major bottleneck in the analysis phase. While deep learning has emerged as a powerful tool capable of providing a wide range of purely data-driven solutions, it requires a substantial amount of labeled data for training and manual annotation of SR-CT datasets is impractical in practice. In this paper, we introduce a novel framework that enables automatic segmentation of large, high-resolution SR-CT datasets by eliminating the need to hand label images for deep learning training. First, we generate pseudo labels by clustering on the voxel values identifying regions in the volume with similar attenuation coefficients producing an initial semantic map. Afterwards, we train a segmentation model on the pseudo labels before utilizing the Unbiased Teacher approach to self-correct them ensuring accurate final segmentations. We find our approach improves pixel-wise accuracy and mIoU by 13.31% and 15.94%, respectively, over the baseline pseudo labels when using a magnesium crystal SR-CT sample. Additionally, we extensively evaluate the different components of our workflow including segmentation model, loss function, pseudo labeling strategy, and input type. Finally, we evaluate our approach on to two additional samples highlighting our frameworks ability to produce segmentations that are considerably better than the original pseudo labels.
Abstract (translated)
X射线计算机断层扫描（CT）是一种广泛使用的成像技术，可以详细检查物体的内部结构。同步加速器CT（SR-CT）通过使用高能单色X射线，能够提供改进的数据质量。虽然SR-CT允许更高的分辨率、时间解析实验以及减少成像伪影，但它也会生成比传统CT大得多的数据集。这些大数据集的准确和高效评估是工作流程中的关键组成部分；然而，在分析阶段通常手动进行，这代表了主要瓶颈之一。尽管深度学习已经作为一种能够提供广泛数据驱动解决方案的强大工具出现，它需要大量标注的数据来进行训练，并且在实践中对SR-CT数据集的手动注释几乎是不可行的。 本文介绍了一种新的框架，该框架能够在不需人工标记图像进行深度学习训练的情况下自动分割大规模、高分辨率的SR-CT数据集。首先，我们通过聚类体积中的像素值生成伪标签，识别具有相似衰减系数的区域，以产生初始语义图谱。之后，在伪标签上训练一个分割模型，并利用无偏教师方法自我纠正它们，确保最终分割准确。 我们的研究发现：在使用镁晶体SR-CT样本时，与基线伪标签相比，我们的方法分别提高了像素级精度13.31%和mIoU（平均交并比）15.94%。此外，我们还广泛评估了工作流程的不同组成部分，包括分割模型、损失函数、伪标记策略以及输入类型。最后，我们在两个额外样本上评估了我们的方法，突显了框架生成的分割结果优于原始伪标签的能力。 该研究提出的方法为大规模高分辨率SR-CT数据集提供了自动分割的可能性，并展示了深度学习技术在解决实际问题中的潜力和挑战。
URL
https://arxiv.org/abs/2603.00372
PDF
-
Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition
2026-02-27 10:11:12Mohammadreza Heidarianbaei, Mareike Dorozynski, Hubert Kanyamahanga, Max Mehltretter, Franz RottensteinerAbstract
In this paper, we propose ReSeg-CLIP, a new training-free Open-Vocabulary Semantic Segmentation method for remote sensing data. To compensate for the problems of vision language models, such as CLIP in semantic segmentation caused by inappropriate interactions within the self-attention layers, we introduce a hierarchical scheme utilizing masks generated by SAM to constrain the interactions at multiple scales. We also present a model composition approach that averages the parameters of multiple RS-specific CLIP variants, taking advantage of a new weighting scheme that evaluates representational quality using varying text prompts. Our method achieves state-of-the-art results across three RS benchmarks without additional training.
Abstract (translated)
在这篇论文中，我们提出了一种新的无需训练的开放词汇语义分割方法ReSeg-CLIP，专门用于遥感数据。为了弥补如CLIP等视觉语言模型在语义分割过程中由于自注意力层内部不适当交互而造成的问题，我们引入了一个分层次方案，利用SAM生成的掩码来约束多尺度下的交互过程。此外，我们还提出了一种模型组合方法，该方法通过平均多个针对遥感特定设计的CLIP变体的参数，并采用一种新的加权方案，使用不同文本提示来评估表示质量。我们的方法在无需额外训练的情况下，在三个遥感基准数据集上取得了最先进的结果。
URL
https://arxiv.org/abs/2602.23869
PDF
-
CycleBEV: Regularizing View Transformation Networks via View Cycle Consistency for Bird's-Eye-View Semantic Segmentation
2026-02-27 00:56:02Jeongbin Hong, Dooseop Choi, Taeg-Hyun An, Kyounghwan An, Kyoung-Wook MinAbstract
Transforming image features from perspective view (PV) space to bird's-eye-view (BEV) space remains challenging in autonomous driving due to depth ambiguity and occlusion. Although several view transformation (VT) paradigms have been proposed, the challenge still remains. In this paper, we propose a new regularization framework, dubbed CycleBEV, that enhances existing VT models for BEV semantic segmentation. Inspired by cycle consistency, widely used in image distribution modeling, we devise an inverse view transformation (IVT) network that maps BEV segmentation maps back to PV segmentation maps and use it to regularize VT networks during training through cycle consistency losses, enabling them to capture richer semantic and geometric information from input PV images. To further exploit the capacity of the IVT network, we introduce two novel ideas that extend cycle consistency into geometric and representation spaces. We evaluate CycleBEV on four representative VT models covering three major paradigms using the large-scale nuScenes dataset. Experimental results show consistent improvements -- with gains of up to 0.74, 4.86, and 3.74 mIoU for drivable area, vehicle, and pedestrian classes, respectively -- without increasing inference complexity, since the IVT network is used only during training. The implementation code is available at this https URL.
Abstract (translated)
将图像特征从透视视图（PV）空间转换到鸟瞰视图（BEV）空间，在自动驾驶中仍然是一个挑战，主要是由于深度模糊和遮挡问题。尽管已经提出了几种视角变换（VT）范式，但这一难题依然存在。在本文中，我们提出了一种新的正则化框架，称为CycleBEV，该框架能够增强现有的用于鸟瞰视图语义分割的视角变换模型。 受到循环一致性概念的启发，这种概念广泛应用于图像分布建模领域，我们设计了一个逆向视角变换（IVT）网络，它将鸟瞰视图分割映射转换回透视视图分割映射，并利用该网络在训练过程中通过循环一致性损失来正则化VT模型。这种方法使它们能够从输入的PV图像中捕捉到更丰富的语义和几何信息。 为了进一步挖掘IVT网络的能力，我们引入了两个新颖的想法，这些想法将循环一致性扩展到了几何空间和表示空间。我们在nuScenes大规模数据集上对CycleBEV进行了评估，涵盖了四个具有代表性的VT模型以及三大主要范式。 实验结果表明，在不增加推理复杂度的情况下（由于IVT网络仅在训练期间使用），实现了显著改进——对于可驾驶区域、车辆和行人等类别的平均交并比(mIoU)分别提高了0.74、4.86和3.74。这些结果证明了CycleBEV的有效性。 实现代码可在提供的链接处获取（原文中的链接请自行查找）。
URL
https://arxiv.org/abs/2602.23575
PDF
-
Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception
2026-02-26 14:58:59Yiding Sun, Jihua Zhu, Haozhe Cheng, Chaoyi Lu, Zhichuan Yang, Lin Chen, Yaonan WangAbstract
Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel "Align then Adapt" (PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 \% accuracy on 3D action recognition, $+8.7 \%$ on 4 D action segmentation, and 84.06\% on 4D semantic segmentation.
Abstract (translated)
点云视频理解在机器人技术中至关重要，因为它能准确地编码运动和场景交互。我们认识到，相较于3D数据集而言，4D数据集稀缺得多，这阻碍了自监督4D模型的可扩展性。一种有前景的替代方案是从3D预训练模型迁移至4D感知任务。然而，严格的实证分析揭示出两种关键限制，即过度拟合和模态差距问题，这些问题阻碍了迁移能力。 为了解决这些挑战，我们开发了一种新颖的“对齐后适应”（PointATA）范式，将参数效率高的迁移学习分解成两个连续阶段。利用最优传输理论来量化3D与4D数据集之间的分布差异，使我们的提议点对齐嵌入器能够在第一阶段训练以缓解潜在的模态差距问题。为减轻过度拟合，在第二阶段集成高效的点视频适配器和空间上下文编码器到冻结的3D主干中，从而增强时间建模能力。 值得注意的是，通过上述工程设计，PointATA使一个无时间知识的预训练3D模型能够以较小的参数成本推理出动态视频内容。广泛的实验表明，PointATA可以与强大的全微调模型相匹配甚至超越其性能，同时享受参数效率的优势，例如在3D动作识别中达到97.21%的准确率，在4D动作分割中提高8.7%，以及在4D语义分割中达到84.06%的成绩。
URL
https://arxiv.org/abs/2602.23069
PDF
-
SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Segmentation
2026-02-26 11:07:51Qinfeng Zhu, Yunxi Jiang, Lei FanAbstract
Panoramic semantic segmentation models are typically trained under a strict gravity-aligned assumption. However, real-world captures often deviate from this canonical orientation due to unconstrained camera motions, such as the rotational jitter of handheld devices or the dynamic attitude shifts of aerial platforms. This discrepancy causes standard spherical Transformers to overfit global latitude cues, leading to performance collapse under 3D reorientations. To address this, we introduce SO3UFormer, a rotation-robust architecture designed to learn intrinsic spherical features that are less sensitive to the underlying coordinate frame. Our approach rests on three geometric pillars: (1) an intrinsic feature formulation that decouples the representation from the gravity vector by removing absolute latitude encoding; (2) quadrature-consistent spherical attention that accounts for non-uniform sampling densities; and (3) a gauge-aware relative positional mechanism that encodes local angular geometry using tangent-plane projected angles and discrete gauge pooling, avoiding reliance on global axes. We further use index-based spherical resampling together with a logit-level SO(3)-consistency regularizer during training. To rigorously benchmark robustness, we introduce Pose35, a dataset variant of Stanford2D3D perturbed by random rotations within $\pm 35^\circ$. Under the extreme test of arbitrary full SO(3) rotations, existing SOTAs fail catastrophically: the baseline SphereUFormer drops from 67.53 mIoU to 25.26 mIoU. In contrast, SO3UFormer demonstrates remarkable stability, achieving 72.03 mIoU on Pose35 and retaining 70.67 mIoU under full SO(3) rotations.
Abstract (translated)
全景语义分割模型通常在严格的重力对齐假设下进行训练。然而，由于手持设备的旋转抖动或空中平台的姿态动态变化等不受约束的相机运动，现实世界中的捕捉往往会偏离这种规范的方向。这种差异导致标准球形变换器过度拟合全球纬度线索，在3D重新定向时性能崩溃。为解决此问题，我们引入了SO3UFormer，这是一种设计用于学习对底层坐标框架不那么敏感的固有球体特征、从而更加耐旋转变化的架构。 我们的方法基于三个几何支柱：(1)一种通过去除绝对纬度编码来解耦表示与重力矢量的方法，形成了内在特性表述；(2)一致化的球形注意力机制，考虑了非均匀采样密度的影响；以及 (3) 一个感知规范的相对位置机制，使用切平面投影角度和离散规范池化方法来编码局部角几何形状，从而避免依赖全局轴。 我们进一步利用基于索引的球体重采样技术，并在训练过程中采用logit级别的SO(3)-一致性正则器。为了严格评估鲁棒性，我们引入了Pose35数据集，这是Stanford2D3D经过随机旋转（±35°内）扰动后的变种。 在极端的任意全SO(3)旋转测试中，现有的最先进的模型表现灾难性失败：SphereUFormer的基础线从67.53 mIoU骤降至25.26 mIoU。相比之下，SO3UFormer表现出卓越的稳定性，在Pose35上实现了72.03 mIoU，并在全SO(3)旋转下保持了70.67 mIoU的成绩。
URL
https://arxiv.org/abs/2602.22867
PDF
-
A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling
2026-02-26 10:32:08Chong Wang, Yabin Zhang, Yunhe Gao, Maya Varma, Clemence Mottez, Faidra Patsatzi, Jiaming Liu, Jin Long, Jean-Benoit Delbrouck, Sergios Gatidis, Akshay S. Chaudhari, Curtis P. LanglotzAbstract
Foundation models for medical imaging are typically pretrained on increasingly large datasets, following a "scale-at-all-costs" paradigm. However, this strategy faces two critical challenges: large-scale medical datasets often contain substantial redundancy and severe class imbalance that bias representation learning toward over-represented patterns, and indiscriminate training regardless of heterogeneity in data quality incurs considerable computational inefficiency. Here we demonstrate that active, principled data curation during pretraining can serve as a viable, cost-effective alternative to brute-force dataset enlargement. We introduce CheXficient, a chest X-ray (CXR) foundation model that selectively prioritizes informative training samples. CheXficient is pretrained on only 22.7% of 1,235,004 paired CXR images and reports while consuming under 27.3% of the total compute budget, yet achieving comparable or superior performance to its full-data counterpart and other large-scale pretrained models. We assess CheXficient across 20 individual benchmarks spanning 5 task types, including non-adapted off-the-shelf evaluations (zero-shot findings classification and crossmodal retrieval) and adapted downstream tasks (disease prediction, semantic segmentation, and radiology report generation). Further analyses show that CheXficient systematically prioritizes under-represented training samples, improving generalizability on long-tailed or rare conditions. Overall, our work offers practical insights into the data and computation demands for efficient pretraining and downstream adaptation of medical vision-language foundation models.
Abstract (translated)
医学成像的基础模型通常在越来越大的数据集上进行预训练，遵循“不惜一切代价扩大规模”的理念。然而，这种策略面临两个关键挑战：大规模的医疗数据集中往往包含大量的冗余和严重的类别不平衡，这会导致表征学习偏向于过度表示的模式；此外，在不考虑数据质量异质性的情况下无差别地进行训练会带来相当大的计算低效性。在这里，我们证明了在预训练期间采取主动、有原则的数据管理可以作为单纯扩大数据集规模的一种可行且成本效益更高的替代方案。 为此，我们引入了一种胸部X光（CXR）基础模型——CheXficient，该模型选择性地优先考虑具有信息量的训练样本。CheXficient仅在1,235,004对配对的CXR图像和报告中的22.7%上进行预训练，并且计算资源消耗不到总预算的27.3%，但其性能与全数据集版本和其他大规模预训练模型相当或更优。我们通过对包括零样本发现分类、跨模态检索（未适应现成评估）、疾病预测、语义分割和放射报告生成在内的五类任务类型中的二十个独立基准进行评测，来评估CheXficient的性能。 进一步分析表明，CheXficient系统地优先考虑代表性较低的训练样本，在长尾或罕见条件下具有更好的泛化能力。总体而言，我们的工作为医学视觉-语言基础模型的有效预训练和下游任务适应的数据及计算需求提供了实用见解。
URL
https://arxiv.org/abs/2602.22843
PDF
-
Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos
2026-02-25 16:38:53Matthew Strong, Wei-Jer Chang, Quentin Herau, Jiezhi Yang, Yihan Hu, Chensheng Peng, Wei ZhanAbstract
Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos. Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks. Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks. These geometry and motion-aware features position LFG as a compelling video-centric foundation model for autonomous driving.
Abstract (translated)
在线可用的以自我为中心的驾驶视频为自主驾驶提供了丰富的视觉数据来源，但缺乏标注使得学习捕捉语义结构和3D几何的表示变得困难。近期，在大型前馈空间模型方面的进展表明，点地图和自运动可以在一次前向传递中被推断出来，这预示着可扩展驾驶感知的一个有前景的方向。因此，我们提出了一种无标签、由教师指导的框架，直接从未标注视频中学习自主驾驶表示。与之前的自我监督方法主要关注帧到帧的一致性不同，我们认为安全和反应性的驾驶关键在于时间上下文。为此，我们利用了一个配备轻量级自回归模块的前馈架构，并通过多模态监管信号进行训练，这些信号引导模型同时预测当前和未来的点地图、相机姿态、语义分割以及运动掩码。 多模态教师提供序列级别的伪监督，使LFG能够从没有任何姿势、标签或激光雷达数据的原始YouTube视频中学习统一的伪4D表示。生成的编码器不仅在下游自动驾驶规划任务（如NAVSIM基准测试）上有效转移，并且仅使用单目相机就超过了多摄像头和激光雷达基线方法，而且还在一系列语义、几何以及定性和运动预测任务上表现出色。 这种具有几何与运动意识的功能使LFG成为自主驾驶视频中心基础模型的有力候选者。
URL
https://arxiv.org/abs/2602.22091
PDF
-
Assessing airborne laser scanning and aerial photogrammetry for deep learning-based stand delineation
2026-02-25 09:16:28H{\aa}kon N{\ae}ss Sandum, Hans Ole {\O}rka, Oliver Tomic, Terje GobakkenAbstract
Accurate forest stand delineation is essential for forest inventory and management but remains a largely manual and subjective process. A recent study has shown that deep learning can produce stand delineations comparable to expert interpreters when combining aerial imagery and airborne laser scanning (ALS) data. However, temporal misalignment between data sources limits operational scalability. Canopy height models (CHMs) derived from digital photogrammetry (DAP) offer better temporal alignment but may smoothen canopy surface and canopy gaps, raising the question of whether they can reliably replace ALS-derived CHMs. Similarly, the inclusion of a digital terrain model (DTM) has been suggested to improve delineation performance, but has remained untested in published literature. Using expert-delineated forest stands as reference data, we assessed a U-Net-based semantic segmentation framework with municipality-level cross-validation across six municipalities in southeastern Norway. We compared multispectral aerial imagery combined with (i) an ALS-derived CHM, (ii) a DAP-derived CHM, and (iii) a DAP-derived CHM in combination with a DTM. Results showed comparable performance across all data combinations, reaching overall accuracy values between 0.90-0.91. Agreement between model predictions was substantially larger than agreement with the reference data, highlighting both model consistency and the inherent subjectivity of stand delineation. The similar performance of DAP-CHMs, despite the reduced structural detail, and the lack of improvements of the DTM indicate that the framework is resilient to variations in input data. These findings indicate that large datasets for deep learning-based stand delineations can be assembled using projects including temporally aligned ALS data and DAP point clouds.
Abstract (translated)
准确的森林地块划分对于林业调查和管理至关重要，但这一过程仍然主要依赖手工操作且具有主观性。最近的一项研究表明，深度学习技术在结合航空影像与机载激光扫描（ALS）数据时，能够产生与专家解释结果相当的地块划分效果。然而，由于不同数据源之间的时间错配问题限制了其实际应用中的可扩展性。基于数字摄影测量（DAP）生成的冠层高度模型（CHM）提供了更好的时间对齐度，但可能会使树冠表面和空隙变得平滑，从而引发了它们是否能够可靠地替代ALS生成的冠层高度模型的问题。同样，虽然有人建议引入数字地形模型（DTM）可以提高地块划分性能，但在已发表文献中尚未对其进行测试。 我们使用专家划定的森林地块作为参考数据，在挪威东南部六个市镇范围内采用基于U-Net的语义分割框架进行跨市镇验证。我们比较了多光谱航空影像与以下三种组合：(i) ALS生成的冠层高度模型，(ii) DAP生成的冠层高度模型以及(iii) 在DAP生成的冠层高度模型基础上结合数字地形模型（DTM）。结果显示，在所有数据组合中，模型性能相当，总体准确率在0.90-0.91之间。模型预测之间的吻合度显著高于与参考数据的吻合程度，这既体现了模型的一致性也揭示了地块划分过程中的内在主观性。 尽管DAP生成的冠层高度模型减少了结构细节，其性能仍然类似，并且DTM未能带来改进的效果表明该框架具有较强的输入数据变异性抗性。这些发现暗示通过包括时间对齐的ALS数据和DAP点云项目可以构建用于基于深度学习的地块划分的大规模数据集。
URL
https://arxiv.org/abs/2602.21709
PDF
-
Tokenizing Semantic Segmentation with RLE
2026-02-25 06:44:13Abhineet Singh, Justin Rozeboom, Nilanjan RayAbstract
This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks and then train a modified version of Pix2Seq \cite{p2s} to output these RLE tokens through autoregression. We propose novel tokenization strategies to compress the length of the token sequence to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our proposed models on two datasets to show that they are competitive with the state of the art in spite of being bottlenecked by our limited computational resources.
Abstract (translated)
本文提出了一种新的统一方法，用于图像和视频的语义分割，该方法通过使用语言模型以离散令牌序列的形式输出掩码来实现。我们采用游程编码（RLE）将分割掩码进行离散化处理，并训练了Pix2Seq的一个修改版本\cite{p2s}，使其能够通过自回归方式生成这些RLE令牌。为了使该方法在视频应用中具有实用性，我们提出了一些新颖的标记策略来压缩令牌序列长度。此外，我们展示了如何将实例信息整合到标记化过程中以执行全景分割（panoptic segmentation）。我们在两个数据集上评估了所提出的模型，并表明尽管受到计算资源限制的影响，我们的模型仍能与当前最佳方法相媲美。
URL
https://arxiv.org/abs/2602.21627
PDF
-
DA-Cal: Towards Cross-Domain Calibration in Semantic Segmentation
2026-02-24 13:03:41Wangkai Li, Rui Sun, Zhaoyang Li, Yujia Chen, Tianzhu ZhangAbstract
While existing unsupervised domain adaptation (UDA) methods greatly enhance target domain performance in semantic segmentation, they often neglect network calibration quality, resulting in misalignment between prediction confidence and actual accuracy -- a significant risk in safety-critical applications. Our key insight emerges from observing that performance degrades substantially when soft pseudo-labels replace hard pseudo-labels in cross-domain scenarios due to poor calibration, despite the theoretical equivalence of perfectly calibrated soft pseudo-labels to hard pseudo-labels. Based on this finding, we propose DA-Cal, a dedicated cross-domain calibration framework that transforms target domain calibration into soft pseudo-label optimization. DA-Cal introduces a Meta Temperature Network to generate pixel-level calibration parameters and employs bi-level optimization to establish the relationship between soft pseudo-labels and UDA supervision, while utilizing complementary domain-mixing strategies to prevent overfitting and reduce domain discrepancies. Experiments demonstrate that DA-Cal seamlessly integrates with existing self-training frameworks across multiple UDA segmentation benchmarks, significantly improving target domain calibration while delivering performance gains without inference overhead. The code will be released.
Abstract (translated)
虽然现有的无监督领域适应（UDA）方法在语义分割的目标域性能上有了显著的提升，但它们通常忽略了网络校准质量的问题，导致预测置信度与实际准确性之间存在不一致——这在安全关键的应用中是一个重大风险。我们的主要见解来源于观察到，在跨域场景中当软伪标签取代硬伪标签时，尽管理论上完美校准的软伪标签等同于硬伪标签，但由于校准不佳，性能会显著下降。 基于这一发现，我们提出了DA-Cal，这是一种专门针对跨域校准的框架，它将目标领域的校准转化为对软伪标签的优化。DA-Cal 引入了一个元温度网络来生成像素级校准参数，并采用双层优化方法建立软伪标签与UDA监督之间的关系，同时利用互补的领域混合策略来防止过拟合并减少领域差异。 实验表明，DA-Cal 可以无缝集成到现有的自我训练框架中，在多个UDA分割基准测试上显著提升了目标域的校准质量，而不会增加推理开销，并且提高了性能。代码将对外发布。
URL
https://arxiv.org/abs/2602.20860
PDF
-
Onboard-Targeted Segmentation of Straylight in Space Camera Sensors
2026-02-24 09:15:13Riccardo Gallon, Fabian Schiemenz, Alessandra Menicucci, Eberhard GillAbstract
This study details an artificial intelligence (AI)-based methodology for the semantic segmentation of space camera faults. Specifically, we address the segmentation of straylight effects induced by solar presence around the camera's Field of View (FoV). Anomalous images are sourced from our published dataset. Our approach emphasizes generalization across diverse flare textures, leveraging pre-training on a public dataset (Flare7k++) including flares in various non-space contexts to mitigate the scarcity of realistic space-specific data. A DeepLabV3 model with MobileNetV3 backbone performs the segmentation task. The model design targets deployment in spacecraft resource-constrained hardware. Finally, based on a proposed interface between our model and the onboard navigation pipeline, we develop custom metrics to assess the model's performance in the system-level context.
Abstract (translated)
这项研究详细介绍了基于人工智能（AI）的方法，用于对空间相机故障进行语义分割。具体而言，我们解决了由于太阳在相机视场（FoV）周围存在而引起的杂光效应的分割问题。异常图像来源于我们已发布的数据集。我们的方法强调跨多种耀斑纹理的一般化能力，并利用公开数据集（Flare7k++），该数据集中包含各种非空间环境中的耀斑，以缓解真实太空特定数据不足的问题。 采用带有MobileNetV3骨干网络的DeepLabV3模型来执行分割任务。模型设计旨在适应航天器资源受限硬件的需求。最后，基于我们模型与机载导航管道之间提出的接口，开发了自定义指标以评估模型在系统级别上下文中的性能。
URL
https://arxiv.org/abs/2602.20709
PDF
-
Using Unsupervised Domain Adaptation Semantic Segmentation for Pulmonary Embolism Detection in Computed Tomography Pulmonary Angiogram Images
2026-02-23 14:33:24Wen-Liang Lin, Yun-Chien ChengAbstract
While deep learning has demonstrated considerable promise in computer-aided diagnosis for pulmonary embolism (PE), practical deployment in Computed Tomography Pulmonary Angiography (CTPA) is often hindered by "domain shift" and the prohibitive cost of expert annotations. To address these challenges, an unsupervised domain adaptation (UDA) framework is proposed, utilizing a Transformer backbone and a Mean-Teacher architecture for cross-center semantic segmentation. The primary focus is placed on enhancing pseudo-label reliability by learning deep structural information within the feature space. Specifically, three modules are integrated and designed for this task: (1) a Prototype Alignment (PA) mechanism to reduce category-level distribution discrepancies; (2) Global and Local Contrastive Learning (GLCL) to capture both pixel-level topological relationships and global semantic representations; and (3) an Attention-based Auxiliary Local Prediction (AALP) module designed to reinforce sensitivity to small PE lesions by automatically extracting high-information slices from Transformer attention maps. Experimental validation conducted on cross-center datasets (FUMPE and CAD-PE) demonstrates significant performance gains. In the FUMPE -> CAD-PE task, the IoU increased from 0.1152 to 0.4153, while the CAD-PE -> FUMPE task saw an improvement from 0.1705 to 0.4302. Furthermore, the proposed method achieved a 69.9% Dice score in the CT -> MRI cross-modality task on the MMWHS dataset without utilizing any target-domain labels for model selection, confirming its robustness and generalizability for diverse clinical environments.
Abstract (translated)
尽管深度学习在计算机辅助肺栓塞（PE）诊断中展示了很大的潜力，但在计算机断层扫描肺血管造影（CTPA）中的实际部署通常会受到“领域偏移”和专家标注成本过高的限制。为了解决这些挑战，提出了一种无监督域自适应（UDA）框架，该框架采用了Transformer骨干网络以及Mean-Teacher架构来进行跨中心的语义分割。主要关注通过学习特征空间内的深层结构信息来提高伪标签的可靠性。为此，设计并集成了三个模块：(1) 原型对齐(PA)机制以减少类别级别的分布差异；(2) 全局和局部对比学习(GLCL)，用于捕捉像素级拓扑关系以及全局语义表示；(3) 一种基于注意力的辅助局部预测(AALP)模块，通过自动从Transformer注意图中提取高信息切片来增强对小PE病灶的敏感性。 在跨中心数据集（FUMPE和CAD-PE）上的实验验证显示了显著性能提升。在FUMPE到CAD-PE的任务中，IoU从0.1152增加到了0.4153；而在CAD-PE到FUMPE的任务中，则是从0.1705提高至了0.4302。此外，在MMWHS数据集上的CT到MRI跨模态任务中，所提出的方法在没有使用任何目标领域标签的情况下进行模型选择时，达到了69.9%的Dice分数，这证实了其在多样化的临床环境中的鲁棒性和泛化能力。
URL
https://arxiv.org/abs/2602.19891
PDF
-
Brewing Stronger Features: Dual-Teacher Distillation for Multispectral Earth Observation
2026-02-23 14:09:01Filip Wolf, Bla\v{z} Rolih, Luka \v{C}ehovin ZajcAbstract
Foundation models are transforming Earth Observation (EO), yet the diversity of EO sensors and modalities makes a single universal model unrealistic. Multiple specialized EO foundation models (EOFMs) will likely coexist, making efficient knowledge transfer across modalities essential. Most existing EO pretraining relies on masked image modeling, which emphasizes local reconstruction but provides limited control over global semantic structure. To address this, we propose a dual-teacher contrastive distillation framework for multispectral imagery that aligns the student's pretraining objective with the contrastive self-distillation paradigm of modern optical vision foundation models (VFMs). Our approach combines a multispectral teacher with an optical VFM teacher, enabling coherent cross-modal representation learning. Experiments across diverse optical and multispectral benchmarks show that our model adapts to multispectral data without compromising performance on optical-only inputs, achieving state-of-the-art results in both settings, with an average improvement of 3.64 percentage points in semantic segmentation, 1.2 in change detection, and 1.31 in classification tasks. This demonstrates that contrastive distillation provides a principled and efficient approach to scalable representation learning across heterogeneous EO data sources. Code: Coming soon.
Abstract (translated)
基础模型正在改变地球观测（EO），然而，各种不同的EO传感器和模式使得创建单一通用模型不切实际。多个专业的EO基础模型(EOFMs)可能会共存，因此在不同模态之间进行有效的知识转移至关重要。现有的大多数EO预训练依赖于掩码图像建模，这强调了局部重建但对全局语义结构的控制有限。为了解决这个问题，我们提出了一种双教师对比蒸馏框架，用于多光谱影像，使学生模型的预训练目标与现代光学视觉基础模型(VFMs)中的对比自蒸馏范式相一致。我们的方法结合了多光谱和光学VFM两个老师，从而能够进行连贯的跨模态表示学习。在各种各样的光学和多光谱基准测试上进行的实验表明，我们的模型在适应多光谱数据的同时，并不会牺牲对仅光学输入的表现，在语义分割、变化检测以及分类任务中分别取得了平均3.64个百分点、1.2个百分点和1.31个百分点的进步。这证明了对比蒸馏为跨异构EO数据源的可扩展表示学习提供了一个原理上正确且有效的方法。 代码：即将发布。
URL
https://arxiv.org/abs/2602.19863
PDF
-
InfScene-SR: Spatially Continuous Inference for Arbitrary-Size Image Super-Resolution
2026-02-23 11:34:59Shoukun Sun, Zhe Wang, Xiang Que, Jiyin Zhang, Xiaogang MaAbstract
Image Super-Resolution (SR) aims to recover high-resolution (HR) details from low-resolution (LR) inputs, a task where Denoising Diffusion Probabilistic Models (DDPMs) have recently shown superior performance compared to Generative Adversarial Networks (GANs) based approaches. However, standard diffusion-based SR models, such as SR3, are typically trained on fixed-size patches and struggle to scale to arbitrary-sized images due to memory constraints. Applying these models via independent patch processing leads to visible seams and inconsistent textures across boundaries. In this paper, we propose InfScene-SR, a framework enabling spatially continuous super-resolution for large, arbitrary scenes. We adapt the iterative refinement process of diffusion models with a novel guided and variance-corrected fusion mechanism, allowing for the seamless generation of large-scale high-resolution imagery without retraining. We validate our approach on remote sensing datasets, demonstrating that InfScene-SR not only reconstructs fine details with high perceptual quality but also eliminates boundary artifacts, benefiting downstream tasks such as semantic segmentation.
Abstract (translated)
图像超分辨率（SR）旨在从低分辨率（LR）输入中恢复高分辨率（HR）细节，这一任务最近表明，去噪扩散概率模型（DDPMs）的性能优于基于生成对抗网络（GANs）的方法。然而，标准的基于扩散的超分辨率模型（如SR3）通常是在固定大小的图像块上进行训练的，在面对任意尺寸图像时会由于内存限制而难以扩展。通过独立处理这些图像块会导致可见的接缝和跨边界不一致的纹理问题。在本文中，我们提出了InfScene-SR框架，该框架能够实现大范围、任意场景的空间连续超分辨率生成。我们通过对扩散模型的迭代细化过程进行适应，并引入了一种新的引导式且方差校正融合机制，使得无需重新训练即可无缝地生成大规模高分辨率图像。我们在遥感数据集上验证了这种方法的有效性，证明InfScene-SR不仅能够以高度感知质量重建细节，还消除了边界伪影，从而有利于下游任务如语义分割。
URL
https://arxiv.org/abs/2602.19736
PDF
-
Segmenting Low-Contrast XCTs of Concretes: An Unsupervised Approach
2026-02-23 11:07:33Kaustav Das, Gaston Rauchs, Jan Sykora, Anna KucerovaAbstract
This work tests a self-annotation-based unsupervised methodology for training a convolutional neural network (CNN) model for semantic segmentation of X-ray computed tomography (XCT) scans of concretes. Concrete poses a unique challenge for XCT imaging due to similar X-ray attenuation coefficients of aggregates and mortar, resulting in low-contrast between the two phases in the ensuing images. While CNN-based models are a proven technique for semantic segmentation in such challenging cases, they typically require labeled training data, which is often unavailable for new datasets or are costly to obtain. To counter that limitation, a self-annotation technique is used here which leverages superpixel algorithms to identify perceptually similar local regions in an image and relates them to the global context in the image by utilizing the receptive field of a CNN-based model. This enables the model to learn a global-local relationship in the images and enables identification of semantically similar structures. We therefore present the performance of the unsupervised training methodology on our XCT datasets and discuss potential avenues for further improvements.
Abstract (translated)
这项工作测试了一种基于自我标注的无监督方法，用于训练卷积神经网络（CNN）模型对混凝土X射线计算机断层扫描（XCT）图像进行语义分割。混凝土为XCT成像带来了独特的挑战，因为骨料和砂浆具有相似的X射线衰减系数，在生成的图像中导致两个相之间的对比度较低。虽然基于CNN的模型已被证明在类似困难案例中的语义分割是有效的技术，但它们通常需要标注的训练数据，而这些数据对于新的数据集往往不可用或难以获取。为了克服这一限制，本文采用了一种自我标注技术，该技术利用超像素算法来识别图像中感知上相似的局部区域，并通过基于CNN模型的感受野将它们与图像中的全局上下文相关联。这使得模型能够学习图像中的全局-局部关系，并能够识别语义上类似的结构。因此，我们展示了在我们的XCT数据集上的无监督训练方法的表现，并讨论了进一步改进的可能性途径。
URL
https://arxiv.org/abs/2603.00127
PDF
-
Enhancing 3D LiDAR Segmentation by Shaping Dense and Accurate 2D Semantic Predictions
2026-02-21 15:33:41Xiaoyu Dong, Tiankui Xian, Wanshui Gan, Naoto YokoyaAbstract
Semantic segmentation of 3D LiDAR point clouds is important in urban remote sensing for understanding real-world street environments. This task, by projecting LiDAR point clouds and 3D semantic labels as sparse maps, can be reformulated as a 2D problem. However, the intrinsic sparsity of the projected LiDAR and label maps can result in sparse and inaccurate intermediate 2D semantic predictions, which in return limits the final 3D accuracy. To address this issue, we enhance this task by shaping dense and accurate 2D predictions. Specifically, we develop a multi-modal segmentation model, MM2D3D. By leveraging camera images as auxiliary data, we introduce cross-modal guided filtering to overcome label map sparsity by constraining intermediate 2D semantic predictions with dense semantic relations derived from the camera images; and we introduce dynamic cross pseudo supervision to overcome LiDAR map sparsity by encouraging the 2D predictions to emulate the dense distribution of the semantic predictions from the camera images. Experiments show that our techniques enable our model to achieve intermediate 2D semantic predictions with dense distribution and higher accuracy, which effectively enhances the final 3D accuracy. Comparisons with previous methods demonstrate our superior performance in both 2D and 3D spaces.
Abstract (translated)
三维激光雷达点云的语义分割在城市遥感中对于理解现实世界街道环境非常重要。通过将激光雷达点云和三维语义标签投影为稀疏地图，这一任务可以被重新表述为一个二维问题。然而，投影后的激光雷达图和标签图所固有的稀疏性会导致中间二维语义预测既稀疏又不准确，进而限制了最终的三维精度。 为了应对这个问题，我们通过塑造密集且准确的二维预测来增强此任务。具体而言，我们开发了一种多模态分割模型MM2D3D。利用摄像机图像作为辅助数据，我们引入了跨模态引导过滤器，以克服标签图稀疏性的问题，通过用从摄像机图像中衍生出的密集语义关系约束中间二维语义预测来实现这一目标；同时，我们还引入了动态交叉伪监督，通过鼓励二维预测模仿来自摄像机图像的语义预测的密集分布，从而解决激光雷达地图稀疏性问题。 实验表明，我们的技术使模型能够获得具有密集分布和更高准确性的中间二维语义预测，这有效地提高了最终的三维精度。与先前的方法相比，我们在二维和三维空间中的性能均表现出优越性。
URL
https://arxiv.org/abs/2602.18869
PDF
-
Open-Vocabulary Domain Generalization in Urban-Scene Segmentation
2026-02-21 14:32:27Dong Zhao, Qi Zang, Nan Pu, Wenjing Li, Nicu Sebe, Zhun ZhongAbstract
Domain Generalization in Semantic Segmentation (DG-SS) aims to enable segmentation models to perform robustly in unseen environments. However, conventional DG-SS methods are restricted to a fixed set of known categories, limiting their applicability in open-world scenarios. Recent progress in Vision-Language Models (VLMs) has advanced Open-Vocabulary Semantic Segmentation (OV-SS) by enabling models to recognize a broader range of concepts. Yet, these models remain sensitive to domain shifts and struggle to maintain robustness when deployed in unseen environments, a challenge that is particularly severe in urban-driving scenarios. To bridge this gap, we introduce Open-Vocabulary Domain Generalization in Semantic Segmentation (OVDG-SS), a new setting that jointly addresses unseen domains and unseen categories. We introduce the first benchmark for OVDG-SS in autonomous driving, addressing a previously unexplored problem and covering both synthetic-to-real and real-to-real generalization across diverse unseen domains and unseen categories. In OVDG-SS, we observe that domain shifts often distort text-image correlations in pre-trained VLMs, which hinders the performance of OV-SS models. To tackle this challenge, we propose S2-Corr, a state-space-driven text-image correlation refinement mechanism that mitigates domain-induced distortions and produces more consistent text-image correlations under distribution changes. Extensive experiments on our constructed benchmark demonstrate that the proposed method achieves superior cross-domain performance and efficiency compared to existing OV-SS approaches.
Abstract (translated)
领域泛化在语义分割（Domain Generalization in Semantic Segmentation，DG-SS）旨在使分割模型能够在未见过的环境中稳健地运行。然而，传统的DG-SS方法局限于一组已知类别，这限制了它们在开放世界场景中的应用能力。近期视觉语言模型（Vision-Language Models, VLMs）的进步推动了开放词汇语义分割（Open-Vocabulary Semantic Segmentation，OV-SS），使模型能够识别更广泛的概念范围。然而，这些模型仍然对领域偏移敏感，并且难以保持在未见过环境中部署的稳健性，在城市驾驶场景中这一问题尤为严重。 为弥合这一差距，我们提出了开放词汇领域泛化的语义分割（Open-Vocabulary Domain Generalization in Semantic Segmentation, OVDG-SS），这是一个同时处理未见领域和未见类别的新设定。我们在自主驾驶环境中引入了第一个针对OVDG-SS的基准测试，解决了之前未探索的问题，并涵盖了从合成到现实以及真实环境间多样化的未见过领域的泛化。 在OVDG-SS中，我们观察到域偏移常常扭曲预训练VLM中的文本图像相关性，这阻碍了OV-SS模型的表现。为解决这一挑战，我们提出了S2-Corr机制，这是一种基于状态空间的文本图像相关性改进方法，能够减轻领域引起的扭曲，并在分布变化下产生更一致的文本图像相关性。 在构建的基准上进行的大量实验表明，所提出的方法相比现有的OV-SS方法，在跨域性能和效率方面都表现出了显著的优势。
URL
https://arxiv.org/abs/2602.18853
PDF
