Semantic_Segmentation
-
Semantic Misalignment in Vision-Language Models under Perceptual Degradation
2026-01-13 09:13:05Guo ChengAbstract
Vision-Language Models (VLMs) are increasingly deployed in autonomous driving and embodied AI systems, where reliable perception is critical for safe semantic reasoning and decision-making. While recent VLMs demonstrate strong performance on multimodal benchmarks, their robustness to realistic perception degradation remains poorly understood. In this work, we systematically study semantic misalignment in VLMs under controlled degradation of upstream visual perception, using semantic segmentation on the Cityscapes dataset as a representative perception module. We introduce perception-realistic corruptions that induce only moderate drops in conventional segmentation metrics, yet observe severe failures in downstream VLM behavior, including hallucinated object mentions, omission of safety-critical entities, and inconsistent safety judgments. To quantify these effects, we propose a set of language-level misalignment metrics that capture hallucination, critical omission, and safety misinterpretation, and analyze their relationship with segmentation quality across multiple contrastive and generative VLMs. Our results reveal a clear disconnect between pixel-level robustness and multimodal semantic reliability, highlighting a critical limitation of current VLM-based systems and motivating the need for evaluation frameworks that explicitly account for perception uncertainty in safety-critical applications.
Abstract (translated)
视觉-语言模型（VLM）在自动驾驶和具身AI系统中越来越广泛地被部署，而在这些领域中可靠的感知对于安全的语义推理和决策至关重要。尽管最近的VLM在多模态基准测试中表现出强大的性能，但它们对现实世界中感知退化情况下的鲁棒性仍然知之甚少。在这项工作中，我们系统地研究了在上游视觉感知受控退化的条件下VLM中的语义错配问题，使用Cityscapes数据集上的语义分割作为典型的感知模块。我们引入了一组逼真的感知退化形式，这些退化仅导致传统分割指标有轻微下降，但观察到下游VLM行为出现严重故障，包括虚构对象提及、关键实体遗漏以及不一致的安全判断。为了量化这些效应，我们提出了一系列语言层面的错配度量标准，捕捉虚幻生成、关键信息遗漏和安全误解，并分析了它们与多个对比和生成型VLM之间分割质量的关系。我们的结果揭示了一个清晰的事实：像素级鲁棒性和多模态语义可靠性之间的明显脱节，突显出当前基于VLM系统的重大限制，并强调在安全性至关重要的应用中需要明确考虑感知不确定性评估框架的必要性。
URL
https://arxiv.org/abs/2601.08355
PDF
-
Human-inspired Global-to-Parallel Multi-scale Encoding for Lightweight Vision Models
2026-01-13 03:42:04Wei XuAbstract
Lightweight vision networks have witnessed remarkable progress in recent years, yet achieving a satisfactory balance among parameter scale, computational overhead, and task performance remains difficult. Although many existing lightweight models manage to reduce computation considerably, they often do so at the expense of a substantial increase in parameter count (e.g., LSNet, MobileMamba), which still poses obstacles for deployment on resource-limited devices. In parallel, some studies attempt to draw inspiration from human visual perception, but their modeling tends to oversimplify the visual process, making it hard to reflect how perception truly operates. Revisiting the cooperative mechanism of the human visual system, we propose GPM (Global-to-Parallel Multi-scale Encoding). GPM first employs a Global Insight Generator (GIG) to extract holistic cues, and subsequently processes features of different scales through parallel branches: LSAE emphasizes mid-/large-scale semantic relations, while IRB (Inverted Residual Block) preserves fine-grained texture information, jointly enabling coherent representation of global and local features. As such, GPM conforms to two characteristic behaviors of human vision perceiving the whole before focusing on details, and maintaining broad contextual awareness even during local attention. Built upon GPM, we further develop the lightweight H-GPE network. Experiments on image classification, object detection, and semantic segmentation show that H-GPE achieves strong performance while maintaining a balanced footprint in both FLOPs and parameters, delivering a more favorable accuracy-efficiency trade-off compared with recent state-of-the-art lightweight models.
Abstract (translated)
近年来，轻量级视觉网络取得了显著的进步，但要在参数规模、计算开销和任务性能之间取得满意的平衡仍然具有挑战性。虽然许多现有的轻量级模型能够大幅减少计算量，但这通常是以增加大量参数数量为代价的（例如LSNet、MobileMamba），这仍对资源受限设备的应用造成了障碍。与此同时，一些研究试图从人类视觉感知中汲取灵感，但其建模往往过于简化了视觉过程，难以真实反映感知机制。 重新审视人眼视觉系统的协同机制，我们提出了GPM（全局到并行多尺度编码）。GPM首先使用全局洞察生成器（Global Insight Generator, GIG）提取整体线索，并随后通过平行分支处理不同规模的特征：LSAE强调中/大规模语义关系，而IRB（倒残差块）则保留精细纹理信息。这种机制共同实现了对全球和局部特征的一致性表示。因此，GPM符合人类视觉感知先整体后细节、在局部注意时仍保持广泛上下文意识的两个特性。 基于GPM，我们进一步开发了轻量级H-GPE网络。实验结果表明，在图像分类、目标检测和语义分割任务中，H-GPE在网络性能强劲的同时，还能维持计算量（FLOPs）与参数数量之间的平衡，相较于近期的先进轻量模型而言，它在精度-效率权衡方面表现更佳。
URL
https://arxiv.org/abs/2601.08190
PDF
-
How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?
2026-01-13 01:53:20Peng Gao, Yujian Lee, Yongqi Xu, Wentao FanAbstract
Audio-visual semantic segmentation (AVSS) represents an extension of the audio-visual segmentation (AVS) task, necessitating a semantic understanding of audio-visual scenes beyond merely identifying sound-emitting objects at the visual pixel level. Contrary to a previous methodology, by decomposing the AVSS task into two discrete subtasks by initially providing a prompted segmentation mask to facilitate subsequent semantic analysis, our approach innovates on this foundational strategy. We introduce a novel collaborative framework, \textit{S}tepping \textit{S}tone \textit{P}lus (SSP), which integrates optical flow and textual prompts to assist the segmentation process. In scenarios where sound sources frequently coexist with moving objects, our pre-mask technique leverages optical flow to capture motion dynamics, providing essential temporal context for precise segmentation. To address the challenge posed by stationary sound-emitting objects, such as alarm clocks, SSP incorporates two specific textual prompts: one identifies the category of the sound-emitting object, and the other provides a broader description of the scene. Additionally, we implement a visual-textual alignment module (VTA) to facilitate cross-modal integration, delivering more coherent and contextually relevant semantic interpretations. Our training regimen involves a post-mask technique aimed at compelling the model to learn the diagram of the optical flow. Experimental results demonstrate that SSP outperforms existing AVS methods, delivering efficient and precise segmentation results.
Abstract (translated)
音频-视觉语义分割（AVSS）是音频-视觉分割（AVS）任务的扩展，它要求对音频-视觉场景进行语义理解，而不仅仅是识别发出声音的对象在视觉像素层面的位置。与先前的方法不同，我们通过将AVSS任务分解为两个独立的子任务，并首先提供一个提示性分段掩模以促进后续语义分析来创新这一基础策略。为此，我们引入了一个新颖的合作框架——Stepping Stone Plus（SSP），该框架集成了光流和文本提示以辅助分割过程。 在声源经常与移动对象共存的情况下，我们的预分段技术利用光流捕捉运动动态，为精确的分割提供必要的时间上下文。为了应对静止发声物体的挑战，如闹钟，SSP引入了两个特定的文本提示：一个用于识别发出声音的对象类别，另一个则提供了场景的更广泛描述。此外，我们实施了一个视觉-文本对齐模块（VTA），以促进跨模态整合，提供更为连贯且上下文相关的语义解释。 我们的训练方案包括一种后分段技术，旨在促使模型学习光流图的结构。实验结果表明，SSP优于现有的AVS方法，在效率和精度方面均表现出色。
URL
https://arxiv.org/abs/2601.08133
PDF
-
Exchange Is All You Need for Remote Sensing Change Detection
2026-01-12 18:36:51Sijun Dong, Siming Fu, Kaiyu Li, Xiangyong Cao, Xiaoliang Meng, Bo DuAbstract
Remote sensing change detection fundamentally relies on the effective fusion and discrimination of bi-temporal features. Prevailing paradigms typically utilize Siamese encoders bridged by explicit difference computation modules, such as subtraction or concatenation, to identify changes. In this work, we challenge this complexity with SEED (Siamese Encoder-Exchange-Decoder), a streamlined paradigm that replaces explicit differencing with parameter-free feature exchange. By sharing weights across both Siamese encoders and decoders, SEED effectively operates as a single parameter set model. Theoretically, we formalize feature exchange as an orthogonal permutation operator and prove that, under pixel consistency, this mechanism preserves mutual information and Bayes optimal risk, whereas common arithmetic fusion methods often introduce information loss. Extensive experiments across five benchmarks, including SYSU-CD, LEVIR-CD, PX-CLCD, WaterCD, and CDD, and three backbones, namely SwinT, EfficientNet, and ResNet, demonstrate that SEED matches or surpasses state of the art methods despite its simplicity. Furthermore, we reveal that standard semantic segmentation models can be transformed into competitive change detectors solely by inserting this exchange mechanism, referred to as SEG2CD. The proposed paradigm offers a robust, unified, and interpretable framework for change detection, demonstrating that simple feature exchange is sufficient for high performance information fusion. Code and full training and evaluation protocols will be released at this https URL.
Abstract (translated)
遥感变化检测主要依赖于双时相特征的有效融合和区分。现有的方法通常使用通过明确的差分计算模块（如减法或串联）连接的暹罗编码器来识别变化。在本文中，我们提出了一种名为SEED (Siamese Encoder-Exchange-Decoder) 的简化范式，该范式用无参数特征交换替代了这种复杂的差分操作，并通过共享两个暹罗编码器和解码器之间的权重，使其作为一个单一的参数集模型进行有效运作。理论上，我们将特征交换形式化为一个正交置换算子，并证明在像素一致性的前提下，这一机制能保持互信息和贝叶斯最优风险，而常见的算术融合方法往往会引入信息损失。 通过五个基准测试（包括SYSU-CD、LEVIR-CD、PX-CLCD、WaterCD以及CDD）和三种骨干网络模型（SwinT、EfficientNet 和 ResNet），广泛的实验表明，SEED 方法在简单性的同时能够与现有最佳方法相匹配或超越。此外，我们揭示了标准的语义分割模型仅通过插入这一交换机制即可被转换为竞争力强的变化检测器，这种方法被称为SEG2CD。 所提出的范式提供了一种稳健、统一且可解释性强的框架以实现变化检测，并表明简单的特征交换足以达到高水平的信息融合性能。代码和完整的训练与评估协议将在[此处](https://example.com)发布（注：实际链接请参考原文）。
URL
https://arxiv.org/abs/2601.07805
PDF
-
PanoSAMic: Panoramic Image Segmentation from SAM Feature Encoding and Dual View Fusion
2026-01-12 11:39:36Mahdi Chamseddine, Didier Stricker, Jason RambachAbstract
Existing image foundation models are not optimized for spherical images having been trained primarily on perspective images. PanoSAMic integrates the pre-trained Segment Anything (SAM) encoder to make use of its extensive training and integrate it into a semantic segmentation model for panoramic images using multiple modalities. We modify the SAM encoder to output multi-stage features and introduce a novel spatio-modal fusion module that allows the model to select the relevant modalities and best features from each modality for different areas of the input. Furthermore, our semantic decoder uses spherical attention and dual view fusion to overcome the distortions and edge discontinuity often associated with panoramic images. PanoSAMic achieves state-of-the-art (SotA) results on Stanford2D3DS for RGB, RGB-D, and RGB-D-N modalities and on Matterport3D for RGB and RGB-D modalities. this https URL
Abstract (translated)
现有的图像基础模型主要是基于透视图的图像进行训练，并未针对球形图像进行优化。PanoSAMic 将预先训练好的 Segment Anything (SAM) 编码器集成到语义分割全景图像的多模态模型中，以利用其广泛的预训练成果。我们对 SAM 编码器进行了修改，使其能够输出多阶段特征，并引入了一个新颖的空间-模式融合模块，使模型能够在不同输入区域选择相关模式和每个模式的最佳特征。 此外，我们的语义解码器使用球形注意力机制以及双视图融合来克服全景图像常有的扭曲和边缘不连续问题。PanoSAMic 在 Stanford2D3DS 数据集上针对 RGB、RGB-D 和 RGB-D-N 模态以及在 Matterport3D 数据集上的 RGB 和 RGB-D 模态实现了当前最佳（SotA）的结果。 原文链接：[这里插入 URL]
URL
https://arxiv.org/abs/2601.07447
PDF
-
Boosting Overlapping Organoid Instance Segmentation Using Pseudo-Label Unmixing and Synthesis-Assisted Learning
2026-01-10 17:51:09Gui Huang, Kangyuan Zheng, Xuan Cai, Jiaqi Wang, Jianjia Zhang, Kaida Ning, Wenbo Wei, Yujuan Zhu, Jiong Zhang, Mengting LiuAbstract
Organoids, sophisticated in vitro models of human tissues, are crucial for medical research due to their ability to simulate organ functions and assess drug responses accurately. Accurate organoid instance segmentation is critical for quantifying their dynamic behaviors, yet remains profoundly limited by high-quality annotated datasets and pervasive overlap in microscopy imaging. While semi-supervised learning (SSL) offers a solution to alleviate reliance on scarce labeled data, conventional SSL frameworks suffer from biases induced by noisy pseudo-labels, particularly in overlapping regions. Synthesis-assisted SSL (SA-SSL) has been proposed for mitigating training biases in semi-supervised semantic segmentation. We present the first adaptation of SA-SSL to organoid instance segmentation and reveal that SA-SSL struggles to disentangle intertwined organoids, often misrepresenting overlapping instances as a single entity. To overcome this, we propose Pseudo-Label Unmixing (PLU), which identifies erroneous pseudo-labels for overlapping instances and then regenerates organoid labels through instance decomposition. For image synthesis, we apply a contour-based approach to synthesize organoid instances efficiently, particularly for overlapping cases. Instance-level augmentations (IA) on pseudo-labels before image synthesis further enhances the effect of synthetic data (SD). Rigorous experiments on two organoid datasets demonstrate our method's effectiveness, achieving performance comparable to fully supervised models using only 10% labeled data, and state-of-the-art results. Ablation studies validate the contributions of PLU, contour-based synthesis, and augmentation-aware training. By addressing overlap at both pseudo-label and synthesis levels, our work advances scalable, label-efficient organoid analysis, unlocking new potential for high-throughput applications in precision medicine.
Abstract (translated)
类器官，即人类组织的复杂体外模型，在医学研究中至关重要，因为它们能够模拟器官功能并准确评估药物反应。精确的类器官实例分割对于量化其动态行为是至关重要的，但这一过程受到高质量标注数据集稀缺以及显微图像重叠现象的影响而受限。虽然半监督学习（SSL）可以作为缓解对稀有标签数据依赖的一种解决方案，但是传统的SSL框架因噪声伪标签所诱导的偏差问题，在重叠区域尤为明显。合成辅助的半监督学习（SA-SSL）被提出用于减轻半监督语义分割中的训练偏差。我们首次将SA-SSL应用于类器官实例分割，并揭示出SA-SSL在区分缠绕在一起的类器官时存在困难，往往将重叠实例误判为单一实体。 为了克服这个问题，我们提出了伪标签解混（PLU），这种方法能够识别重叠实例上的错误伪标签，并通过实例分解来再生类器官标签。对于图像合成，我们采用了一种基于轮廓的方法，以高效地生成类器官实例，特别是针对重叠情况特别有效。在进行图像合成之前，在伪标签上应用实例级别增强（IA）进一步增强了合成数据的效果。 我们在两个类器官数据集上的严格实验表明了该方法的有效性，使用仅10%的标注数据便能达到与全监督模型相媲美的性能，并且取得了最新的研究成果。消融研究表明PLU、基于轮廓的合成以及感知增益训练的贡献是有效的。通过在伪标签和合成水平上解决重叠问题，我们的工作推进了大规模高效标记类器官分析的发展，为精准医疗中的高通量应用解锁了新的潜力。
URL
https://arxiv.org/abs/2601.06642
PDF
-
Adapting Vision Transformers to Ultra-High Resolution Semantic Segmentation with Relay Tokens
2026-01-09 16:41:08Yohann Perron, Vladyslav Sydorov, Christophe Pottier, Loic LandrieuAbstract
Current approaches for segmenting ultra high resolution images either slide a window, thereby discarding global context, or downsample and lose fine detail. We propose a simple yet effective method that brings explicit multi scale reasoning to vision transformers, simultaneously preserving local details and global awareness. Concretely, we process each image in parallel at a local scale (high resolution, small crops) and a global scale (low resolution, large crops), and aggregate and propagate features between the two branches with a small set of learnable relay tokens. The design plugs directly into standard transformer backbones (eg ViT and Swin) and adds fewer than 2 % parameters. Extensive experiments on three ultra high resolution segmentation benchmarks, Archaeoscape, URUR, and Gleason, and on the conventional Cityscapes dataset show consistent gains, with up to 15 % relative mIoU improvement. Code and pretrained models are available at this https URL .
Abstract (translated)
目前用于分割超高分辨率图像的方法要么通过滑动窗口的方式忽略全局上下文，要么通过降采样而丢失细节信息。我们提出了一种简单且有效的方法，该方法将显式的多尺度推理引入视觉变压器中，在保留局部细节的同时保持全局感知能力。具体来说，我们在处理每张图像时同时使用局部尺度（高分辨率、小区域）和全局尺度（低分辨率、大区域），并通过一组可学习的中继令牌在两个分支之间聚合和传播特征。该设计可以直接集成到标准变压器骨干网络（如ViT和Swin）中，并且只增加了不到2%的参数量。我们在三个超高分辨率分割基准数据集Archaeoscape、URUR 和 Gleason以及传统的Cityscapes 数据集上进行了广泛的实验，结果显示了一致性的性能提升，在某些情况下，mIoU相对提高了高达15%。代码和预训练模型可在[此链接](https://this-https-url.com)获取。
URL
https://arxiv.org/abs/2601.05927
PDF
-
UniLiPs: Unified LiDAR Pseudo-Labeling with Geometry-Grounded Dynamic Scene Decomposition
2026-01-08 16:52:28Filippo Ghilotti, Samuel Brucker, Nahku Saidy, Matteo Matteucci, Mario Bijelic, Felix HeideAbstract
Unlabeled LiDAR logs, in autonomous driving applications, are inherently a gold mine of dense 3D geometry hiding in plain sight - yet they are almost useless without human labels, highlighting a dominant cost barrier for autonomous-perception research. In this work we tackle this bottleneck by leveraging temporal-geometric consistency across LiDAR sweeps to lift and fuse cues from text and 2D vision foundation models directly into 3D, without any manual input. We introduce an unsupervised multi-modal pseudo-labeling method relying on strong geometric priors learned from temporally accumulated LiDAR maps, alongside with a novel iterative update rule that enforces joint geometric-semantic consistency, and vice-versa detecting moving objects from inconsistencies. Our method simultaneously produces 3D semantic labels, 3D bounding boxes, and dense LiDAR scans, demonstrating robust generalization across three datasets. We experimentally validate that our method compares favorably to existing semantic segmentation and object detection pseudo-labeling methods, which often require additional manual supervision. We confirm that even a small fraction of our geometrically consistent, densified LiDAR improves depth prediction by 51.5% and 22.0% MAE in the 80-150 and 150-250 meters range, respectively.
Abstract (translated)
在自动驾驶应用中，未标注的LiDAR日志本质上隐藏着丰富的3D几何信息——然而，在没有人工标签的情况下，这些数据几乎无法使用，这突出显示了自主感知研究中的主要成本障碍。在这项工作中，我们通过利用跨时间的几何一致性来解决这一瓶颈问题，直接将文本和2D视觉基础模型中的线索提升并融合到3D空间中，无需任何手动输入。我们介绍了一种无监督的多模态伪标签生成方法，该方法依赖于从累积的时间LiDAR地图中学得的强大几何先验，并且引入了一个新颖的迭代更新规则，以强制执行几何语义的一致性，反之亦可检测移动物体中的不一致之处。 我们的方法同时产生3D语义标签、3D边界框以及密集的LiDAR扫描数据，在三个不同的数据集上展示了稳健的泛化能力。我们通过实验验证了该方法与现有的语义分割和目标检测伪标签生成方法相比具有优势，而这些现有方法往往需要额外的手动监督。 我们确认，即使少量我们的几何一致、密度增强的LiDAR数据也能使深度预测在80-150米范围内的MAE误差降低51.5%，以及在150-250米范围内的MAE误差降低22%。
URL
https://arxiv.org/abs/2601.05105
PDF
-
When Imbalance Comes Twice: Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation
2026-01-08 14:14:06Julien Combes (SVH), Alexandre Derville (Michelin), Jean-Fran\c{c}ois Coeurjolly (SVH)Abstract
The aim of Active Learning is to select the most informative samples from an unlabelled set of data. This is useful in cases where the amount of data is large and labelling is expensive, such as in machine vision or medical imaging. Two particularities of machine vision are first, that most of the images produced are free of defects, and second, that the amount of images produced is so big that we cannot store all acquired images. This results, on the one hand, in a strong class imbalance in defect distribution and, on the other hand, in a potential label shift caused by limited storage. To understand how these two forms of imbalance affect active learning algorithms, we propose a simulation study based on two open-source datasets. We artificially create datasets for which we control the levels of class imbalance and label shift. Three standard active learning selection strategies are compared: random sampling, entropy-based selection, and core-set selection. We demonstrate that active learning strategies, and in particular the entropy-based and core-set selections, remain interesting and efficient even for highly imbalanced datasets. We also illustrate and measure the loss of efficiency that occurs in the situation a strong label shift.
Abstract (translated)
主动学习的目标是从未标记的数据集中挑选出最具信息量的样本。这在数据量大且标注成本高的情况下非常有用，比如机器视觉或医学影像领域。机器视觉有两个特点：一是生成的大部分图像没有缺陷；二是产生的图像数量庞大到无法全部存储。这两点导致了两个问题：一方面，缺陷分布呈现出强烈的类别不平衡；另一方面，由于有限的存储空间，可能会发生标签偏移。 为了理解这两种不平衡如何影响主动学习算法，我们基于两个开源数据集进行了一项模拟研究，并人为创建了一些具有不同类不平衡和标签偏移级别的数据集。我们比较了三种标准的主动学习选择策略：随机抽样、熵基选择以及核心集选择。研究表明，即使在极端类别不平衡的数据集中，主动学习策略（特别是基于熵的选择和核心集选择）仍然有效且高效。同时，我们也展示了并量化了强标签偏移情况下效率降低的情况。
URL
https://arxiv.org/abs/2601.06209
PDF
-
TEA: Temporal Adaptive Satellite Image Semantic Segmentation
2026-01-08 14:02:28Juyuan Kang, Hao Zhu, Yan Zhu, Wei Zhang, Jianing Chen, Tianxiang Xiao, Yike Ma, Hao Jiang, Feng DaiAbstract
Crop mapping based on satellite images time-series (SITS) holds substantial economic value in agricultural production settings, in which parcel segmentation is an essential step. Existing approaches have achieved notable advancements in SITS segmentation with predetermined sequence lengths. However, we found that these approaches overlooked the generalization capability of models across scenarios with varying temporal length, leading to markedly poor segmentation results in such cases. To address this issue, we propose TEA, a TEmporal Adaptive SITS semantic segmentation method to enhance the model's resilience under varying sequence lengths. We introduce a teacher model that encapsulates the global sequence knowledge to guide a student model with adaptive temporal input lengths. Specifically, teacher shapes the student's feature space via intermediate embedding, prototypes and soft label perspectives to realize knowledge transfer, while dynamically aggregating student model to mitigate knowledge forgetting. Finally, we introduce full-sequence reconstruction as an auxiliary task to further enhance the quality of representations across inputs of varying temporal lengths. Through extensive experiments, we demonstrate that our method brings remarkable improvements across inputs of different temporal lengths on common benchmarks. Our code will be publicly available.
Abstract (translated)
基于卫星图像时间序列（SITS）的作物地图绘制在农业生产中具有重要的经济价值，其中地块分割是关键步骤之一。现有方法已经在预设的时间序列长度上实现了显著的进展。然而，我们发现这些方法忽略了模型在不同时间长度场景下的泛化能力，导致此类情况下的分割效果明显较差。为解决这一问题，我们提出了TEA（Temporal Adaptive SITS Semantic Segmentation），这是一种增强型时间适应性SITS语义分割方法，旨在提高模型在变化序列长度下工作的韧性。 我们的方法引入了一个教师模型，该模型封装了整个序列的知识，来指导一个可以自适应调整时间输入长度的学生模型。具体来说，通过中间嵌入、原型和软标签视角，教师模型塑造学生模型的特征空间以实现知识转移，并动态聚合学生模型以减轻知识遗忘问题。 最后，我们引入全序列重构作为辅助任务，进一步提升不同时间长度输入下的表示质量。通过广泛的实验，我们在常用基准数据集上展示了我们的方法在不同时间长度输入下取得了显著改进的效果。我们的代码将公开发布。
URL
https://arxiv.org/abs/2601.04956
PDF
-
Physics-Constrained Cross-Resolution Enhancement Network for Optics-Guided Thermal UAV Image Super-Resolution
2026-01-07 02:30:27Zhicheng Zhao, Fengjiao Peng, Jinquan Yan, Wei Lu, Chenglong Li, Jin TangAbstract
Optics-guided thermal UAV image super-resolution has attracted significant research interest due to its potential in all-weather monitoring applications. However, existing methods typically compress optical features to match thermal feature dimensions for cross-modal alignment and fusion, which not only causes the loss of high-frequency information that is beneficial for thermal super-resolution, but also introduces physically inconsistent artifacts such as texture distortions and edge blurring by overlooking differences in the imaging physics between modalities. To address these challenges, we propose PCNet to achieve cross-resolution mutual enhancement between optical and thermal modalities, while physically constraining the optical guidance process via thermal conduction to enable robust thermal UAV image super-resolution. In particular, we design a Cross-Resolution Mutual Enhancement Module (CRME) to jointly optimize thermal image super-resolution and optical-to-thermal modality conversion, facilitating effective bidirectional feature interaction across resolutions while preserving high-frequency optical priors. Moreover, we propose a Physics-Driven Thermal Conduction Module (PDTM) that incorporates two-dimensional heat conduction into optical guidance, modeling spatially-varying heat conduction properties to prevent inconsistent artifacts. In addition, we introduce a temperature consistency loss that enforces regional distribution consistency and boundary gradient smoothness to ensure generated thermal images align with real-world thermal radiation principles. Extensive experiments on VGTSR2.0 and DroneVehicle datasets demonstrate that PCNet significantly outperforms state-of-the-art methods on both reconstruction quality and downstream tasks including semantic segmentation and object detection.
Abstract (translated)
基于光学引导的热气球无人机图像超分辨率技术因其在全天候监测应用中的潜力而引起了广泛的研究兴趣。然而，现有的方法通常通过压缩光学特征以匹配热特征维度来进行跨模态对齐和融合，这不仅导致了有益于热超分辨率的高频信息丢失，而且还因忽视了不同成像物理之间的差异而导致诸如纹理扭曲和边缘模糊等物理不一致的现象。 为了解决这些挑战，我们提出了PCNet（Physics-Driven Cross-modal Network），它在光学与热模态之间实现了跨分辨率相互增强，并通过基于热传导的物理约束来强化光学引导过程，从而能够实现鲁棒性的热气球无人机图像超分辨率。具体来说，我们设计了跨分辨率互增强模块（CRME），以共同优化热图的超分辨率和从光学到热的模态转换，促进不同分辨率间的有效双向特征交互，并保持高频光学先验信息。 此外，我们还提出了物理驱动的热传导模块（PDTM），该模块将二维热传导融入光学引导过程中，通过建模空间变化的热传导特性来防止不一致的现象。同时，我们引入了温度一致性损失，确保生成的热图在区域分布和边界梯度平滑性方面与现实世界的热辐射原理保持一致。 我们在VGTSR2.0和DroneVehicle数据集上的广泛实验表明，PCNet在重建质量和包括语义分割和目标检测在内的下游任务上都显著优于现有最先进的方法。
URL
https://arxiv.org/abs/2601.03526
PDF
-
A Vision-Language-Action Model with Visual Prompt for OFF-Road Autonomous Driving
2026-01-07 02:08:18Liangdong Zhang, Yiming Nie, Haoyang Li, Fanjie Kong, Baobao Zhang, Shunxin Huang, Kai Fu, Chen Min, Liang XiaoAbstract
Efficient trajectory planning in off-road terrains presents a formidable challenge for autonomous vehicles, often necessitating complex multi-step pipelines. However, traditional approaches exhibit limited adaptability in dynamic environments. To address these limitations, this paper proposes OFF-EMMA, a novel end-to-end multimodal framework designed to overcome the deficiencies of insufficient spatial perception and unstable reasoning in visual-language-action (VLA) models for off-road autonomous driving scenarios. The framework explicitly annotates input images through the design of a visual prompt block and introduces a chain-of-thought with self-consistency (COT-SC) reasoning strategy to enhance the accuracy and robustness of trajectory planning. The visual prompt block utilizes semantic segmentation masks as visual prompts, enhancing the spatial understanding ability of pre-trained visual-language models for complex terrains. The COT- SC strategy effectively mitigates the error impact of outliers on planning performance through a multi-path reasoning mechanism. Experimental results on the RELLIS-3D off-road dataset demonstrate that OFF-EMMA significantly outperforms existing methods, reducing the average L2 error of the Qwen backbone model by 13.3% and decreasing the failure rate from 16.52% to 6.56%.
Abstract (translated)
在越野地形中为自主车辆进行高效的路径规划是一项艰巨的挑战，通常需要复杂的多步骤流程。然而，传统的解决方案在动态环境中表现出较低的适应性。为了应对这些限制，本文提出了一种新型端到端多模态框架OFF-EMMA，旨在克服视觉-语言-行动（VLA）模型在越野自主驾驶场景中空间感知不足和推理不稳定的缺点。该框架通过设计视觉提示模块来显式标注输入图像，并引入了链式思维与自我一致性(COT-SC)推理策略以提高路径规划的准确性和鲁棒性。视觉提示模块利用语义分割掩码作为视觉提示，增强了预训练模型在复杂地形中的空间理解能力。COT-SC策略通过多路径推理机制有效地减轻了异常值对规划性能的影响。 在RELLIS-3D越野数据集上的实验结果表明，OFF-EMMA显著优于现有方法，在降低Qwen骨干网络的平均L2误差方面表现出13.3%的改进，并将失败率从16.52%降至6.56%。
URL
https://arxiv.org/abs/2601.03519
PDF
-
G2P: Gaussian-to-Point Attribute Alignment for Boundary-Aware 3D Semantic Segmentation
2026-01-07 01:44:29Hojun Song, Chae-yeong Song, Jeong-hun Hong, Chaewon Moon, Dong-hwi Kim, Gahyeon Kim, Soo Ye Kim, Yiyi Liao, Jaehyup Lee, Sang-hyo ParkAbstract
Semantic segmentation on point clouds is critical for 3D scene understanding. However, sparse and irregular point distributions provide limited appearance evidence, making geometry-only features insufficient to distinguish objects with similar shapes but distinct appearances (e.g., color, texture, material). We propose Gaussian-to-Point (G2P), which transfers appearance-aware attributes from 3D Gaussian Splatting to point clouds for more discriminative and appearance-consistent segmentation. Our G2P address the misalignment between optimized Gaussians and original point geometry by establishing point-wise correspondences. By leveraging Gaussian opacity attributes, we resolve the geometric ambiguity that limits existing models. Additionally, Gaussian scale attributes enable precise boundary localization in complex 3D scenes. Extensive experiments demonstrate that our approach achieves superior performance on standard benchmarks and shows significant improvements on geometrically challenging classes, all without any 2D or language supervision.
Abstract (translated)
点云的语义分割对于三维场景理解至关重要。然而，稀疏且不规则的点分布提供的外观证据有限，仅依靠几何特征不足以区分形状相似但外观不同（如颜色、纹理、材质）的对象。为此，我们提出了Gaussian-to-Point (G2P) 方法，该方法将基于3D高斯散布的感知属性转移到点云中，以实现更具判别力和外观一致性的分割。 我们的G2P通过建立逐点对应关系解决了优化后的高斯分布与原始点几何之间的对齐问题。利用高斯不透明度属性，我们解决了限制现有模型的几何模糊性。此外，高斯尺度属性使得在复杂的三维场景中能够精确定位边界。 大量的实验表明，我们的方法在标准基准测试中的表现优于其他方法，并且对于几何结构复杂度较高的类别也有显著改进，而无需任何二维或语言监督。
URL
https://arxiv.org/abs/2601.03510
PDF
-
EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework
2026-01-06 07:41:44Junjue Wang, Yanfei Zhong, Zihang Chen, Zhuo Zheng, Ailong Ma, Liangpei ZhangAbstract
Earth vision has achieved milestones in geospatial object recognition but lacks exploration in object-relational reasoning, limiting comprehensive scene understanding. To address this, a progressive Earth vision-language understanding and generation framework is proposed, including a multi-task dataset (EarthVLSet) and a semantic-guided network (EarthVLNet). Focusing on city planning applications, EarthVLSet includes 10.9k sub-meter resolution remote sensing images, land-cover masks, and 761.5k textual pairs involving both multiple-choice and open-ended visual question answering (VQA) tasks. In an object-centric way, EarthVLNet is proposed to progressively achieve semantic segmentation, relational reasoning, and comprehensive understanding. The first stage involves land-cover segmentation to generate object semantics for VQA guidance. Guided by pixel-wise semantics, the object awareness based large language model (LLM) performs relational reasoning and knowledge summarization to generate the required answers. As for optimization, the numerical difference loss is proposed to dynamically add difference penalties, addressing the various objects' statistics. Three benchmarks, including semantic segmentation, multiple-choice, and open-ended VQA demonstrated the superiorities of EarthVLNet, yielding three future directions: 1) segmentation features consistently enhance VQA performance even in cross-dataset scenarios; 2) multiple-choice tasks show greater sensitivity to the vision encoder than to the language decoder; and 3) open-ended tasks necessitate advanced vision encoders and language decoders for an optimal performance. We believe this dataset and method will provide a beneficial benchmark that connects ''image-mask-text'', advancing geographical applications for Earth vision.
Abstract (translated)
地球视觉在地理空间对象识别方面取得了里程碑式的成就，但在物体间关系推理方面的探索不足，这限制了对场景的全面理解。为了解决这一问题，提出了一种渐进式地球视觉-语言理解和生成框架，包括一个多任务数据集（EarthVLSet）和一个语义引导网络（EarthVLNet）。该框架特别关注城市规划应用领域，EarthVLSet包含了10.9k张亚米级分辨率的遥感图像、土地覆盖掩模以及761.5万个涉及多项选择题和开放式视觉问答(VQA)任务的文本对。以对象为中心的方式，EarthVLNet旨在逐步实现语义分割、关系推理和综合理解。第一阶段是进行土地覆盖分割，生成用于VQA引导的对象语义信息。通过像素级语义指导，基于大语言模型（LLM）的对象意识执行关系推理和知识总结，从而生成所需答案。为了优化性能，提出了数值差异损失函数，动态添加差异惩罚以应对各种对象的统计特性。 在三个基准测试中——包括语义分割、多项选择题以及开放式VQA任务，EarthVLNet表现出了优越性，并指出了未来研究的三大方向：1）段落特征能够持续提升VQA性能，即使是在跨数据集场景下；2）多项选择任务对视觉编码器比语言解码器更为敏感；3）开放式任务需要更先进的视觉和语言模型以实现最佳效果。我们相信这一数据集和方法将为连接“图像-掩模-文本”的研究提供一个有益的基准，从而促进地球视觉在地理应用领域的进一步发展。
URL
https://arxiv.org/abs/2601.02783
PDF
-
M-SEVIQ: A Multi-band Stereo Event Visual-Inertial Quadruped-based Dataset for Perception under Rapid Motion and Challenging Illumination
2026-01-06 07:23:34Jingcheng Cao, Chaoran Xiong, Jianmin Song, Shang Yan, Jiachen Liu, Ling PeiAbstract
Agile locomotion in legged robots poses significant challenges for visual perception. Traditional frame-based cameras often fail in these scenarios for producing blurred images, particularly under low-light conditions. In contrast, event cameras capture changes in brightness asynchronously, offering low latency, high temporal resolution, and high dynamic range. These advantages make them suitable for robust perception during rapid motion and under challenging illumination. However, existing event camera datasets exhibit limitations in stereo configurations and multi-band sensing domains under various illumination conditions. To address this gap, we present M-SEVIQ, a multi-band stereo event visual and inertial quadruped dataset collected using a Unitree Go2 equipped with stereo event cameras, a frame-based camera, an inertial measurement unit (IMU), and joint encoders. This dataset contains more than 30 real-world sequences captured across different velocity levels, illumination wavelengths, and lighting conditions. In addition, comprehensive calibration data, including intrinsic, extrinsic, and temporal alignments, are provided to facilitate accurate sensor fusion and benchmarking. Our M-SEVIQ can be used to support research in agile robot perception, sensor fusion, semantic segmentation and multi-modal vision in challenging environments.
Abstract (translated)
敏捷腿足机器人在视觉感知方面面临重大挑战。传统的基于帧的摄像头在这种情况下往往表现不佳，特别是在低光条件下会拍摄到模糊图像。相比之下，事件相机以异步方式捕捉亮度变化，提供低延迟、高时间分辨率和宽动态范围的优势。这些优势使它们非常适合快速移动时以及各种照明条件下的稳健感知。然而，现有的事件相机数据集在立体配置和多频带感应领域存在局限性，尤其是在不同的光照条件下。 为解决这一缺口，我们提出了M-SEVIQ，这是一个使用配备立体事件相机、基于帧的摄像头、惯性测量单元（IMU）和关节编码器的Unitree Go2收集的多频带立体事件视觉与惯性四足机器人数据集。该数据集包含超过30个在不同速度等级、光照波长以及照明条件下的真实世界序列。此外，还提供了全面的校准数据，包括固有参数、外在参数和时间对齐信息，以支持精确的传感器融合和基准测试。我们的M-SEVIQ可以用于研究敏捷机器人感知、传感器融合、语义分割及多模态视觉在具有挑战性环境中的应用。
URL
https://arxiv.org/abs/2601.02777
PDF
-
Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding
2026-01-05 18:33:50Jingming He, Chongyi Li, Shiqi Wang, Sam KwongAbstract
Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.
Abstract (translated)
最近的研究提出将3DGS（三维高斯模型）扩展为使用语义特征向量，以同时进行语义分割和图像渲染。然而，这些方法通常分别处理语义分支和渲染分支，并且主要依赖于2D监督信息，而忽略了3D高斯几何结构。此外，现有的自适应策略仅根据渲染梯度调整高斯集，在细节较少或无纹理的区域中可能效果不足。 在本工作中，我们提出了一种联合增强框架用于3D语义高斯建模，该框架将语义分支和渲染分支协同起来共同工作。首先，不同于传统的点云形状编码方法，我们引入了一个各向异性三维高斯切比雪夫描述符（使用拉普拉斯-贝尔特拉米算子），以捕捉精细的3D形状细节，并区分具有类似外观的对象，同时减少了对潜在噪声较多的2D引导信息的依赖。此外，在不完全依赖于渲染梯度的情况下，我们根据局部语义和形状信号自适应调整高斯分配以及球谐函数，通过选择性资源分配提升渲染效率。 最后，我们采用跨场景知识转移模块来持续更新已学习到的形状模式，这使得在面对新场景时无需从零开始重新学习形状信息即可实现更快收敛和稳健表示。多项数据集上的实验表明，在保持高渲染帧率的同时，我们的方法可以提高分割准确性和渲染质量。
URL
https://arxiv.org/abs/2601.02339
PDF
-
Prithvi-Complimentary Adaptive Fusion Encoder : unlocking full-potential for flood inundation mapping
2026-01-05 18:07:21Saurabh Kaushik, Lalit Maurya, Beth TellmanAbstract
Geo-Foundation Models (GFMs), have proven effective in diverse downstream applications, including semantic segmentation, classification, and regression tasks. However, in case of flood mapping using Sen1Flood11 dataset as a downstream task, GFMs struggles to outperform the baseline U-Net, highlighting model's limitation in capturing critical local nuances. To address this, we present the Prithvi-Complementary Adaptive Fusion Encoder (CAFE), which integrate Prithvi GFM pretrained encoder with a parallel CNN residual branch enhanced by Convolutional Attention Modules (CAM). Prithvi-CAFE enables fast and efficient fine-tuning through adapters in Prithvi and performs multi-scale, multi-level fusion with CNN features, capturing critical local details while preserving long-range dependencies. We achieve state-of-the-art results on two comprehensive flood mapping datasets: Sen1Flood11 and FloodPlanet. On Sen1Flood11 test data, Prithvi-CAFE (IoU 83.41) outperforms the original Prithvi (IoU 82.50) and other major GFMs (TerraMind 82.90, DOFA 81.54, spectralGPT: 81.02). The improvement is even more pronounced on the hold-out test site, where Prithvi-CAFE achieves an IoU of 81.37 compared to the baseline U-Net (70.57) and original Prithvi (72.42). On FloodPlanet, Prithvi-CAFE also surpasses the baseline U-Net and other GFMs, achieving an IoU of 64.70 compared to U-Net (60.14), Terramind (62.33), DOFA (59.15) and Prithvi 2.0 (61.91). Our proposed simple yet effective Prithvi-CAFE demonstrates strong potential for improving segmentation tasks where multi-channel and multi-modal data provide complementary information and local details are critical. The code is released on \href{this https URL}{Prithvi-CAFE Github}
Abstract (translated)
地理基础模型（GFMs）在语义分割、分类和回归任务等不同下游应用中已被证明是有效的。然而，当使用Sen1Flood11数据集进行洪水制图这一特定任务时，GFMs的表现不如基线U-Net，这表明了模型在捕捉关键局部细节方面存在局限性。 为了解决这个问题，我们提出了Prithvi-互补自适应融合编码器（CAFE），它将预训练的Prithvi GFM编码器与增强有卷积注意模块（CAM）的平行CNN残差分支集成在一起。Prithvi-CAFE通过在Prithvi中使用适配器进行快速高效的微调，并且能够利用CNN特征实现多尺度、多层次融合，从而捕捉到关键局部细节同时保持长程依赖性。 我们在两个全面的洪水制图数据集（Sen1Flood11和FloodPlanet）上实现了最先进的成果。在Sen1Flood11测试数据集中，Prithvi-CAFE (IoU 83.41) 超过了原始Prithvi模型 (IoU 82.50)，以及其他的GFMs（TerraMind 82.90, DOFA 81.54, spectralGPT: 81.02）。这种改进在独立的测试站点上更为明显，其中Prithvi-CAFE实现了81.37的IoU值，而基线U-Net为70.57，原始Prithvi模型为72.42。在FloodPlanet数据集上，Prithvi-CAFE也优于基线U-Net和其他GFMs，在IoU指标上达到了64.70，相比之下，U-Net为60.14、TerraMind为62.33、DOFA为59.15和Prithvi 2.0为61.91。 我们提出的简单而有效的Prithvi-CAFE模型显示了在多通道及多模态数据提供互补信息且局部细节至关重要的分割任务中提升性能的巨大潜力。相关代码已发布至[Prithvi-CAFE Github](https://github.com/your-repo-url-here)（请将URL替换为实际的GitHub地址）。
URL
https://arxiv.org/abs/2601.02315
PDF
-
TopoLoRA-SAM: Topology-Aware Parameter-Efficient Adaptation of Foundation Segmenters for Thin-Structure and Cross-Domain Binary Semantic Segmentation
2026-01-05 17:03:45Salim KhazemAbstract
Foundation segmentation models such as the Segment Anything Model (SAM) exhibit strong zero-shot generalization through large-scale pretraining, but adapting them to domain-specific semantic segmentation remains challenging, particularly for thin structures (e.g., retinal vessels) and noisy modalities (e.g., SAR imagery). Full fine-tuning is computationally expensive and risks catastrophic forgetting. We propose \textbf{TopoLoRA-SAM}, a topology-aware and parameter-efficient adaptation framework for binary semantic segmentation. TopoLoRA-SAM injects Low-Rank Adaptation (LoRA) into the frozen ViT encoder, augmented with a lightweight spatial convolutional adapter and optional topology-aware supervision via differentiable clDice. We evaluate our approach on five benchmarks spanning retinal vessel segmentation (DRIVE, STARE, CHASE\_DB1), polyp segmentation (Kvasir-SEG), and SAR sea/land segmentation (SL-SSDD), comparing against U-Net, DeepLabV3+, SegFormer, and Mask2Former. TopoLoRA-SAM achieves the best retina-average Dice and the best overall average Dice across datasets, while training only \textbf{5.2\%} of model parameters ($\sim$4.9M). On the challenging CHASE\_DB1 dataset, our method substantially improves segmentation accuracy and robustness, demonstrating that topology-aware parameter-efficient adaptation can match or exceed fully fine-tuned specialist models. Code is available at : this https URL
Abstract (translated)
基于大范围预训练的分割模型（如Segment Anything Model，SAM）展现出强大的零样本泛化能力。然而，将这些基础模型调整为特定领域的语义分割任务仍然具有挑战性，尤其是在处理细长结构（例如视网膜血管）和噪声模式（例如SAR图像）时更为困难。全量微调在计算上是昂贵的，并且存在灾难性遗忘的风险。 我们提出了一种基于拓扑感知并参数高效的适应框架——TopoLoRA-SAM，用于二值语义分割任务。该方法向冻结的ViT编码器中注入低秩适配（Low-Rank Adaptation, LoRA），结合了一个轻量级的空间卷积适配器，并可选地通过不同的clDice进行拓扑感知监督。 我们在五个基准数据集上进行了评估，这些数据集涵盖了视网膜血管分割（DRIVE、STARE和CHASE_DB1）、息肉分割（Kvasir-SEG）以及SAR海陆分割（SL-SSDD），并与U-Net、DeepLabV3+、SegFormer及Mask2Former等方法进行了比较。TopoLoRA-SAM在所有数据集上获得了最高的视网膜平均Dice值和总体平均Dice值，同时仅训练了模型参数的5.2%（约490万）。 特别是在具有挑战性的CHASE_DB1数据集上，我们的方法显著提高了分割准确性和鲁棒性，证明了拓扑感知且参数高效的适应可以匹配或超过全量微调的专业模型。代码可在提供的链接处获取：this https URL
URL
https://arxiv.org/abs/2601.02273
PDF
-
Leveraging 2D-VLM for Label-Free 3D Segmentation in Large-Scale Outdoor Scene Understanding
2026-01-05 11:42:49Toshihiko Nishimura, Hirofumi Abe, Kazuhiko Murasaki, Taiga Yoshida, Ryuichi TanidaAbstract
This paper presents a novel 3D semantic segmentation method for large-scale point cloud data that does not require annotated 3D training data or paired RGB images. The proposed approach projects 3D point clouds onto 2D images using virtual cameras and performs semantic segmentation via a foundation 2D model guided by natural language prompts. 3D segmentation is achieved by aggregating predictions from multiple viewpoints through weighted voting. Our method outperforms existing training-free approaches and achieves segmentation accuracy comparable to supervised methods. Moreover, it supports open-vocabulary recognition, enabling users to detect objects using arbitrary text queries, thus overcoming the limitations of traditional supervised approaches.
Abstract (translated)
本文提出了一种新颖的三维语义分割方法，专门用于大规模点云数据，并且该方法不需要标注的3D训练数据或配对的RGB图像。所提出的方案通过虚拟相机将3D点云投影到2D图像上，并使用基础2D模型和自然语言提示进行语义分割。3D分割是通过从多个视角预测并通过加权投票聚合来实现的。我们的方法超越了现有的无训练需求的方法，实现了与监督学习方法相当的分割精度。此外，该方法支持开放式词汇识别，允许用户利用任意文本查询检测对象，从而克服了传统监督方法的局限性。
URL
https://arxiv.org/abs/2601.02029
PDF
-
Subimage Overlap Prediction: Task-Aligned Self-Supervised Pretraining For Semantic Segmentation In Remote Sensing Imagery
2026-01-05 04:28:49Lakshay Sharma, Alex MarinAbstract
Self-supervised learning (SSL) methods have become a dominant paradigm for creating general purpose models whose capabilities can be transferred to downstream supervised learning tasks. However, most such methods rely on vast amounts of pretraining data. This work introduces Subimage Overlap Prediction, a novel self-supervised pretraining task to aid semantic segmentation in remote sensing imagery that uses significantly lesser pretraining imagery. Given an image, a sub-image is extracted and the model is trained to produce a semantic mask of the location of the extracted sub-image within the original image. We demonstrate that pretraining with this task results in significantly faster convergence, and equal or better performance (measured via mIoU) on downstream segmentation. This gap in convergence and performance widens when labeled training data is reduced. We show this across multiple architecture types, and with multiple downstream datasets. We also show that our method matches or exceeds performance while requiring significantly lesser pretraining data relative to other SSL methods. Code and model weights are provided at \href{this https URL}{this http URL}.
Abstract (translated)
自监督学习（SSL）方法已成为创建通用模型的主导范式，这些模型的能力可以转移到下游的监督学习任务中。然而，大多数这样的方法依赖于大量的预训练数据。本工作引入了一种新颖的自监督预训练任务——子图像重叠预测（Subimage Overlap Prediction），旨在帮助遥感图像中的语义分割，并且使用显著较少的预训练图像。给定一张图片后，从中抽取一个子图像，模型被训练以生成表示该子图在原图中位置的语义掩码。 我们证明了利用此任务进行预训练可以实现更快的收敛速度，并且在下游分割任务上的表现（通过mIoU衡量）至少与现有方法持平或更好。当减少标注训练数据时，这种差距会进一步扩大。我们在多种架构类型和多个下游数据集上展示了这一点。此外，我们的方法在需要显著较少预训练数据的情况下仍能达到匹配甚至超过其他SSL方法的性能。 代码和模型权重可在[此处](this https URL)获取。
URL
https://arxiv.org/abs/2601.01781
PDF
