Semantic_Segmentation
-
ViT$^3$: Unlocking Test-Time Training in Vision
2025-12-01 13:14:48Dongchen Han, Yining Li, Tianyu Li, Zixuan Cao, Ziming Wang, Jun Song, Yu Cheng, Bo Zheng, Gao HuangAbstract
Test-Time Training (TTT) has recently emerged as a promising direction for efficient sequence modeling. TTT reformulates attention operation as an online learning problem, constructing a compact inner model from key-value pairs at test time. This reformulation opens a rich and flexible design space while achieving linear computational complexity. However, crafting a powerful visual TTT design remains challenging: fundamental choices for the inner module and inner training lack comprehensive understanding and practical guidelines. To bridge this critical gap, in this paper, we present a systematic empirical study of TTT designs for visual sequence modeling. From a series of experiments and analyses, we distill six practical insights that establish design principles for effective visual TTT and illuminate paths for future improvement. These findings culminate in the Vision Test-Time Training (ViT$^3$) model, a pure TTT architecture that achieves linear complexity and parallelizable computation. We evaluate ViT$^3$ across diverse visual tasks, including image classification, image generation, object detection, and semantic segmentation. Results show that ViT$^3$ consistently matches or outperforms advanced linear-complexity models (e.g., Mamba and linear attention variants) and effectively narrows the gap to highly optimized vision Transformers. We hope this study and the ViT$^3$ baseline can facilitate future work on visual TTT models. Code is available at this https URL.
Abstract (translated)
测试时间训练（TTT，Test-Time Training）最近作为高效序列建模的一个有前景的方向而出现。TTT将注意力操作重新定义为一个在线学习问题，在测试时从键值对构造出一个紧凑的内部模型。这种重新定义不仅开辟了一个丰富且灵活的设计空间，还实现了线性计算复杂度。然而，构建强大的视觉TTT设计仍然具有挑战性：对于内核模块和内核训练的基本选择缺乏全面的理解和实用指南。为弥合这一关键差距，本文中我们对视觉序列建模中的TTT设计进行了系统的实证研究。通过一系列实验和分析，我们提炼出了六项实践洞察，这些洞察确立了有效视觉TTT的设计原则，并指明了未来改进的道路。这些发现最终形成了Vision Test-Time Training（ViT$^3$）模型，这是一个纯粹的TTT架构，在实现线性复杂度的同时还支持并行计算。我们在包括图像分类、图像生成、目标检测和语义分割在内的多种视觉任务中评估了ViT$^3$。结果表明，ViT$^3$在性能上始终与最先进的线性复杂度模型（如Mamba和线性注意变体）匹配或超越，并有效地缩小了与高度优化的视觉Transformer之间的差距。我们希望这项研究以及ViT$^3$基准能促进未来关于视觉TTT模型的工作。代码可在此URL获取。 通过这一系统性的实证研究，我们不仅展示了如何设计高效的视觉TTT架构，还为后续的研究工作提供了宝贵的洞见和方向指引。
URL
https://arxiv.org/abs/2512.01643
PDF
-
Toward Content-based Indexing and Retrieval of Head and Neck CT with Abscess Segmentation
2025-12-01 12:04:24Thao Thi Phuong Dao, Tan-Cong Nguyen, Trong-Le Do, Truong Hoang Viet, Nguyen Chi Thanh, Huynh Nguyen Thuan, Do Vo Cong Nguyen, Minh-Khoi Pham, Mai-Khiem Tran, Viet-Tham Huynh, Trong-Thuan Nguyen, Trung-Nghia Le, Vo Thanh Toan, Tam V. Nguyen, Minh-Triet Tran, Thanh Dinh LeAbstract
Abscesses in the head and neck represent an acute infectious process that can potentially lead to sepsis or mortality if not diagnosed and managed promptly. Accurate detection and delineation of these lesions on imaging are essential for diagnosis, treatment planning, and surgical intervention. In this study, we introduce AbscessHeNe, a curated and comprehensively annotated dataset comprising 4,926 contrast-enhanced CT slices with clinically confirmed head and neck abscesses. The dataset is designed to facilitate the development of robust semantic segmentation models that can accurately delineate abscess boundaries and evaluate deep neck space involvement, thereby supporting informed clinical decision-making. To establish performance baselines, we evaluate several state-of-the-art segmentation architectures, including CNN, Transformer, and Mamba-based models. The highest-performing model achieved a Dice Similarity Coefficient of 0.39, Intersection-over-Union of 0.27, and Normalized Surface Distance of 0.67, indicating the challenges of this task and the need for further research. Beyond segmentation, AbscessHeNe is structured for future applications in content-based multimedia indexing and case-based retrieval. Each CT scan is linked with pixel-level annotations and clinical metadata, providing a foundation for building intelligent retrieval systems and supporting knowledge-driven clinical workflows. The dataset will be made publicly available at this https URL.
Abstract (translated)
头部和颈部的脓肿是一种急性感染过程，如果不及时诊断和管理，可能会导致败血症或死亡。在影像学上准确地检测和勾画这些病变对于诊断、治疗计划制定以及手术干预至关重要。在这项研究中，我们介绍了AbscessHeNe数据集，这是一个精心策划且全面标注的数据集，包含4,926张具有临床确认的头部和颈部脓肿的增强CT切片。该数据集旨在促进开发稳健的语义分割模型，这些模型可以准确地勾画脓肿边界并评估深部颈腔的受累情况，从而支持知情的临床决策制定。为了建立性能基准，我们对几种最先进的分割架构进行了评估，包括基于CNN、Transformer和Mamba的模型。其中表现最佳的模型达到了0.39的Dice相似系数（Dice Similarity Coefficient）、0.27的交并比（Intersection-over-Union）以及0.67的归一化表面距离（Normalized Surface Distance），表明了这一任务面临的挑战，并强调需要进一步的研究。 除了分割之外，AbscessHeNe还为未来在基于内容的多媒体索引和案例检索中的应用做好了准备。每个CT扫描都与像素级别的注释及临床元数据相关联，从而为构建智能检索系统和支持知识驱动的临床工作流程奠定了基础。该数据集将在以下网址上公开发布：[请在此处提供链接]。 请注意，原文中提供的URL位置应根据实际数据集发布的地址进行替换。
URL
https://arxiv.org/abs/2512.01589
PDF
-
FlowEO: Generative Unsupervised Domain Adaptation for Earth Observation
2025-12-01 10:29:01Georges Le Bellier (CEDRIC - VERTIGO, Cnam), Nicolas Audebert (LaSTIG, IGN, CEDRIC - VERTIGO)Abstract
The increasing availability of Earth observation data offers unprecedented opportunities for large-scale environmental monitoring and analysis. However, these datasets are inherently heterogeneous, stemming from diverse sensors, geographical regions, acquisition times, and atmospheric conditions. Distribution shifts between training and deployment domains severely limit the generalization of pretrained remote sensing models, making unsupervised domain adaptation (UDA) crucial for real-world applications. We introduce FlowEO, a novel framework that leverages generative models for image-space UDA in Earth observation. We leverage flow matching to learn a semantically preserving mapping that transports from the source to the target image distribution. This allows us to tackle challenging domain adaptation configurations for classification and semantic segmentation of Earth observation images. We conduct extensive experiments across four datasets covering adaptation scenarios such as SAR to optical translation and temporal and semantic shifts caused by natural disasters. Experimental results demonstrate that FlowEO outperforms existing image translation approaches for domain adaptation while achieving on-par or better perceptual image quality, highlighting the potential of flow-matching-based UDA for remote sensing.
Abstract (translated)
地球观测数据的日益增多为大规模环境监测和分析提供了前所未有的机遇。然而，这些数据集本质上是异构的，源于不同的传感器、地理区域、获取时间和大气条件。训练域与部署域之间的分布变化极大地限制了预训练遥感模型的泛化能力，使得无监督领域适应（UDA）对于实际应用至关重要。我们引入了一个新的框架FlowEO，该框架利用生成模型来进行地球观测中的图像空间UDA。通过流匹配来学习一种保持语义的映射方法，将源域图像分布迁移到目标域图像分布。这使我们能够应对地球观测图像分类和语义分割中更具挑战性的领域适应配置。 我们在四个数据集上进行了广泛的实验，涵盖了从合成孔径雷达（SAR）到光学传感器转换以及由自然灾害引起的时间和语义变化等适应场景。实验结果表明，FlowEO在领域适应方面超过了现有的图像翻译方法，并且达到了同等或更好的感知图像质量，突显了基于流匹配的UDA对于遥感应用的巨大潜力。
URL
https://arxiv.org/abs/2512.05140
PDF
-
Panda: Self-distillation of Reusable Sensor-level Representations for High Energy Physics
2025-12-01 06:28:11Samuel Young, Kazuhiro TeraoAbstract
Liquid argon time projection chambers (LArTPCs) provide dense, high-fidelity 3D measurements of particle interactions and underpin current and future neutrino and rare-event experiments. Physics reconstruction typically relies on complex detector-specific pipelines that use tens of hand-engineered pattern recognition algorithms or cascades of task-specific neural networks that require extensive, labeled simulation that requires a careful, time-consuming calibration process. We introduce \textbf{Panda}, a model that learns reusable sensor-level representations directly from raw unlabeled LArTPC data. Panda couples a hierarchical sparse 3D encoder with a multi-view, prototype-based self-distillation objective. On a simulated dataset, Panda substantially improves label efficiency and reconstruction quality, beating the previous state-of-the-art semantic segmentation model with 1,000$\times$ fewer labels. We also show that a single set-prediction head 1/20th the size of the backbone with no physical priors trained on frozen outputs from Panda can result in particle identification that is comparable with state-of-the-art (SOTA) reconstruction tools. Full fine-tuning further improves performance across all tasks.
Abstract (translated)
液氩时间投影室（LArTPC）提供了粒子相互作用的密集且高保真的三维测量，并支撑着当前和未来的中微子以及罕见事件实验。物理重建通常依赖于复杂的、特定探测器的管道系统，这些系统使用数十种手工设计的模式识别算法或一系列针对具体任务的神经网络组合，后者需要大量标注的模拟数据及细致入微且耗时的校准过程。 我们介绍了一种名为**Panda**的新模型，该模型直接从原始未标记的LArTPC数据中学习可重复使用的传感器级表示。Panda结合了一个层次化的稀疏三维编码器和一个多视角、基于原型的自蒸馏目标函数。在模拟数据集上，Panda大幅提高了标签效率并改善了重建质量，在仅使用当前最先进语义分割模型所需标记数目的千分之一的情况下就超过了之前的记录。 我们还展示了这样一个事实：一个单一集合预测头（其大小仅为骨干网络的1/20）并在没有任何物理先验的前提下训练于Panda冻结输出之上，就能获得与最先进的重建工具相当的粒子识别效果。全面微调则进一步提升了所有任务上的性能表现。
URL
https://arxiv.org/abs/2512.01324
PDF
-
Stronger is not better: Better Augmentations in Contrastive Learning for Medical Image Segmentation
2025-11-30 22:42:49Azeez Idris, Abdurahman Ali Mohammed, Samuel FanijoAbstract
Self-supervised contrastive learning is among the recent representation learning methods that have shown performance gains in several downstream tasks including semantic segmentation. This paper evaluates strong data augmentation, one of the most important components for self-supervised contrastive learning's improved performance. Strong data augmentation involves applying the composition of multiple augmentation techniques on images. Surprisingly, we find that the existing data augmentations do not always improve performance for semantic segmentation for medical images. We experiment with other augmentations that provide improved performance.
Abstract (translated)
自监督对比学习是最近出现的表示学习方法之一，已在包括语义分割在内的多个下游任务中表现出性能提升。本文评估了强数据增强技术，这是自监督对比学习改进性能的关键组成部分之一。强数据增强涉及到对图像应用多种数据增强技术的组合。令人惊讶的是，我们发现现有的数据增强技术并不总是能够提高医学图像语义分割的性能。我们在实验中尝试了其他一些能提供更好效果的数据增强方法。
URL
https://arxiv.org/abs/2512.05992
PDF
-
Binary-Gaussian: Compact and Progressive Representation for 3D Gaussian Segmentation
2025-11-30 15:51:30An Yang, Chenyu Liu, Jun Du, Jianqing Gao, Jia Pan, Jinshui Hu, Baocai Yin, Bing Yin, Cong LiuAbstract
3D Gaussian Splatting (3D-GS) has emerged as an efficient 3D representation and a promising foundation for semantic tasks like segmentation. However, existing 3D-GS-based segmentation methods typically rely on high-dimensional category features, which introduce substantial memory overhead. Moreover, fine-grained segmentation remains challenging due to label space congestion and the lack of stable multi-granularity control mechanisms. To address these limitations, we propose a coarse-to-fine binary encoding scheme for per-Gaussian category representation, which compresses each feature into a single integer via the binary-to-decimal mapping, drastically reducing memory usage. We further design a progressive training strategy that decomposes panoptic segmentation into a series of independent sub-tasks, reducing inter-class conflicts and thereby enhancing fine-grained segmentation capability. Additionally, we fine-tune opacity during segmentation training to address the incompatibility between photometric rendering and semantic segmentation, which often leads to foreground-background confusion. Extensive experiments on multiple benchmarks demonstrate that our method achieves state-of-the-art segmentation performance while significantly reducing memory consumption and accelerating inference.
Abstract (translated)
3D高斯点绘（3D-GS）作为一种高效的三维表示方法，已成为语义任务如分割的基础，并展现出巨大潜力。然而，现有的基于3D-GS的分割方法通常依赖于高维类别特征，这会带来显著的内存开销。此外，由于标签空间拥挤和缺乏稳定的多粒度控制机制，细粒度分割仍然具有挑战性。为了解决这些限制，我们提出了一种从粗到细的二进制编码方案，用于每个高斯点类别的表示。该方案将每个特征压缩成一个单一整数，通过二进制到十进制的映射，大大减少了内存使用量。此外，我们还设计了一个渐进式训练策略，将全景分割分解为一系列独立子任务，从而减少类别间的冲突，并提高细粒度分割能力。另外，在分割训练过程中微调不透明度，以解决光度渲染与语义分割之间的兼容性问题，这通常会导致前景和背景的混淆。 在多个基准测试上的广泛实验表明，我们的方法不仅实现了最先进的分割性能，还显著减少了内存消耗并加速了推理过程。
URL
https://arxiv.org/abs/2512.00944
PDF
-
EZ-SP: Fast and Lightweight Superpoint-Based 3D Segmentation
2025-11-29 08:21:51Louis Geist, Loic Landrieu, Damien RobertAbstract
Superpoint-based pipelines provide an efficient alternative to point- or voxel-based 3D semantic segmentation, but are often bottlenecked by their CPU-bound partition step. We propose a learnable, fully GPU partitioning algorithm that generates geometrically and semantically coherent superpoints 13$\times$ faster than prior methods. Our module is compact (under 60k parameters), trains in under 20 minutes with a differentiable surrogate loss, and requires no handcrafted features. Combine with a lightweight superpoint classifier, the full pipeline fits in $<$2 MB of VRAM, scales to multi-million-point scenes, and supports real-time inference. With 72$\times$ faster inference and 120$\times$ fewer parameters, EZ-SP matches the accuracy of point-based SOTA models across three domains: indoor scans (S3DIS), autonomous driving (KITTI-360), and aerial LiDAR (DALES). Code and pretrained models are accessible at this http URL.
Abstract (translated)
基于Superpoint的管道为点或体素基的三维语义分割提供了一种高效的替代方案，但常常受限于其CPU绑定的分区步骤。我们提出了一种可学习、完全在GPU上运行的分区算法，该算法能以比先前方法快13倍的速度生成几何和语义连贯性良好的超点。我们的模块小巧（参数数量低于60k），仅需不到20分钟即可使用可微分代理损失进行训练，并且不需要手工制作的功能特征。结合轻量级的超点分类器，整个管道占用的VRAM小于2MB，能够扩展到包含数百万个点的场景，并支持实时推理。通过实现72倍的速度提升和参数减少至原来的1/120倍，EZ-SP在三个领域（室内扫描[S3DIS]、自动驾驶[KITTI-360]以及航空激光雷达[DALES]）上与基于点的最佳模型精度相匹配。代码及预训练模型可在提供的链接处访问。 请注意，原文中的“this http URL”是一个占位符，实际使用时应该替换为具体的网址链接以便读者访问相关资源。
URL
https://arxiv.org/abs/2512.00385
PDF
-
Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
2025-11-29 07:30:37Aparajitha Allamraju, Maitreya Prafulla Chitale, Hiranmai Sri Adibhatla, Rahul Mishra, Manish ShrivastavaAbstract
Document chunking is a crucial component of Retrieval-Augmented Generation (RAG), as it directly affects the retrieval of relevant and precise context. Conventional fixed-length and recursive splitters often produce arbitrary, incoherent segments that fail to preserve semantic structure. Although semantic chunking has gained traction, its influence on generation quality remains underexplored. This paper introduces two efficient semantic chunking methods, Projected Similarity Chunking (PSC) and Metric Fusion Chunking (MFC), trained on PubMed data using three different embedding models. We further present an evaluation framework that measures the effect of chunking on both retrieval and generation by augmenting PubMedQA with full-text PubMed Central articles. Our results show substantial retrieval improvements (24x with PSC) in MRR and higher Hits@k on PubMedQA. We provide a comprehensive analysis, including statistical significance and response-time comparisons with common chunking libraries. Despite being trained on a single domain, PSC and MFC also generalize well, achieving strong out-of-domain generation performance across multiple datasets. Overall, our findings confirm that our semantic chunkers, especially PSC, consistently deliver superior performance.
Abstract (translated)
文档分块是检索增强生成（RAG）中的一个关键组件，因为它直接影响到相关和精确上下文的检索。传统的固定长度和递归分割方法通常会产生任意且不连贯的片段，无法保留语义结构。尽管基于语义的分块已逐渐受到关注，但其对生成质量的影响仍缺乏深入研究。本文介绍了两种高效的语义分块方法：投影相似度分块（PSC）和度量融合分块（MFC），这些方法是在PubMed数据上使用三种不同的嵌入模型进行训练的。我们进一步提出了一种评估框架，该框架通过将PubMedQA与全文PubMed Central文章结合来衡量分块对检索和生成的影响。我们的结果表明，在MRR（平均倒数排名）指标上有显著提升（PSC提高了24倍），并在PubMedQA上的Hits@k也有所提高。我们还提供了全面的分析，包括统计显著性和响应时间对比与常见分割库的结果。尽管是在单一领域内进行训练，但PSC和MFC方法也能很好地泛化，在多个数据集上实现了强大的跨域生成性能。总体而言，我们的研究结果证实了我们的语义分块器（特别是PSC）在各种任务中始终如一地表现出色。
URL
https://arxiv.org/abs/2512.00367
PDF
-
Towards aligned body representations in vision models
2025-11-29 07:25:32Andrey Gizdov, Andrea Procopio, Yichen Li, Daniel Harari, Tomer UllmanAbstract
Human physical reasoning relies on internal "body" representations - coarse, volumetric approximations that capture an object's extent and support intuitive predictions about motion and physics. While psychophysical evidence suggests humans use such coarse representations, their internal structure remains largely unknown. Here we test whether vision models trained for segmentation develop comparable representations. We adapt a psychophysical experiment conducted with 50 human participants to a semantic segmentation task and test a family of seven segmentation networks, varying in size. We find that smaller models naturally form human-like coarse body representations, whereas larger models tend toward overly detailed, fine-grain encodings. Our results demonstrate that coarse representations can emerge under limited computational resources, and that machine representations can provide a scalable path toward understanding the structure of physical reasoning in the brain.
Abstract (translated)
人类的物理推理依赖于内部“身体”表示形式——这是一种粗糙、体积化的近似，能够捕捉到物体的范围，并支持对运动和物理现象的直观预测。虽然心理物理学证据表明人类使用这种粗略的表示方法，但其内在结构仍然很大程度上未知。在此研究中，我们测试了视觉模型在训练分割任务时是否发展出了类似的表示形式。我们将一项与50名人类参与者进行的心理物理学实验改编为一个语义分割任务，并对七种不同大小的分割网络进行了测试。我们发现较小规模的模型自然地形成了类似人体粗略的身体表示，而较大规模的模型则倾向于产生过于详细、细粒度的编码。我们的研究结果表明，在有限计算资源下可以出现粗糙的表示形式，并且机器表示为理解大脑物理推理结构提供了一条可扩展路径。
URL
https://arxiv.org/abs/2512.00365
PDF
-
Taming the Light: Illumination-Invariant Semantic 3DGS-SLAM
2025-11-28 08:19:40Shouhe Zhang, Dayong Ren, Sensen Song, Yurong Qian, Zhenhong JiaAbstract
Extreme exposure degrades both the 3D map reconstruction and semantic segmentation accuracy, which is particularly detrimental to tightly-coupled systems. To achieve illumination invariance, we propose a novel semantic SLAM framework with two designs. First, the Intrinsic Appearance Normalization (IAN) module proactively disentangles the scene's intrinsic properties, such as albedo, from transient lighting. By learning a standardized, illumination-invariant appearance model, it assigns a stable and consistent color representation to each Gaussian primitive. Second, the Dynamic Radiance Balancing Loss (DRB-Loss) reactively handles frames with extreme exposure. It activates only when an image's exposure is poor, operating directly on the radiance field to guide targeted optimization. This prevents error accumulation from extreme lighting without compromising performance under normal conditions. The synergy between IAN's proactive invariance and DRB-Loss's reactive correction endows our system with unprecedented robustness. Evaluations on public datasets demonstrate state-of-the-art performance in camera tracking, map quality, and semantic and geometric accuracy.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22968
PDF
-
Do We Need Perfect Data? Leveraging Noise for Domain Generalized Segmentation
2025-11-28 07:46:32Taeyeong Kim, SeungJoon Lee, Jung Uk Kim, MyeongAh ChoAbstract
Domain generalization in semantic segmentation faces challenges from domain shifts, particularly under adverse conditions. While diffusion-based data generation methods show promise, they introduce inherent misalignment between generated images and semantic masks. This paper presents FLEX-Seg (FLexible Edge eXploitation for Segmentation), a framework that transforms this limitation into an opportunity for robust learning. FLEX-Seg comprises three key components: (1) Granular Adaptive Prototypes that captures boundary characteristics across multiple scales, (2) Uncertainty Boundary Emphasis that dynamically adjusts learning emphasis based on prediction entropy, and (3) Hardness-Aware Sampling that progressively focuses on challenging examples. By leveraging inherent misalignment rather than enforcing strict alignment, FLEX-Seg learns robust representations while capturing rich stylistic variations. Experiments across five real-world datasets demonstrate consistent improvements over state-of-the-art methods, achieving 2.44% and 2.63% mIoU gains on ACDC and Dark Zurich. Our findings validate that adaptive strategies for handling imperfect synthetic data lead to superior domain generalization. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22948
PDF
-
GazeTrack: High-Precision Eye Tracking Based on Regularization and Spatial Computing
2025-11-27 16:41:32Xiaoyin YangAbstract
Eye tracking has become increasingly important in virtual and augmented reality applications; however, the current gaze accuracy falls short of meeting the requirements for spatial computing. We designed a gaze collection framework and utilized high-precision equipment to gather the first precise benchmark dataset, GazeTrack, encompassing diverse ethnicities, ages, and visual acuity conditions for pupil localization and gaze tracking. We propose a novel shape error regularization method to constrain pupil ellipse fitting and train on open-source datasets, enhancing semantic segmentation and pupil position prediction accuracy. Additionally, we invent a novel coordinate transformation method similar to paper unfolding to accurately predict gaze vectors on the GazeTrack dataset. Finally, we built a gaze vector generation model that achieves reduced gaze angle error with lower computational complexity compared to other methods.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22607
PDF
-
3D Affordance Keypoint Detection for Robotic Manipulation
2025-11-27 08:05:03Zhiyang Liu, Ruiteng Zhao, Lei Zhou, Chengran Yuan, Yuwei Wu, Sheng Guo, Zhengshen Zhang, Chenchen Liu, Marcelo H Ang JrAbstract
This paper presents a novel approach for affordance-informed robotic manipulation by introducing 3D keypoints to enhance the understanding of object parts' functionality. The proposed approach provides direct information about what the potential use of objects is, as well as guidance on where and how a manipulator should engage, whereas conventional methods treat affordance detection as a semantic segmentation task, focusing solely on answering the what question. To address this gap, we propose a Fusion-based Affordance Keypoint Network (FAKP-Net) by introducing 3D keypoint quadruplet that harnesses the synergistic potential of RGB and Depth image to provide information on execution position, direction, and extent. Benchmark testing demonstrates that FAKP-Net outperforms existing models by significant margins in affordance segmentation task and keypoint detection task. Real-world experiments also showcase the reliability of our method in accomplishing manipulation tasks with previously unseen objects.
Abstract (translated)
URL
https://arxiv.org/abs/2511.22195
PDF
-
Shift-Equivariant Complex-Valued Convolutional Neural Networks
2025-11-26 10:29:42Quentin Gabot, Teck-Yian Lim, J\'er\'emy Fix, Joana Frontera-Pons, Chengfang Ren, Jean-Philippe OvarlezAbstract
Convolutional neural networks have shown remarkable performance in recent years on various computer vision problems. However, the traditional convolutional neural network architecture lacks a critical property: shift equivariance and invariance, broken by downsampling and upsampling operations. Although data augmentation techniques can help the model learn the latter property empirically, a consistent and systematic way to achieve this goal is by designing downsampling and upsampling layers that theoretically guarantee these properties by construction. Adaptive Polyphase Sampling (APS) introduced the cornerstone for shift invariance, later extended to shift equivariance with Learnable Polyphase up/downsampling (LPS) applied to real-valued neural networks. In this paper, we extend the work on LPS to complex-valued neural networks both from a theoretical perspective and with a novel building block of a projection layer from $\mathbb{C}$ to $\mathbb{R}$ before the Gumbel Softmax. We finally evaluate this extension on several computer vision problems, specifically for either the invariance property in classification tasks or the equivariance property in both reconstruction and semantic segmentation problems, using polarimetric Synthetic Aperture Radar images.
Abstract (translated)
URL
https://arxiv.org/abs/2511.21250
PDF
-
Open Vocabulary Compositional Explanations for Neuron Alignment
2025-11-25 23:45:37Biagio La Rosa, Leilani H. GilpinAbstract
Neurons are the fundamental building blocks of deep neural networks, and their interconnections allow AI to achieve unprecedented results. Motivated by the goal of understanding how neurons encode information, compositional explanations leverage logical relationships between concepts to express the spatial alignment between neuron activations and human knowledge. However, these explanations rely on human-annotated datasets, restricting their applicability to specific domains and predefined concepts. This paper addresses this limitation by introducing a framework for the vision domain that allows users to probe neurons for arbitrary concepts and datasets. Specifically, the framework leverages masks generated by open vocabulary semantic segmentation to compute open vocabulary compositional explanations. The proposed framework consists of three steps: specifying arbitrary concepts, generating semantic segmentation masks using open vocabulary models, and deriving compositional explanations from these masks. The paper compares the proposed framework with previous methods for computing compositional explanations both in terms of quantitative metrics and human interpretability, analyzes the differences in explanations when shifting from human-annotated data to model-annotated data, and showcases the additional capabilities provided by the framework in terms of flexibility of the explanations with respect to the tasks and properties of interest.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20931
PDF
-
Automated Monitoring of Cultural Heritage Artifacts Using Semantic Segmentation
2025-11-25 17:42:11Andrea Ranieri, Giorgio Palmieri, Silvia BiasottiAbstract
This paper addresses the critical need for automated crack detection in the preservation of cultural heritage through semantic segmentation. We present a comparative study of U-Net architectures, using various convolutional neural network (CNN) encoders, for pixel-level crack identification on statues and monuments. A comparative quantitative evaluation is performed on the test set of the OmniCrack30k dataset [1] using popular segmentation metrics including Mean Intersection over Union (mIoU), Dice coefficient, and Jaccard index. This is complemented by an out-of-distribution qualitative evaluation on an unlabeled test set of real-world cracked statues and monuments. Our findings provide valuable insights into the capabilities of different CNN- based encoders for fine-grained crack segmentation. We show that the models exhibit promising generalization capabilities to unseen cultural heritage contexts, despite never having been explicitly trained on images of statues or monuments.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20541
PDF
-
CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentation
2025-11-25 13:41:59Shilei Cao, Ziyang Gong, Hehai Lin, Yang Liu, Jiashun Cheng, Xiaoxing Hu, Haoyuan Liang, Guowen Li, Chengwei Qin, Hong Cheng, Xue Yang, Juepeng Zheng, Haohuan FuAbstract
In Remote Sensing (RS), Parameter-Efficient Fine-Tuning (PEFT) has emerged as a key approach to activate the generalizable representation ability of foundation models for downstream tasks. However, existing specialized PEFT methods often fail when applied to large-scale Earth observation tasks, as they are unable to fully handle the multifaceted and unpredictable domain gaps (\eg, spatial, semantic, and frequency shifts) inherent in RS data. To overcome this, we propose CrossEarth-Gate, which introduces two primary contributions. First, we establish a comprehensive RS module toolbox to address multifaceted domain gaps, comprising spatial, semantic, and frequency modules. Second, we develop a Fisher-guided adaptive selection mechanism that operates on this toolbox. This selection is guided by Fisher Information to quantify each module's importance by measuring its contribution to the task-specific gradient flow. It dynamically activates only the most critical modules at the appropriate layers, guiding the gradient flow to maximize adaptation effectiveness and efficiency. Comprehensive experiments validate the efficacy and generalizability of our method, where CrossEarth-Gate achieves state-of-the-art performance across 16 cross-domain benchmarks for RS semantic segmentation. The code of the work will be released.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20302
PDF
-
SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM
2025-11-25 07:52:07Lin Chen, Yingjian Zhu, Qi Yang, Xin Niu, Kun Ding, Shiming XiangAbstract
Open-vocabulary semantic segmentation (OVSS) aims to segment and recognize objects universally. Trained on extensive high-quality segmentation data, the segment anything model (SAM) has demonstrated remarkable universal segmentation capabilities, offering valuable support for OVSS. Although previous methods have made progress in leveraging SAM for OVSS, there are still some challenges: (1) SAM's tendency to over-segment and (2) hard combinations between fixed masks and labels. This paper introduces a novel mask-injected framework, SAM-MI, which effectively integrates SAM with OVSS models to address these challenges. Initially, SAM-MI employs a Text-guided Sparse Point Prompter to sample sparse prompts for SAM instead of previous dense grid-like prompts, thus significantly accelerating the mask generation process. The framework then introduces Shallow Mask Aggregation (SMAgg) to merge partial masks to mitigate the SAM's over-segmentation issue. Finally, Decoupled Mask Injection (DMI) incorporates SAM-generated masks for guidance at low-frequency and high-frequency separately, rather than directly combining them with labels. Extensive experiments on multiple benchmarks validate the superiority of SAM-MI. Notably, the proposed method achieves a 16.7% relative improvement in mIoU over Grounded-SAM on the MESS benchmark, along with a 1.6$\times$ speedup. We hope SAM-MI can serve as an alternative methodology to effectively equip the OVSS model with SAM.
Abstract (translated)
URL
https://arxiv.org/abs/2511.20027
PDF
-
Lightweight Transformer Framework for Weakly Supervised Semantic Segmentation
2025-11-24 22:40:57Ali Torabi, Sanjog Gaihre, Yaqoob MajeedAbstract
Weakly supervised semantic segmentation (WSSS) must learn dense masks from noisy, under-specified cues. We revisit the SegFormer decoder and show that three small, synergistic changes make weak supervision markedly more effective-without altering the MiT backbone or relying on heavy post-processing. Our method, CrispFormer, augments the decoder with: (1) a boundary branch that supervises thin object contours using a lightweight edge head and a boundary-aware loss; (2) an uncertainty-guided refiner that predicts per-pixel aleatoric uncertainty and uses it to weight losses and gate a residual correction of the segmentation logits; and (3) a dynamic multi-scale fusion layer that replaces static concatenation with spatial softmax gating over multi-resolution features, optionally modulated by uncertainty. The result is a single-pass model that preserves crisp boundaries, selects appropriate scales per location, and resists label noise from weak cues. Integrated into a standard WSSS pipeline (seed, student, and EMA relabeling), CrispFormer consistently improves boundary F-score, small-object recall, and mIoU over SegFormer baselines trained on the same seeds, while adding minimal compute. Our decoder-centric formulation is simple to implement, broadly compatible with existing SegFormer variants, and offers a reproducible path to higher-fidelity masks from image-level supervision.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19765
PDF
-
RADSeg: Unleashing Parameter and Compute Efficient Zero-Shot Open-Vocabulary Segmentation Using Agglomerative Models
2025-11-24 21:15:01Omar Alama, Darshil Jariwala, Avigyan Bhattacharya, Seungchan Kim, Wenshan Wang, Sebastian SchererAbstract
Open-vocabulary semantic segmentation (OVSS) underpins many vision and robotics tasks that require generalizable semantic understanding. Existing approaches either rely on limited segmentation training data, which hinders generalization, or apply zero-shot heuristics to vision-language models (e.g CLIP), while the most competitive approaches combine multiple models to improve performance at the cost of high computational and memory demands. In this work, we leverage an overlooked agglomerative vision foundation model, RADIO, to improve zero-shot OVSS along three key axes simultaneously: mIoU, latency, and parameter efficiency. We present the first comprehensive study of RADIO for zero-shot OVSS and enhance its performance through self-correlating recursive attention, self-correlating global aggregation, and computationally efficient mask refinement. Our approach, RADSeg, achieves 6-30% mIoU improvement in the base ViT class while being 3.95x faster and using 2.5x fewer parameters. Surprisingly, RADSeg-base (105M) outperforms previous combinations of huge vision models (850-1350M) in mIoU, achieving state-of-the-art accuracy with substantially lower computational and memory cost.
Abstract (translated)
URL
https://arxiv.org/abs/2511.19704
PDF
