Semantic_Segmentation
-
FedDSR: Federated Deep Supervision and Regularization Towards Autonomous Driving
2025-12-07 06:23:59Wei-Bin Kou, Guangxu Zhu, Bingyang Cheng, Chen Zhang, Yik-Chung Wu, Jianping WangAbstract
Federated Learning (FL) enables collaborative training of autonomous driving (AD) models across distributed vehicles while preserving data privacy. However, FL encounters critical challenges such as poor generalization and slow convergence due to non-independent and identically distributed (non-IID) data from diverse driving environments. To overcome these obstacles, we introduce Federated Deep Supervision and Regularization (FedDSR), a paradigm that incorporates multi-access intermediate layer supervision and regularization within federated AD system. Specifically, FedDSR comprises following integral strategies: (I) to select multiple intermediate layers based on predefined architecture-agnostic standards. (II) to compute mutual information (MI) and negative entropy (NE) on those selected layers to serve as intermediate loss and regularizer. These terms are integrated into the output-layer loss to form a unified optimization objective, enabling comprehensive optimization across the network hierarchy. (III) to aggregate models from vehicles trained based on aforementioned rules of (I) and (II) to generate the global model on central server. By guiding and penalizing the learning of feature representations at intermediate stages, FedDSR enhances the model generalization and accelerates model convergence for federated AD. We then take the semantic segmentation task as an example to assess FedDSR and apply FedDSR to multiple model architectures and FL algorithms. Extensive experiments demonstrate that FedDSR achieves up to 8.93% improvement in mIoU and 28.57% reduction in training rounds, compared to other FL baselines, making it highly suitable for practical deployment in federated AD ecosystems.
Abstract (translated)
联邦学习（FL）使分散车辆之间能够协作训练自动驾驶（AD）模型，同时保护数据隐私。然而，由于来自不同驾驶环境的数据不具备独立同分布特性（non-IID），联邦学习面临着诸如泛化能力差和收敛速度慢等关键挑战。为克服这些障碍，我们引入了联邦深度监督与正则化（FedDSR）范式，在该范式中，多通道中间层监督和正则化被整合到联邦自动驾驶系统中。具体而言，FedDSR包含以下核心策略：(I) 根据预定义的架构无关标准选择多个中间层。(II) 在所选的这些层上计算互信息（MI）和负熵（NE），以充当中间损失函数和正则化项。这些术语被整合到输出层损失中，形成统一的优化目标，从而在整个网络层次结构中实现全面优化。(III) 将基于上述(I)和(II)规则训练的车辆模型聚合在中央服务器上生成全局模型。 通过指导和惩罚中间阶段特征表示的学习过程，FedDSR提升了模型的泛化能力和加快了联邦自动驾驶中的模型收敛速度。然后以语义分割任务为例评估FedDSR，并将其应用于多种模型架构和联邦学习算法中。广泛的实验表明，与其它联邦学习基准相比，FedDSR在mIoU方面提高了高达8.93%，并减少了28.57%的训练轮次，这使得它非常适合部署于联邦自动驾驶生态系统中。
URL
https://arxiv.org/abs/2512.06676
PDF
-
Statistic-Augmented, Decoupled MoE Routing and Aggregating in Autonomous Driving
2025-12-07 05:28:40Wei-Bin Kou, Guangxu Zhu, Jingreng Lei, Chen Zhang, Yik-Chung Wu, Jianping WangAbstract
Autonomous driving (AD) scenarios are inherently complex and diverse, posing significant challenges for a single deep learning model to effectively cover all possible conditions, such as varying weather, traffic densities, and road types. Large Model (LM)-Driven Mixture of Experts (MoE) paradigm offers a promising solution, where LM serves as the backbone to extract latent features while MoE serves as the downstream head to dynamically select and aggregate specialized experts to adapt to different scenarios. However, routing and aggregating in MoE face intrinsic challenges, including imprecise expert selection due to flawed routing strategy and inefficient expert aggregation leading to suboptimal prediction. To address these issues, we propose a statistic-augmented, decoupled MoE }outing and Aggregating Mechanism (MoE-RAM) driven by LM. Specifically, on the one hand, MoE-RAM enhances expert routing by incorporating statistical retrieval mechanism to match LM-extracted latent features with cached prototypical features of the most relevant experts; on the other hand, MoE-RAM adaptively reweights experts' outputs in fusion by measuring statistical distances of experts' instant features against LM-extracted latent features. Benefiting from the synergy of the statistic-augmented MoE's routing and aggregating, MoE-RAM ultimately improves the prediction performance. We take the AD semantic segmentation task as an example to assess the proposed MoE-RAM. Extensive experiments on AD datasets demonstrate the superiority of MoE-RAM compared to other MoE baselines and conventional single-model approaches.
Abstract (translated)
自动驾驶（AD）场景本质上是复杂且多样的，这给单一的深度学习模型有效覆盖所有可能条件带来了巨大挑战，这些条件包括不同的天气状况、交通密度和道路类型。大型模型（LM）驱动的专家混合体（MoE）范式提供了一个有希望的解决方案，在这种方案中，LM作为主干来提取潜在特征，而MoE则作为下游模块动态选择并聚合专门化的专家以适应不同场景。然而，MoE中的路由和聚集面对着固有的挑战，包括由于路由策略不当导致的不精确的专家选择以及低效的专家聚合从而导致次优预测的问题。 为了应对这些问题，我们提出了一种由LM驱动、统计增强且解耦的MoE路由与聚合机制（MoE-RAM）。具体来说，在一方面，通过引入统计检索机制来匹配从LM提取出的潜在特征和缓存的相关专家原型特征，MoE-RAM增强了专家选择过程；另一方面，MoE-RAM在融合过程中根据即时特征与从LM提取的潜在特征之间的统计距离自适应地重新加权各个专家的输出。得益于这种统计增强型MoE路由和聚集机制的协同作用，MoE-RAM最终提高了预测性能。 我们选取了AD语义分割任务作为示例来评估所提出的MoE-RAM。在一系列AD数据集上的广泛实验表明，相较于其他MoE基准模型及传统的单模型方法，MoE-RAM表现出显著优势。
URL
https://arxiv.org/abs/2512.06664
PDF
-
CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks
2025-12-07 05:26:30Yu Qi, Yumeng Zhang, Chenting Gong, Xiao Tan, Weiming Zhang, Wei Zhang, Jingdong WangAbstract
Large Vision-Language Models (LVLMs) have demonstrated remarkable success in a broad range of vision-language tasks, such as general visual question answering and optical character recognition (OCR). However, their performance on perception-centric tasks -- such as object detection, semantic segmentation, and depth estimation -- remains significantly inferior to that of task-specific expert models. For example, Qwen2.5-VL-7B-Instruct achieves only 19% mAP on COCO2017 val, particularly struggling with dense scenes and small object recall. In this work, we introduce Chain-of-Thought for Detection (CoT4Det), a simple but efficient strategy that reformulates perception tasks into three interpretable steps: classification, counting, and grounding -- each more naturally aligned with the reasoning capabilities of LVLMs. Extensive experiments demonstrate that our method significantly improves perception performance without compromising general vision language capabilities. With a standard Qwen2.5-VL-7B-Instruct, CoT4Det boosts mAP from 19.0% to 33.0% on COCO2017 val and achieves competitive results across a variety of perception benchmarks, outperforming baselines by +2% on RefCOCO series and 19% on Flickr30k entities.
Abstract (translated)
大型视觉-语言模型（LVLMs）在广泛的视觉-语言任务中展示了显著的成功，例如通用的视觉问答和光学字符识别（OCR）。然而，在以感知为中心的任务——如物体检测、语义分割和深度估计上，它们的表现仍然明显低于特定任务的专家模型。例如，Qwen2.5-VL-7B-Instruct在COCO2017验证集上的mAP仅为19%，尤其在密集场景和小目标召回方面表现不佳。 在这项工作中，我们介绍了用于检测的Chain-of-Thought方法（CoT4Det），这是一种简单而高效的策略，它将感知任务重新表述为三个可解释的步骤：分类、计数和定位——每个步骤都更自然地与LVLMs的推理能力相匹配。广泛的实验表明，我们的方法在不牺牲通用视觉语言能力的情况下显著提高了感知性能。使用标准的Qwen2.5-VL-7B-Instruct模型，CoT4Det将COCO2017验证集上的mAP从19.0%提升到33.0%，并在各种感知基准测试中取得了具有竞争力的结果，在RefCOCO系列上比基线高出+2%，在Flickr30k实体数据集上高出了19%。
URL
https://arxiv.org/abs/2512.06663
PDF
-
Pseudo-Label Refinement for Robust Wheat Head Segmentation via Two-Stage Hybrid Training
2025-12-07 02:48:50Jiahao Jiang, Zhangrui Yang, Xuanhan Wang, Jingkuan SongAbstract
This extended abstract details our solution for the Global Wheat Full Semantic Segmentation Competition. We developed a systematic self-training framework. This framework combines a two-stage hybrid training strategy with extensive data augmentation. Our core model is SegFormer with a Mix Transformer (MiT-B4) backbone. We employ an iterative teacher-student loop. This loop progressively refines model accuracy. It also maximizes data utilization. Our method achieved competitive performance. This was evident on both the Development and Testing Phase datasets.
Abstract (translated)
这篇扩展摘要详细介绍了我们为全球小麦全语义分割竞赛提出的解决方案。我们开发了一个系统的自我训练框架，该框架结合了两阶段混合训练策略和广泛的数据增强技术。我们的核心模型是采用Mix Transformer（MiT-B4）骨干网络的SegFormer模型。我们采用了迭代的教师-学生循环机制，这一机制逐步提高模型的准确性，并最大化数据利用率。我们的方法在开发和测试阶段的数据集上均取得了具有竞争力的表现。
URL
https://arxiv.org/abs/2512.11874
PDF
-
LPD: Learnable Prototypes with Diversity Regularization for Weakly Supervised Histopathology Segmentation
2025-12-05 17:59:16Khang Le, Anh Mai Vu, Thi Kim Trang Vo, Ha Thach, Ngoc Bui Lam Quang, Thanh-Huy Nguyen, Minh H. N. Le, Zhu Han, Chandra Mohan, Hien Van NguyenAbstract
Weakly supervised semantic segmentation (WSSS) in histopathology reduces pixel-level labeling by learning from image-level labels, but it is hindered by inter-class homogeneity, intra-class heterogeneity, and CAM-induced region shrinkage (global pooling-based class activation maps whose activations highlight only the most distinctive areas and miss nearby class regions). Recent works address these challenges by constructing a clustering prototype bank and then refining masks in a separate stage; however, such two-stage pipelines are costly, sensitive to hyperparameters, and decouple prototype discovery from segmentation learning, limiting their effectiveness and efficiency. We propose a cluster-free, one-stage learnable-prototype framework with diversity regularization to enhance morphological intra-class heterogeneity coverage. Our approach achieves state-of-the-art (SOTA) performance on BCSS-WSSS, outperforming prior methods in mIoU and mDice. Qualitative segmentation maps show sharper boundaries and fewer mislabels, and activation heatmaps further reveal that, compared with clustering-based prototypes, our learnable prototypes cover more diverse and complementary regions within each class, providing consistent qualitative evidence for their effectiveness.
Abstract (translated)
弱监督语义分割（WSSS）在组织病理学中通过从图像级别标签学习来减少像素级标注，但该方法受到类内异质性、类间同质性和CAM诱导的区域收缩等问题的影响。最近的研究通过构建聚类原型库并在独立阶段进行掩码细化来解决这些问题；然而，这类两阶段管道成本高，对超参数敏感，并且将原型发现与分割学习解耦，限制了其有效性和效率。我们提出了一种无需集群的一阶段可学习原型框架，并采用多样性正则化以增强形态类内异质性的覆盖范围。我们的方法在BCSS-WSSS数据集上达到了最先进的（SOTA）性能，在mIoU和mDice指标上超越了以前的方法。定性分割图显示出了更清晰的边界并且错误标注较少，而激活热图进一步揭示了相比于基于聚类的原型，我们的可学习原型能够覆盖每个类别内更多样化且互补的区域，提供了其有效性的持续证据。
URL
https://arxiv.org/abs/2512.05922
PDF
-
Label-Efficient Point Cloud Segmentation with Active Learning
2025-12-05 14:47:24Johannes Meyer, Jasper Hoffmann, Felix Schulz, Dominik Merkle, Daniel Buescher, Alexander Reiterer, Joschka Boedecker, Wolfram BurgardAbstract
Semantic segmentation of 3D point cloud data often comes with high annotation costs. Active learning automates the process of selecting which data to annotate, reducing the total amount of annotation needed to achieve satisfactory performance. Recent approaches to active learning for 3D point clouds are often based on sophisticated heuristics for both, splitting point clouds into annotatable regions and selecting the most beneficial for further neural network training. In this work, we propose a novel and easy-to-implement strategy to separate the point cloud into annotatable regions. In our approach, we utilize a 2D grid to subdivide the point cloud into columns. To identify the next data to be annotated, we employ a network ensemble to estimate the uncertainty in the network output. We evaluate our method on the S3DIS dataset, the Toronto-3D dataset, and a large-scale urban 3D point cloud of the city of Freiburg, which we labeled in parts manually. The extensive evaluation shows that our method yields performance on par with, or even better than, complex state-of-the-art methods on all datasets. Furthermore, we provide results suggesting that in the context of point clouds the annotated area can be a more meaningful measure for active learning algorithms than the number of annotated points.
Abstract (translated)
三维点云数据的语义分割通常伴随着高昂的数据标注成本。主动学习通过自动化选择需要标注的数据的过程，从而减少了实现满意性能所需的总标注量。最近针对3D点云的主动学习方法经常基于复杂的启发式算法来划分点云成可注释区域以及选择对后续神经网络训练最有益的数据。在这项工作中，我们提出了一种新颖且易于实施的战略，用于将点云划分为可注释区域。在我们的方法中，我们利用一个二维网格将点云划分为列。为了确定下一次要标注的数据，我们采用了一个网络集成来估计网络输出的不确定性。 我们在S3DIS数据集、多伦多-3D数据集以及弗赖堡市的大规模城市三维点云上评估了我们的方法。这些大规模数据集中的一部分是我们手动标记的。详细的评估表明，我们的方法在所有数据集上的表现都与复杂且最先进的方法相当，甚至更好。 此外，我们还提供了结果，显示在点云场景中，标注区域面积可以作为主动学习算法更有意义的度量标准，而不是标注点的数量。
URL
https://arxiv.org/abs/2512.05759
PDF
-
DistillFSS: Synthesizing Few-Shot Knowledge into a Lightweight Segmentation Model
2025-12-05 10:54:23Pasquale De Marinis, Pieter M. Blok, Uzay Kaymak, Rogier Brussee, Gennaro Vessio, Giovanna CastellanoAbstract
Cross-Domain Few-Shot Semantic Segmentation (CD-FSS) seeks to segment unknown classes in unseen domains using only a few annotated examples. This setting is inherently challenging: source and target domains exhibit substantial distribution shifts, label spaces are disjoint, and support images are scarce--making standard episodic methods unreliable and computationally demanding at test time. To address these constraints, we propose DistillFSS, a framework that embeds support-set knowledge directly into a model's parameters through a teacher--student distillation process. By internalizing few-shot reasoning into a dedicated layer within the student network, DistillFSS eliminates the need for support images at test time, enabling fast, lightweight inference, while allowing efficient extension to novel classes in unseen domains through rapid teacher-driven specialization. Combined with fine-tuning, the approach scales efficiently to large support sets and significantly reduces computational overhead. To evaluate the framework under realistic conditions, we introduce a new CD-FSS benchmark spanning medical imaging, industrial inspection, and remote sensing, with disjoint label spaces and variable support sizes. Experiments show that DistillFSS matches or surpasses state-of-the-art baselines, particularly in multi-class and multi-shot scenarios, while offering substantial efficiency gains. The code is available at this https URL.
Abstract (translated)
跨域少量样本语义分割（CD-FSS）旨在仅使用少数标注示例，在未见领域中对未知类别进行分割。这种设置具有固有的挑战性：源和目标领域的分布差异显著，标签空间不相交，并且支持图像稀缺——这些因素使得标准的分段方法在测试时不可靠且计算成本高昂。为了解决这些问题，我们提出了DistillFSS框架，该框架通过教师-学生蒸馏过程将支持集知识直接嵌入到模型参数中。通过将少量样本推理内化为学生网络中的专用层，DistillFSS消除了测试时对支持图像的需求，实现了快速轻量级的推断，并且能够通过快速的教师驱动专业化有效地扩展至未见领域中的新类别。结合微调技术，该方法可以有效处理大量支持集并显著减少计算开销。 为了在现实条件下评估该框架，我们引入了一个新的CD-FSS基准测试，涵盖了医疗成像、工业检测和遥感领域，并且具有不相交的标签空间和可变的支持集大小。实验表明，DistillFSS在多类别和多样本场景中与最先进的基线方法相当或超越它们，并提供了显著的效率提升。 该框架的代码可在以下链接获取：[请在此处插入实际的URL]。
URL
https://arxiv.org/abs/2512.05613
PDF
-
GeoPE:A Unified Geometric Positional Embedding for Structured Tensors
2025-12-04 16:31:12Yupu Yao, Bowen YangAbstract
Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.
Abstract (translated)
标准的视觉变换模型将二维图像展平为一维序列，这破坏了自然的空间拓扑结构。虽然旋转位置编码（RoPE）在处理一维数据时表现优异，但它继承了这一限制，常常将空间上距离较远的补丁（例如位于行边缘的补丁）视为序列中的邻居。现有的二维方法通常独立地处理空间轴，未能解开这种错误的顺序接近与真实的空间距离之间的关联。 为了恢复二维的空间流形结构，我们引入了几何位置编码（GeoPE），这是一个通过四元数将旋转扩展到三维欧几里得空间的框架。为克服非交换性并确保对称性，GeoPE通过在李代数中计算几何平均值来构建一个统一的旋转变换算符。这创建了一个几何耦合编码，有效地分离了空间维度。 大量的实验结果表明，在图像分类、目标检测和三维语义分割任务上，GeoPE持续优于现有的二维RoPE变体，并显著增强了形状偏向性，证明其能够捕捉真实的几何结构。
URL
https://arxiv.org/abs/2512.04963
PDF
-
MT-Depth: Multi-task Instance feature analysis for the Depth Completion
2025-12-04 12:17:33Abdul Haseeb Nizamani, Dandi Zhou, Xinhai SunAbstract
Depth completion plays a vital role in 3D perception systems, especially in scenarios where sparse depth data must be densified for tasks such as autonomous driving, robotics, and augmented reality. While many existing approaches rely on semantic segmentation to guide depth completion, they often overlook the benefits of object-level understanding. In this work, we introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions. Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head. The instance segmentation branch generates per-image foreground masks that guide the depth branch via cross-attention, allowing the network to focus on object-centric regions during refinement. We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower RMSE compared to both a U-Net-only baseline and previous semantic-guided methods, while maintaining competitive MAE. Qualitative and quantitative results demonstrate that the proposed model effectively enhances depth accuracy near object boundaries, occlusions, and thin structures. Our findings suggest that incorporating instance-aware cues offers a promising direction for improving depth completion without relying on dense semantic labels.
Abstract (translated)
深度完成在3D感知系统中扮演着至关重要的角色，特别是在需要将稀疏的深度数据密集化的场景下，例如自动驾驶、机器人技术和增强现实。尽管许多现有的方法依赖于语义分割来指导深度完成任务，但它们往往忽略了对象级别理解的好处。在这项工作中，我们介绍了一种实例感知的深度完成框架，该框架明确地整合了二值实例掩模作为空间先验，以细化深度预测。我们的模型结合了四个主要组件：冻结版YOLO V11实例分割分支、基于U-Net的深度完成骨干网络、跨注意力融合模块和注意力引导的预测头。实例分割分支生成每张图像的前景掩模，并通过跨注意力机制指导深度分支，使网络在细化过程中能够专注于对象中心区域。 我们在虚拟KITTI 2数据集上验证了我们的方法，结果显示与仅基于U-Net的方法以及之前的语义导向方法相比，其均方根误差（RMSE）更低，同时保持了竞争性的平均绝对误差（MAE）。定性和定量结果表明，所提出的模型能够有效提升对象边界、遮挡和细长结构附近的深度精度。我们的研究结果表明，在不依赖密集的语义标签的情况下，融入实例感知线索为提高深度完成性能提供了一条有前景的方向。
URL
https://arxiv.org/abs/2512.04734
PDF
-
DuGI-MAE: Improving Infrared Mask Autoencoders via Dual-Domain Guidance
2025-12-04 06:45:20Yinghui Xing, Xiaoting Su, Shizhou Zhang, Donghao Chu, Di XuAbstract
Infrared imaging plays a critical role in low-light and adverse weather conditions. However, due to the distinct characteristics of infrared images, existing foundation models such as Masked Autoencoder (MAE) trained on visible data perform suboptimal in infrared image interpretation tasks. To bridge this gap, an infrared foundation model known as InfMAE was developed and pre-trained on large-scale infrared datasets. Despite its effectiveness, InfMAE still faces several limitations, including the omission of informative tokens, insufficient modeling of global associations, and neglect of non-uniform noise. In this paper, we propose a Dual-domain Guided Infrared foundation model based on MAE (DuGI-MAE). First, we design a deterministic masking strategy based on token entropy, preserving only high-entropy tokens for reconstruction to enhance informativeness. Next, we introduce a Dual-Domain Guidance (DDG) module, which simultaneously captures global token relationships and adaptively filters non-uniform background noise commonly present in infrared imagery. To facilitate large-scale pretraining, we construct Inf-590K, a comprehensive infrared image dataset encompassing diverse scenes, various target types, and multiple spatial resolutions. Pretrained on Inf-590K, DuGI-MAE demonstrates strong generalization capabilities across various downstream tasks, including infrared object detection, semantic segmentation, and small target detection. Experimental results validate the superiority of the proposed method over both supervised and self-supervised comparison methods. Our code is available in the supplementary material.
Abstract (translated)
红外成像在低光和恶劣天气条件下发挥着关键作用。然而，由于红外图像具有独特的特性，现有的基于可见光数据训练的Masked Autoencoder (MAE)等基础模型，在处理红外图像解释任务时表现不佳。为了弥合这一差距，开发了一个名为InfMAE的红外基础模型，并在大规模红外数据集上进行了预训练。尽管效果显著，InfMAE仍然面临一些限制，包括信息性令牌缺失、全局关联建模不足以及忽视非均匀噪声的问题。 为此，在本文中我们提出了一种基于双域引导的Masked Autoencoder (MAE)红外基础模型（DuGI-MAE）。首先，我们设计了一个基于令牌熵的确定性掩码策略，仅保留高熵令牌进行重建以增强信息量。接下来，我们引入了双域指导模块（DDG），该模块同时捕捉全局令牌关系并自适应地过滤掉通常存在于红外图像中的非均匀背景噪声。 为了支持大规模预训练，我们构建了一个名为Inf-590K的全面红外图像数据集，涵盖了各种场景、不同类型的目标和多个空间分辨率。在Inf-590K上进行预训练后，DuGI-MAE在包括红外目标检测、语义分割以及小目标检测在内的多种下游任务中展示了强大的泛化能力。 实验结果验证了所提出方法相较于监督学习及自监督对比方法的优越性。我们的代码可在补充材料中获取。
URL
https://arxiv.org/abs/2512.04511
PDF
-
A Novel Approach to Tomato Harvesting Using a Hybrid Gripper with Semantic Segmentation and Keypoint Detection
2025-12-03 11:24:44Shahid Ansari, Mahendra Kumar Gohil, Yusuke Maeda, Bishakh BhattacharyaAbstract
This paper presents an autonomous tomato-harvesting system built around a hybrid robotic gripper that combines six soft auxetic fingers with a rigid exoskeleton and a latex basket to achieve gentle, cage-like grasping. The gripper is driven by a servo-actuated Scotch--yoke mechanism, and includes separator leaves that form a conical frustum for fruit isolation, with an integrated micro-servo cutter for pedicel cutting. For perception, an RGB--D camera and a Detectron2-based pipeline perform semantic segmentation of ripe/unripe tomatoes and keypoint localization of the pedicel and fruit center under occlusion and variable illumination. An analytical model derived using the principle of virtual work relates servo torque to grasp force, enabling design-level reasoning about actuation requirements. During execution, closed-loop grasp-force regulation is achieved using a proportional--integral--derivative controller with feedback from force-sensitive resistors mounted on selected fingers to prevent slip and bruising. Motion execution is supported by Particle Swarm Optimization (PSO)--based trajectory planning for a 5-DOF manipulator. Experiments demonstrate complete picking cycles (approach, separation, cutting, grasping, transport, release) with an average cycle time of 24.34~s and an overall success rate of approximately 80\%, while maintaining low grasp forces (0.20--0.50~N). These results validate the proposed hybrid gripper and integrated vision--control pipeline for reliable harvesting in cluttered environments.
Abstract (translated)
本文介绍了一种自主番茄采摘系统，该系统围绕一种混合型机器人夹爪构建而成，这种夹爪结合了六个软辅特异手指、刚性外骨骼和一个乳胶篮子，以实现温和的笼式抓取。夹爪通过伺服驱动的斯科奇-尤克机制进行操作，并包含分隔叶片形成圆锥台状结构用于水果隔离，以及集成微伺服切割器用于果柄切断。在感知方面，RGB-D相机和基于Detectron2的工作流程执行对成熟与未成熟番茄的语义分割，并在遮挡和光照变化下定位果柄和果实中心的关键点位置。利用虚功原理推导出分析模型，将伺服扭矩与抓取力联系起来，从而能够从设计层面理解驱动需求。在实际操作中，使用配备在选定手指上的力敏电阻反馈的PID控制器实现闭环抓握力调节，以防止滑动和擦伤。通过粒子群优化（PSO）轨迹规划支持五自由度机械臂的动作执行。实验表明，在平均周期时间为24.34秒的情况下，该系统可以完成完整的采摘循环（接近、分离、切割、抓取、运输、释放），整体成功率为约80%，同时保持较低的抓握力（0.20至0.50牛顿）。这些结果验证了所提出的混合夹爪及其集成视觉-控制管道在复杂环境中进行可靠收获的能力。
URL
https://arxiv.org/abs/2512.03684
PDF
-
Fast and Flexible Robustness Certificates for Semantic Segmentation
2025-12-03 10:10:16Thomas Massena (IRIT-MISFIT, DTIPG - SNCF, UT3), Corentin Friedrich (IRIT-MISFIT), Franck Mamalet (IRIT-MISFIT), Mathieu Serrurier (IRIT-MISFIT)Abstract
Deep Neural Networks are vulnerable to small perturbations that can drastically alter their predictions for perceptually unchanged inputs. The literature on adversarially robust Deep Learning attempts to either enhance the robustness of neural networks (e.g, via adversarial training) or to certify their decisions up to a given robustness level (e.g, by using randomized smoothing, formal methods or Lipschitz bounds). These studies mostly focus on classification tasks and few efficient certification procedures currently exist for semantic segmentation. In this work, we introduce a new class of certifiably robust Semantic Segmentation networks with built-in Lipschitz constraints that are efficiently trainable and achieve competitive pixel accuracy on challenging datasets such as Cityscapes. Additionally, we provide a novel framework that generalizes robustness certificates for semantic segmentation tasks, where we showcase the flexibility and computational efficiency of using Lipschitz networks. Our approach unlocks real-time compatible certifiably robust semantic segmentation for the first time. Moreover, it allows the computation of worst-case performance under $\ell_2$ attacks of radius $\epsilon$ across a wide range of performance measures. Crucially, we benchmark the runtime of our certification process and find our approach to be around 600 times faster than randomized smoothing methods at inference with comparable certificates on an NVIDIA A100 GPU. Finally, we evaluate the tightness of our worstcase certificates against state-of-the-art adversarial attacks to further validate the performance of our method.
Abstract (translated)
深度神经网络容易受到细微扰动的影响，这些扰动虽然对输入图像的感知影响不大，但可能大幅改变其预测结果。对抗性鲁棒深度学习的研究领域致力于增强神经网络的鲁棒性（例如通过对抗训练）或认证其在特定鲁棒水平下的决策（例如使用随机平滑、形式方法或Lipschitz界）。这类研究主要集中在分类任务上，而对于语义分割任务的有效认证程序则很少见。在这项工作中，我们引入了一类新的可证式稳健的语义分割网络，这些网络内置了Lipschitz约束，并且可以高效地训练，在如Cityscapes等具有挑战性的数据集上达到了竞争性的像素精度。 此外，我们提供了一个新颖的框架，该框架将鲁棒性证书推广到了语义分割任务中，并展示了使用Lipschitz网络时的灵活性和计算效率。我们的方法首次实现了与实时兼容的可证式稳健的语义分割。它还允许在整个性能度量范围内对$\ell_2$攻击半径为$\epsilon$下的最坏情况性能进行计算。 至关重要的是，我们基准测试了认证过程的运行时间，并发现在NVIDIA A100 GPU上与随机平滑方法相比，我们的方法在推理时大约快600倍且证书相当。最后，我们将最差情况下的证书紧度与最先进的对抗性攻击进行了评估，进一步验证了该方法的有效性能。
URL
https://arxiv.org/abs/2512.06010
PDF
-
Exploiting Domain Properties in Language-Driven Domain Generalization for Semantic Segmentation
2025-12-03 06:58:38Seogkyu Jeon, Kibeom Hong, Hyeran ByunAbstract
Recent domain generalized semantic segmentation (DGSS) studies have achieved notable improvements by distilling semantic knowledge from Vision-Language Models (VLMs). However, they overlook the semantic misalignment between visual and textual contexts, which arises due to the rigidity of a fixed context prompt learned on a single source domain. To this end, we present a novel domain generalization framework for semantic segmentation, namely Domain-aware Prompt-driven Masked Transformer (DPMFormer). Firstly, we introduce domain-aware prompt learning to facilitate semantic alignment between visual and textual cues. To capture various domain-specific properties with a single source dataset, we propose domain-aware contrastive learning along with the texture perturbation that diversifies the observable domains. Lastly, to establish a framework resilient against diverse environmental changes, we have proposed the domain-robust consistency learning which guides the model to minimize discrepancies of prediction from original and the augmented images. Through experiments and analyses, we demonstrate the superiority of the proposed framework, which establishes a new state-of-the-art on various DGSS benchmarks. The code is available at this https URL.
Abstract (translated)
近期的领域泛化语义分割（DGSS）研究通过从视觉-语言模型（VLMs）中提炼语义知识取得了显著进步。然而，这些方法忽略了由于在单一源域上学习固定上下文提示而导致的视觉和文本上下文之间的语义不匹配问题。为此，我们提出了一种新的面向领域的语义分割框架，即Domain-aware Prompt-driven Masked Transformer（DPMFormer）。首先，我们引入了领域感知提示学习以促进视觉和文本线索之间的语义对齐。为了捕捉单一源数据集中各种特定于域的特性，我们提出了具有纹理扰动的领域感知对比学习方法，这种方法可以多样化观察到的领域。最后，为了建立一个能够抵御多种环境变化的框架，我们提出了一种领域的鲁棒一致性学习方法，该方法引导模型最小化原始图像和增强后图像预测结果之间的差异。 通过实验和分析，我们证明了所提出的框架在各种DGSS基准测试中的优越性，并建立了新的最先进的性能。代码可在[提供的URL]获取。
URL
https://arxiv.org/abs/2512.03508
PDF
-
Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks
2025-12-02 18:41:10Matthew Dutson, Nathan Labiosa, Yin Li, Mohit GuptaAbstract
When applied sequentially to video, frame-based networks often exhibit temporal inconsistency - for example, outputs that flicker between frames. This problem is amplified when the network inputs contain time-varying corruptions. In this work, we introduce a general approach for adapting frame-based models for stable and robust inference on video. We describe a class of stability adapters that can be inserted into virtually any architecture and a resource-efficient training process that can be performed with a frozen base network. We introduce a unified conceptual framework for describing temporal stability and corruption robustness, centered on a proposed accuracy-stability-robustness loss. By analyzing the theoretical properties of this loss, we identify the conditions where it produces well-behaved stabilizer training. Our experiments validate our approach on several vision tasks including denoising (NAFNet), image enhancement (HDRNet), monocular depth (Depth Anything v2), and semantic segmentation (DeepLabv3+). Our method improves temporal stability and robustness against a range of image corruptions (including compression artifacts, noise, and adverse weather), while preserving or improving the quality of predictions.
Abstract (translated)
当顺序应用于视频时，基于帧的网络经常表现出时间上的一致性问题——例如，在帧之间闪烁的输出。当网络输入包含随时间变化的干扰时，这个问题会被放大。在这项工作中，我们介绍了一种通用方法来适应基于帧的模型，以实现对视频进行稳定和鲁棒推断。我们描述了一类可以插入几乎任何架构中的稳定性适配器，以及一个资源高效的训练过程，该过程可以在冻结的基础网络上完成。我们引入了一个统一的概念框架来描述时间上的稳定性和抗干扰性，并围绕所提出的精度-稳定-健壮损失（accuracy-stability-robustness loss）进行构建。通过分析此损失的理论性质，我们识别了它在产生良好行为的稳定性训练条件下的情况。我们的实验验证了我们在包括去噪（NAFNet）、图像增强（HDRNet）、单目深度估计（Depth Anything v2）和语义分割（DeepLabv3+）等几个视觉任务上的方法的有效性。我们的方法改善了一系列图像干扰下的时间稳定性和鲁棒性（包括压缩伪影、噪声以及恶劣天气），同时保持或提高预测的质量。
URL
https://arxiv.org/abs/2512.03014
PDF
-
MitUNet: Enhancing Floor Plan Recognition using a Hybrid Mix-Transformer and U-Net Architecture
2025-12-02 04:47:53Dmitriy Parashchuk, Alexey Kapshitskiy, Yuriy KaryakinAbstract
Automatic 3D reconstruction of indoor spaces from 2D floor plans requires high-precision semantic segmentation of structural elements, particularly walls. However, existing methods optimized for standard metrics often struggle to detect thin structural components and yield masks with irregular boundaries, lacking the geometric precision required for subsequent vectorization. To address this issue, we introduce MitUNet, a hybrid neural network architecture specifically designed for wall segmentation tasks in the context of 3D modeling. In MitUNet, we utilize a hierarchical Mix-Transformer encoder to capture global context and a U-Net decoder enhanced with scSE attention blocks for precise boundary recovery. Furthermore, we propose an optimization strategy based on the Tversky loss function to effectively balance precision and recall. By fine-tuning the hyperparameters of the loss function, we prioritize the suppression of false positive noise along wall boundaries while maintaining high sensitivity to thin structures. Our experiments on the public CubiCasa5k dataset and a proprietary regional dataset demonstrate that the proposed approach ensures the generation of structurally correct masks with high boundary accuracy, outperforming standard single-task models. MitUNet provides a robust tool for data preparation in automated 3D reconstruction pipelines.
Abstract (translated)
从二维平面图自动重建室内空间的三维模型需要对结构元素进行高精度语义分割，尤其是墙壁。然而，现有的针对标准指标优化的方法往往难以检测到薄型结构部件，并且生成的掩码边界不规则，缺乏后续矢量化所需的几何精确度。为了解决这个问题，我们引入了MitUNet，这是一种专门设计用于三维建模中墙壁分割任务的混合神经网络架构。 在MitUNet中，我们使用了一个分层的Mix-Transformer编码器来捕捉全局上下文，并且利用带有scSE注意力模块增强的U-Net解码器来恢复精确边界。此外，我们提出了一种基于Tversky损失函数的优化策略，以有效平衡精度和召回率。通过调整损失函数的超参数，我们在抑制沿墙壁边界出现的假阳性噪声的同时，保持对薄结构的高度敏感性。 在公共CubiCasa5k数据集和一个专有的地区数据集上的实验表明，我们提出的方法能够生成具有高边准确度且结构正确的掩码，在性能上超越了标准单一任务模型。MitUNet为自动化三维重建流水线的数据准备提供了一个稳健的工具。
URL
https://arxiv.org/abs/2512.02413
PDF
-
Reproducing and Extending RaDelft 4D Radar with Camera-Assisted Labels
2025-12-02 04:12:41Kejia Hu, Mohammed Alsakabi, John M. Dolan, Ozan K. TonguzAbstract
Recent advances in 4D radar highlight its potential for robust environment perception under adverse conditions, yet progress in radar semantic segmentation remains constrained by the scarcity of open source datasets and labels. The RaDelft data set, although seminal, provides only LiDAR annotations and no public code to generate radar labels, limiting reproducibility and downstream research. In this work, we reproduce the numerical results of the RaDelft group and demonstrate that a camera-guided radar labeling pipeline can generate accurate labels for radar point clouds without relying on human annotations. By projecting radar point clouds into camera-based semantic segmentation and applying spatial clustering, we create labels that significantly enhance the accuracy of radar labels. These results establish a reproducible framework that allows the research community to train and evaluate the labeled 4D radar data. In addition, we study and quantify how different fog levels affect the radar labeling performance.
Abstract (translated)
最近在4D雷达领域的进展突显了其在恶劣条件下进行稳健环境感知的潜力，然而，由于开源数据集和标签的稀缺性，雷达语义分割的进步受到了限制。虽然RaDelft数据集具有开创性意义，但它仅提供了LiDAR标注，并且没有公开用于生成雷达标签的代码，这限制了研究的可重复性和后续研究的发展。 在本工作中，我们重现了RaDelft团队的数值结果，并展示了通过相机引导的雷达标记流水线可以在不依赖人工注释的情况下为雷达点云生成准确的标签。通过将雷达点云投影到基于相机的语义分割中并应用空间聚类方法，我们可以创建出能够显著提高雷达标签准确性的标注。这些成果建立了一个可重复的研究框架，允许研究社区对标记后的4D雷达数据进行训练和评估。 此外，我们还探讨并量化了不同雾气水平如何影响雷达标记性能。
URL
https://arxiv.org/abs/2512.02394
PDF
-
SAGE: Style-Adaptive Generalization for Privacy-Constrained Semantic Segmentation Across Domains
2025-12-02 03:20:22Qingmei Li, Yang Zhang, Peifeng Zhang, Haohuan Fu, Juepeng ZhengAbstract
Domain generalization for semantic segmentation aims to mitigate the degradation in model performance caused by domain shifts. However, in many real-world scenarios, we are unable to access the model parameters and architectural details due to privacy concerns and security constraints. Traditional fine-tuning or adaptation is hindered, leading to the demand for input-level strategies that can enhance generalization without modifying model weights. To this end, we propose a \textbf{S}tyle-\textbf{A}daptive \textbf{GE}neralization framework (\textbf{SAGE}), which improves the generalization of frozen models under privacy constraints. SAGE learns to synthesize visual prompts that implicitly align feature distributions across styles instead of directly fine-tuning the backbone. Specifically, we first utilize style transfer to construct a diverse style representation of the source domain, thereby learning a set of style characteristics that can cover a wide range of visual features. Then, the model adaptively fuses these style cues according to the visual context of each input, forming a dynamic prompt that harmonizes the image appearance without touching the interior of the model. Through this closed-loop design, SAGE effectively bridges the gap between frozen model invariance and the diversity of unseen domains. Extensive experiments on five benchmark datasets demonstrate that SAGE achieves competitive or superior performance compared to state-of-the-art methods under privacy constraints and outperforms full fine-tuning baselines in all settings.
Abstract (translated)
领域泛化在语义分割中的目标是缓解由于领域偏移导致的模型性能下降。然而，在许多现实场景中，由于隐私和安全限制的原因，我们无法访问模型参数和架构细节。这使得传统的微调或适应方法受到阻碍，从而产生了需求：通过不修改模型权重的方式来提高泛化能力的方法。为此，我们提出了一个名为**SAGE（Style-Adaptive Generalization框架）**的方案，该框架旨在改进在隐私限制下的冻结模型的泛化性能。 SAGE的学习方式是合成视觉提示，以隐式地跨风格对齐特征分布，而不是直接微调骨干网络。具体来说，我们首先利用风格迁移构建源领域的多样化的风格表示，从而学习一套覆盖广泛视觉特性的风格特征。然后，该模型根据每个输入的视觉上下文自适应融合这些风格线索，形成一个动态提示，这有助于协调图像外观而不触及模型内部结构。通过这种闭环设计，SAGE有效弥合了冻结模型不变性和未见领域多样性的差距。 在五个基准数据集上的广泛实验表明，在隐私限制下，SAGE与现有最佳方法相比取得了竞争性或优越的性能，并且在所有设置中均优于全量微调基线。
URL
https://arxiv.org/abs/2512.02369
PDF
-
Multifractal Recalibration of Neural Networks for Medical Imaging Segmentation
2025-12-01 20:43:28Miguel L. Martins, Miguel T. Coimbra, Francesco RennaAbstract
Multifractal analysis has revealed regularities in many self-seeding phenomena, yet its use in modern deep learning remains limited. Existing end-to-end multifractal methods rely on heavy pooling or strong feature-space decimation, which constrain tasks such as semantic segmentation. Motivated by these limitations, we introduce two inductive priors: Monofractal and Multifractal Recalibration. These methods leverage relationships between the probability mass of the exponents and the multifractal spectrum to form statistical descriptions of encoder embeddings, implemented as channel-attention functions in convolutional networks. Using a U-Net-based framework, we show that multifractal recalibration yields substantial gains over a baseline equipped with other channel-attention mechanisms that also use higher-order statistics. Given the proven ability of multifractal analysis to capture pathological regularities, we validate our approach on three public medical-imaging datasets: ISIC18 (dermoscopy), Kvasir-SEG (endoscopy), and BUSI (ultrasound). Our empirical analysis also provides insights into the behavior of these attention layers. We find that excitation responses do not become increasingly specialized with encoder depth in U-Net architectures due to skip connections, and that their effectiveness may relate to global statistics of instance variability.
Abstract (translated)
多分形分析揭示了许多自激发现象中的规律性，但在现代深度学习中的应用仍然有限。现有的端到端多分形方法依赖于重采样池化或强特征空间降解，这限制了诸如语义分割等任务的灵活性。鉴于这些局限性，我们引入了两种归纳先验：单分形和多分形再校准。这两种方法利用指数的概率质量和多分形频谱之间的关系来形成编码器嵌入的统计描述，并将其实现为卷积网络中的通道注意力函数。使用基于U-Net的框架，我们展示了多分形再校准相对于其他采用高阶统计信息的通道注意机制的基础模型产生了显著的优势。 鉴于多分形分析捕捉病理规律的能力已得到证明，我们在三个公开的医学成像数据集上验证了我们的方法：ISIC18（皮肤镜检查），Kvasir-SEG（内窥镜检查）和BUSI（超声）。我们实证分析还提供了关于这些注意层行为的见解。我们发现，在U-Net架构中，由于跳过连接的存在，兴奋响应不会随着编码器深度的增加而越来越专业化，并且它们的有效性可能与实例变异性的一般统计有关。
URL
https://arxiv.org/abs/2512.02198
PDF
-
Evaluating SAM2 for Video Semantic Segmentation
2025-12-01 15:15:16Syed Hesham Syed Ariff, Yun Liu, Guolei Sun, Jing Yang, Henghui Ding, Xue Geng, Xudong JiangAbstract
The Segmentation Anything Model 2 (SAM2) has proven to be a powerful foundation model for promptable visual object segmentation in both images and videos, capable of storing object-aware memories and transferring them temporally through memory blocks. While SAM2 excels in video object segmentation by providing dense segmentation masks based on prompts, extending it to dense Video Semantic Segmentation (VSS) poses challenges due to the need for spatial accuracy, temporal consistency, and the ability to track multiple objects with complex boundaries and varying scales. This paper explores the extension of SAM2 for VSS, focusing on two primary approaches and highlighting firsthand observations and common challenges faced during this process. The first approach involves using SAM2 to extract unique objects as masks from a given image, with a segmentation network employed in parallel to generate and refine initial predictions. The second approach utilizes the predicted masks to extract unique feature vectors, which are then fed into a simple network for classification. The resulting classifications and masks are subsequently combined to produce the final segmentation. Our experiments suggest that leveraging SAM2 enhances overall performance in VSS, primarily due to its precise predictions of object boundaries.
Abstract (translated)
《分割任何模型2（SAM2）》已被证明是图像和视频中提示式视觉对象分割的强大基础模型，能够存储物体感知记忆并通过内存块进行时间上的传递。虽然SAM2在视频对象分割方面表现出色，可以通过提示提供密集的分割掩码，但将其扩展到密集型视频语义分割（VSS）则面临挑战，因为需要空间精度、时间一致性以及跟踪具有复杂边界和不同尺度的多个物体的能力。本文探讨了将SAM2应用于VSS的方法，并重点介绍了两种主要方法及在此过程中亲身经历的问题与共同面临的挑战。 第一种方法是利用SAM2从给定图像中提取独特的对象作为掩码，同时采用分割网络并行生成和优化初始预测。第二种方法则是使用预测的掩码来抽取独特的特征向量，然后将这些向量输入到一个简单的分类网络进行分类处理。最后，通过合并得到的分类结果与掩码来产生最终的分割。 我们的实验表明，利用SAM2可以显著提高VSS的整体性能，主要原因是它能够精确地预测物体边界。
URL
https://arxiv.org/abs/2512.01774
PDF
-
SSR: Semantic and Spatial Rectification for CLIP-based Weakly Supervised Segmentation
2025-12-01 14:06:50Xiuli Bi, Die Xiao, Junchao Fan, Bin XiaoAbstract
In recent years, Contrastive Language-Image Pretraining (CLIP) has been widely applied to Weakly Supervised Semantic Segmentation (WSSS) tasks due to its powerful cross-modal semantic understanding capabilities. This paper proposes a novel Semantic and Spatial Rectification (SSR) method to address the limitations of existing CLIP-based weakly supervised semantic segmentation approaches: over-activation in non-target foreground regions and background areas. Specifically, at the semantic level, the Cross-Modal Prototype Alignment (CMPA) establishes a contrastive learning mechanism to enforce feature space alignment across modalities, reducing inter-class overlap while enhancing semantic correlations, to rectify over-activation in non-target foreground regions effectively; at the spatial level, the Superpixel-Guided Correction (SGC) leverages superpixel-based spatial priors to precisely filter out interference from non-target regions during affinity propagation, significantly rectifying background over-activation. Extensive experiments on the PASCAL VOC and MS COCO datasets demonstrate that our method outperforms all single-stage approaches, as well as more complex multi-stage approaches, achieving mIoU scores of 79.5% and 50.6%, respectively.
Abstract (translated)
近年来，对比语言-图像预训练（CLIP）由于其强大的跨模态语义理解能力，在弱监督语义分割（WSSS）任务中得到了广泛应用。本文提出了一种新颖的语义和空间矫正（SSR）方法，以解决现有基于 CLIP 的弱监督语义分割方法中存在的问题：非目标前景区域和背景区域中的过度激活现象。 具体来说，在语义层面，跨模态原型对齐（CMPA）建立了一个对比学习机制，强制不同模式之间的特征空间对齐，减少类间重叠并增强语义相关性，从而有效矫正非目标前景区域的过度激活；在空间层面上，超像素引导校正（SGC）利用基于超像素的空间先验知识，在亲和力传播过程中精确过滤掉非目标区域的影响，显著减少了背景的过度激活。 通过 PASCAL VOC 和 MS COCO 数据集上的大量实验表明，我们的方法超越了所有单阶段方法，并且在性能上优于更为复杂的多阶段方法，分别取得了 79.5% 和 50.6% 的 mIoU 分数。
URL
https://arxiv.org/abs/2512.01701
PDF
