Semantic_Segmentation
-
A Novel Deep Learning Method for Segmenting the Left Ventricle in Cardiac Cine MRI
2026-01-04 12:43:46Wenhui Chu, Aobo Jin, Hardik A. GohelAbstract
This research aims to develop a novel deep learning network, GBU-Net, utilizing a group-batch-normalized U-Net framework, specifically designed for the precise semantic segmentation of the left ventricle in short-axis cine MRI scans. The methodology includes a down-sampling pathway for feature extraction and an up-sampling pathway for detail restoration, enhanced for medical imaging. Key modifications include techniques for better contextual understanding crucial in cardiac MRI segmentation. The dataset consists of 805 left ventricular MRI scans from 45 patients, with comparative analysis using established metrics such as the dice coefficient and mean perpendicular distance. GBU-Net significantly improves the accuracy of left ventricle segmentation in cine MRI scans. Its innovative design outperforms existing methods in tests, surpassing standard metrics like the dice coefficient and mean perpendicular distance. The approach is unique in its ability to capture contextual information, often missed in traditional CNN-based segmentation. An ensemble of the GBU-Net attains a 97% dice score on the SunnyBrook testing dataset. GBU-Net offers enhanced precision and contextual understanding in left ventricle segmentation for surgical robotics and medical analysis.
Abstract (translated)
这项研究旨在开发一种新型深度学习网络GBU-Net，利用组批归一化（group-batch-normalized）的U型网框架，专门用于短轴心脏电影磁共振成像（cine MRI）扫描中左心室的精确语义分割。该方法包括一个下采样路径进行特征提取和一个上采样路径恢复细节，并针对医学影像进行了增强。关键改进在于采用了有助于改善心血管MRI分割上下文理解的技术。 数据集由来自45名患者的805个左心室MRI扫描图像组成，通过已建立的指标如骰子系数（dice coefficient）和平均垂直距离（mean perpendicular distance）进行比较分析。GBU-Net显著提高了心脏电影MRI中左心室分割的准确性。其创新设计在测试中优于现有方法，在骰子系数和平均垂直距离等标准度量上表现更佳。 该方法的独特之处在于它能够捕捉到上下文信息，这是传统基于CNN（卷积神经网络）的方法常常忽略的重要细节。GBU-Net的集成模型在SunnyBrook测试数据集上达到了97%的骰子分数。GBU-Net为心脏手术机器人和医学分析提供了更精确、更具上下文理解能力的左心室分割方法。
URL
https://arxiv.org/abs/2601.01512
PDF
-
Rethinking Multimodal Few-Shot 3D Point Cloud Segmentation: From Fused Refinement to Decoupled Arbitration
2026-01-04 09:53:49Wentao Bian, Fenglei XuAbstract
In this paper, we revisit multimodal few-shot 3D point cloud semantic segmentation (FS-PCS), identifying a conflict in "Fuse-then-Refine" paradigms: the "Plasticity-Stability Dilemma." In addition, CLIP's inter-class confusion can result in semantic blindness. To address these issues, we present the Decoupled-experts Arbitration Few-Shot SegNet (DA-FSS), a model that effectively distinguishes between semantic and geometric paths and mutually regularizes their gradients to achieve better generalization. DA-FSS employs the same backbone and pre-trained text encoder as MM-FSS to generate text embeddings, which can increase free modalities' utilization rate and better leverage each modality's information space. To achieve this, we propose a Parallel Expert Refinement module to generate each modal correlation. We also propose a Stacked Arbitration Module (SAM) to perform convolutional fusion and arbitrate correlations for each modality pathway. The Parallel Experts decouple two paths: a Geometric Expert maintains plasticity, and a Semantic Expert ensures stability. They are coordinated via a Decoupled Alignment Module (DAM) that transfers knowledge without propagating confusion. Experiments on popular datasets (S3DIS, ScanNet) demonstrate the superiority of DA-FSS over MM-FSS. Meanwhile, geometric boundaries, completeness, and texture differentiation are all superior to the baseline. The code is available at: this https URL.
Abstract (translated)
在这篇论文中，我们重新审视了多模态少量样本的3D点云语义分割（FS-PCS），发现“融合再细化”范式存在一种矛盾：“可塑性与稳定性困境”。此外，CLIP 的类别间混淆可能导致语义盲区。为了解决这些问题，我们提出了解耦专家仲裁少量样本分割网络（DA-FSS），该模型能够有效区分语义和几何路径，并相互正则化其梯度以实现更好的泛化能力。DA-FSS 使用与MM-FSS相同的骨干网和预训练的文本编码器来生成文本嵌入，这可以增加自由模态的利用率并更好地利用每个模态的信息空间。 为此，我们提出了一个平行专家细化模块来生成每种模式的相关性，并提出堆叠仲裁模块（SAM）执行卷积融合并对每条路径的相关性进行仲裁。平行专家将两条路径解耦：几何专家保持可塑性，而语义专家确保稳定性。它们通过去耦对齐模块（DAM）协调工作，该模块在不传播混淆的情况下转移知识。 在流行数据集（S3DIS、ScanNet）上的实验表明，DA-FSS 在多模态少量样本分割方面优于MM-FSS。同时，在几何边界、完整性和纹理区分度上都比基线表现更优。代码可以在以下链接获得：this https URL.
URL
https://arxiv.org/abs/2601.01456
PDF
-
In defense of the two-stage framework for open-set domain adaptive semantic segmentation
2026-01-04 08:58:03Wenqi Ren, Weijie Wang, Meng Zheng, Ziyan Wu, Yang Tang, Zhun Zhong, Nicu SebeAbstract
Open-Set Domain Adaptation for Semantic Segmentation (OSDA-SS) presents a significant challenge, as it requires both domain adaptation for known classes and the distinction of unknowns. Existing methods attempt to address both tasks within a single unified stage. We question this design, as the annotation imbalance between known and unknown classes often leads to negative transfer of known classes and underfitting for unknowns. To overcome these issues, we propose SATS, a Separating-then-Adapting Training Strategy, which addresses OSDA-SS through two sequential steps: known/unknown separation and unknown-aware domain adaptation. By providing the model with more accurate and well-aligned unknown classes, our method ensures a balanced learning of discriminative features for both known and unknown classes, steering the model toward discovering truly unknown objects. Additionally, we present hard unknown exploration, an innovative data augmentation method that exposes the model to more challenging unknowns, strengthening its ability to capture more comprehensive understanding of target unknowns. We evaluate our method on public OSDA-SS benchmarks. Experimental results demonstrate that our method achieves a substantial advancement, with a +3.85% H-Score improvement for GTA5-to-Cityscapes and +18.64% for SYNTHIA-to-Cityscapes, outperforming previous state-of-the-art methods.
Abstract (translated)
开放集领域自适应语义分割（OSDA-SS）提出了一项重大挑战，即需要同时处理已知类别和未知类别的领域适应问题。现有方法试图在一个统一的阶段解决这两个任务。然而，我们对这种设计提出了质疑，因为已知与未知类之间的标注不平衡往往会带来负面迁移效应，导致已知类学习效果下降及对未知类的学习不足。为了解决这些问题，我们提出了一种名为SATS（分离-适应训练策略）的方法，该方法通过两个连续的步骤来解决OSDA-SS问题：已知/未知分离和识别未知类别的领域自适应。通过提供给模型更为准确且对齐良好的未知类别信息，我们的方法能够确保模型对已知和未知类别的判别特征进行平衡学习，并推动模型发现真正的未知对象。 此外，我们还提出了一种创新的数据增强方法——硬未知探索（hard unknown exploration），这种方法使模型面临更具挑战性的未知情况，从而增强了其捕捉目标未知类别全面理解的能力。我们在公开的OSDA-SS基准测试上评估了我们的方法。实验结果表明，我们的方法取得了显著进步，在GTA5到Cityscapes数据集上的H-Score提升了3.85%，在SYNTHIA到Cityscapes数据集上的提升达到了18.64%，优于现有的最先进方法。
URL
https://arxiv.org/abs/2601.01439
PDF
-
Seamlessly Natural: Image Stitching with Natural Appearance Preservation
2026-01-03 18:40:35Gaetane Lorna N. Tchana, Damaris Belle M. Fotso, Antonio Hendricks, Christophe BobdaAbstract
This paper introduces SENA (SEamlessly NAtural), a geometry-driven image stitching approach that prioritizes structural fidelity in challenging real-world scenes characterized by parallax and depth variation. Conventional image stitching relies on homographic alignment, but this rigid planar assumption often fails in dual-camera setups with significant scene depth, leading to distortions such as visible warps and spherical bulging. SENA addresses these fundamental limitations through three key contributions. First, we propose a hierarchical affine-based warping strategy, combining global affine initialization with local affine refinement and smooth free-form deformation. This design preserves local shape, parallelism, and aspect ratios, thereby avoiding the hallucinated structural distortions commonly introduced by homography-based models. Second, we introduce a geometry-driven adequate zone detection mechanism that identifies parallax-minimized regions directly from the disparity consistency of RANSAC-filtered feature correspondences, without relying on semantic segmentation. Third, building upon this adequate zone, we perform anchor-based seamline cutting and segmentation, enforcing a one-to-one geometric correspondence across image pairs by construction, which effectively eliminates ghosting, duplication, and smearing artifacts in the final panorama. Extensive experiments conducted on challenging datasets demonstrate that SENA achieves alignment accuracy comparable to leading homography-based methods, while significantly outperforming them in critical visual metrics such as shape preservation, texture integrity, and overall visual realism.
Abstract (translated)
本文介绍了SENA（SEamlessly NAtural），这是一种以几何为基础的图像拼接方法，优先考虑具有视差和深度变化等挑战性现实场景中的结构保真度。传统的图像拼接依赖于射影对齐，但这种刚性的平面假设在双摄像头设置中面对显著场景深度时往往失效，导致可见扭曲和球面膨胀等问题。SENA通过三项关键贡献解决了这些根本限制。 首先，我们提出了一种基于层次仿射变换的图像变形策略，结合全局仿射初始化与局部仿射细化以及平滑自由形式变形。这种设计保留了局部形状、平行性和纵横比，从而避免了由射影模型引入的结构扭曲问题。 其次，我们引入了一种几何驱动的充分区域检测机制，该机制通过计算RANSAC滤波特征对应关系的一致性差异直接识别出视差最小化的区域，并不需要依赖语义分割技术。 第三，在此基础上，我们基于锚点进行缝线切割和分段处理，强制确保每对图像之间存在一对一的几何对应关系，从而有效地消除了最终全景图中的幽灵、重复和模糊等伪影。 通过在具有挑战性的数据集上进行的大量实验表明，SENA实现了与领先的射影模型方法相媲美的对齐精度，但在诸如形状保持、纹理完整性和整体视觉真实感等关键视觉指标方面表现显著更好。
URL
https://arxiv.org/abs/2601.01257
PDF
-
Cross-Layer Attentive Feature Upsampling for Low-latency Semantic Segmentation
2026-01-03 12:09:49Tianheng Cheng, Xinggang Wang, Junchao Liao, Wenyu LiuAbstract
Semantic segmentation is a fundamental problem in computer vision and it requires high-resolution feature maps for dense prediction. Current coordinate-guided low-resolution feature interpolation methods, e.g., bilinear interpolation, produce coarse high-resolution features which suffer from feature misalignment and insufficient context information. Moreover, enriching semantics to high-resolution features requires a high computation burden, so that it is challenging to meet the requirement of lowlatency inference. We propose a novel Guided Attentive Interpolation (GAI) method to adaptively interpolate fine-grained high-resolution features with semantic features to tackle these issues. Guided Attentive Interpolation determines both spatial and semantic relations of pixels from features of different resolutions and then leverages these relations to interpolate high-resolution features with rich semantics. GAI can be integrated with any deep convolutional network for efficient semantic segmentation. In experiments, the GAI-based semantic segmentation networks, i.e., GAIN, can achieve78.8 mIoU with 22.3 FPS on Cityscapes and 80.6 mIoU with 64.5 on CamVid using an NVIDIA 1080Ti GPU, which are the new state-of-the-art results of low-latency semantic segmentation. Code and models are available at: this https URL.
Abstract (translated)
语义分割是计算机视觉中的一个基本问题，它需要高分辨率的特征图来进行密集预测。当前的坐标引导低分辨率特征插值方法（如双线性插值），会产生粗略的高分辨率特征，这些特征存在特征错位和上下文信息不足的问题。此外，增强高分辨率特征的语义信息会带来较高的计算负担，从而难以满足低延迟推理的需求。为此，我们提出了一种新颖的引导注意力插值（Guided Attentive Interpolation, GAI）方法，该方法能够自适应地将精细粒度的高分辨率特征与语义特征相结合来解决这些问题。引导注意力插值确定了不同分辨率特征图中像素的空间和语义关系，并利用这些关系插值生成包含丰富语义信息的高分辨率特征。GAI可以与任何深度卷积网络结合使用，以实现高效的语义分割。在实验中，基于GAI的语义分割网络（即GAIN）在Cityscapes数据集上达到了78.8 mIoU和22.3 FPS，在CamVid数据集上达到了80.6 mIoU和64.5 FPS的结果，使用NVIDIA 1080Ti GPU进行测试。这些结果代表了低延迟语义分割的最新技术水平。 代码及模型可以在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2601.01167
PDF
-
Learning to Segment Liquids in Real-world Images
2026-01-02 16:16:25Jonas Li, Michelle Li, Luke Liu, Heng FanAbstract
Different types of liquids such as water, wine and medicine appear in all aspects of daily life. However, limited attention has been given to the task, hindering the ability of robots to avoid or interact with liquids safely. The segmentation of liquids is difficult because liquids come in diverse appearances and shapes; moreover, they can be both transparent or reflective, taking on arbitrary objects and scenes from the background or surroundings. To take on this challenge, we construct a large-scale dataset of liquids named LQDS consisting of 5000 real-world images annotated into 14 distinct classes, and design a novel liquid detection model named LQDM, which leverages cross-attention between a dedicated boundary branch and the main segmentation branch to enhance segmentation predictions. Extensive experiments demonstrate the effectiveness of LQDM on the test set of LQDS, outperforming state-of-the-art methods and establishing a strong baseline for the semantic segmentation of liquids.
Abstract (translated)
不同类型的液体，如水、酒和药剂，在日常生活的各个方面都会出现。然而，这个领域的问题并未得到足够的关注，这限制了机器人安全地避开或与液体互动的能力。液体质检的难度在于液体外观多样且形态各异；此外，它们可能是透明的或是反光的，并能随背景或周围环境的变化而变化。为了应对这一挑战，我们构建了一个名为LQDS的大规模液体质检数据集，该数据集中包含5000张真实世界的图片并标注为14种不同的类别。同时，我们设计了一种新型液体检测模型LQDM，它利用专用边界分支和主要分割分支之间的交叉注意力机制来增强分割预测的准确性。 通过广泛的实验验证了LQDM在LQDS测试集上的有效性，并且其性能超过了现有的先进技术，为液体质检的语义分割设定了一个强大的基准。
URL
https://arxiv.org/abs/2601.00940
PDF
-
Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformers
2026-01-01 14:29:31S\"ohnke Benedikt Fischedick, Daniel Seichter, Benedict Stephan, Robin Schmidt, Horst-Michael GrossAbstract
In domestic environments, robots require a comprehensive understanding of their surroundings to interact effectively and intuitively with untrained humans. In this paper, we propose DVEFormer - an efficient RGB-D Transformer-based approach that predicts dense text-aligned visual embeddings (DVE) via knowledge distillation. Instead of directly performing classical semantic segmentation with fixed predefined classes, our method uses teacher embeddings from Alpha-CLIP to guide our efficient student model DVEFormer in learning fine-grained pixel-wise embeddings. While this approach still enables classical semantic segmentation, e.g., via linear probing, it further enables flexible text-based querying and other applications, such as creating comprehensive 3D maps. Evaluations on common indoor datasets demonstrate that our approach achieves competitive performance while meeting real-time requirements, operating at 26.3 FPS for the full model and 77.0 FPS for a smaller variant on an NVIDIA Jetson AGX Orin. Additionally, we show qualitative results that highlight the effectiveness and possible use cases in real-world applications. Overall, our method serves as a drop-in replacement for traditional segmentation approaches while enabling flexible natural-language querying and seamless integration into 3D mapping pipelines for mobile robotics.
Abstract (translated)
在家庭环境中，机器人需要全面了解周围环境，以便与未经培训的人类用户有效且直观地互动。本文提出了DVEFormer——一种基于RGB-D的Transformer方法，通过知识蒸馏预测密集的文字对齐视觉嵌入（DVE）。不同于直接执行具有固定预定义类别划分的传统语义分割任务，我们的方法使用Alpha-CLIP教师模型生成的嵌入来指导学生模型DVEFormer学习精细粒度的像素级嵌入。尽管这种方法仍然支持传统的语义分割任务（例如通过线性探针进行），它还进一步支持基于文本查询和创建全面的3D地图等其他应用。 在常见的室内数据集上的评估表明，我们的方法实现了竞争性的性能并满足了实时需求，在NVIDIA Jetson AGX Orin平台上运行完整模型时帧率为26.3 FPS，较小变体版本为77.0 FPS。此外，我们展示了定性结果以突出该方法的有效性和可能的应用场景。 总体而言，我们的方法可以作为传统分割方法的直接替代品，并且支持灵活的自然语言查询和无缝集成到移动机器人的3D地图构建管道中。
URL
https://arxiv.org/abs/2601.00359
PDF
-
MetaFormer-driven Encoding Network for Robust Medical Semantic Segmentation
2026-01-01 13:45:50Le-Anh Tran, Chung Nguyen Tran, Nhan Cach Dang, Anh Le Van Quoc, Jordi Carrabina, David Castells-Rufas, Minh Son NguyenAbstract
Semantic segmentation is crucial for medical image analysis, enabling precise disease diagnosis and treatment planning. However, many advanced models employ complex architectures, limiting their use in resource-constrained clinical settings. This paper proposes MFEnNet, an efficient medical image segmentation framework that incorporates MetaFormer in the encoding phase of the U-Net backbone. MetaFormer, an architectural abstraction of vision transformers, provides a versatile alternative to convolutional neural networks by transforming tokenized image patches into sequences for global context modeling. To mitigate the substantial computational cost associated with self-attention, the proposed framework replaces conventional transformer modules with pooling transformer blocks, thereby achieving effective global feature aggregation at reduced complexity. In addition, Swish activation is used to achieve smoother gradients and faster convergence, while spatial pyramid pooling is incorporated at the bottleneck to improve multi-scale feature extraction. Comprehensive experiments on different medical segmentation benchmarks demonstrate that the proposed MFEnNet approach attains competitive accuracy while significantly lowering computational cost compared to state-of-the-art models. The source code for this work is available at this https URL.
Abstract (translated)
语义分割在医学图像分析中至关重要，它能够实现精确的疾病诊断和治疗计划制定。然而，许多先进的模型采用了复杂的架构，这限制了它们在资源受限的临床环境中的应用。本文提出了一种名为MFEnNet的高效医疗影像分割框架，该框架将MetaFormer整合到了U-Net骨干网络的编码阶段中。MetaFormer是视觉变换器的一种结构抽象，它提供了一个灵活的选择，能够替代卷积神经网络，在对图像补丁进行标记化处理后形成序列以实现全局上下文建模。为了减轻与自注意力机制相关联的巨大计算成本，所提出的框架用池化变换模块替换了传统的变换模块，从而在降低复杂度的同时实现了有效的全局特征聚合。此外，采用Swish激活函数来获得更平滑的梯度和更快的收敛速度，并且在瓶颈层中融入空间金字塔池以提升多尺度特征提取能力。 通过不同的医学分割基准数据集进行的全面实验表明，提出的MFEnNet方法能够达到与最先进的模型相当的精度，同时大大降低了计算成本。这项工作的源代码可在此[URL]获取。
URL
https://arxiv.org/abs/2601.00922
PDF
-
UniC-Lift: Unified 3D Instance Segmentation via Contrastive Learning
2025-12-31 10:20:01Ankit Dhiman, Srinath R, Jaswanth Reddy, Lokesh R Boregowda, Venkatesh Babu RadhakrishnanAbstract
3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) have advanced novel-view synthesis. Recent methods extend multi-view 2D segmentation to 3D, enabling instance/semantic segmentation for better scene understanding. A key challenge is the inconsistency of 2D instance labels across views, leading to poor 3D predictions. Existing methods use a two-stage approach in which some rely on contrastive learning with hyperparameter-sensitive clustering, while others preprocess labels for consistency. We propose a unified framework that merges these steps, reducing training time and improving performance by introducing a learnable feature embedding for segmentation in Gaussian primitives. This embedding is then efficiently decoded into instance labels through a novel "Embedding-to-Label" process, effectively integrating the optimization. While this unified framework offers substantial benefits, we observed artifacts at the object boundaries. To address the object boundary issues, we propose hard-mining samples along these boundaries. However, directly applying hard mining to the feature embeddings proved unstable. Therefore, we apply a linear layer to the rasterized feature embeddings before calculating the triplet loss, which stabilizes training and significantly improves performance. Our method outperforms baselines qualitatively and quantitatively on the ScanNet, Replica3D, and Messy-Rooms datasets.
Abstract (translated)
3D高斯点置（3D Gaussian Splatting，简称3DGS）和神经辐射场（Neural Radiance Fields，简称NeRF）在新颖视角合成方面取得了进展。最近的方法将多视图2D分割扩展到3D空间中，从而能够进行实例/语义分割，以实现更好的场景理解。一个关键挑战是不同视角之间2D实例标签的一致性问题，这会导致较差的3D预测效果。现有的方法采用两阶段方法，其中一些依赖于敏感于超参数设置的对比学习聚类，而另一些则对标签进行预处理以提高一致性。我们提出了一种统一框架，将这些步骤合并在一起，并通过引入高斯原语中的可学习特征嵌入来实现分割功能，从而减少训练时间并提升性能表现。该嵌入随后通过新颖的“Embedding-to-Label”过程被高效地解码为实例标签，有效集成了优化流程。 尽管这种统一框架带来了显著的好处，但我们观察到在对象边界处存在一些伪影问题。为了应对这些问题，我们提出了一种沿这些边界进行困难样本挖掘的方法。然而，直接将困难采样应用到特征嵌入上显示出不稳定性。因此，在计算三元组损失之前，我们在栅格化后的特征嵌入上添加了一个线性层，这有助于稳定训练过程并显著提升了性能表现。 我们的方法在ScanNet、Replica3D和Messy-Rooms数据集上的定性和定量评估中均优于基准方法。
URL
https://arxiv.org/abs/2512.24763
PDF
-
3D Semantic Segmentation for Post-Disaster Assessment
2025-12-31 03:30:58Nhut Le, Maryam RahnemoonfarAbstract
The increasing frequency of natural disasters poses severe threats to human lives and leads to substantial economic losses. While 3D semantic segmentation is crucial for post-disaster assessment, existing deep learning models lack datasets specifically designed for post-disaster environments. To address this gap, we constructed a specialized 3D dataset using unmanned aerial vehicles (UAVs)-captured aerial footage of Hurricane Ian (2022) over affected areas, employing Structure-from-Motion (SfM) and Multi-View Stereo (MVS) techniques to reconstruct 3D point clouds. We evaluated the state-of-the-art (SOTA) 3D semantic segmentation models, Fast Point Transformer (FPT), Point Transformer v3 (PTv3), and OA-CNNs on this dataset, exposing significant limitations in existing methods for disaster-stricken regions. These findings underscore the urgent need for advancements in 3D segmentation techniques and the development of specialized 3D benchmark datasets to improve post-disaster scene understanding and response.
Abstract (translated)
自然灾害频发的增加给人类生命带来了严重的威胁，并导致了巨大的经济损失。虽然三维语义分割对于灾后的评估至关重要，但现有的深度学习模型缺乏专门为灾后环境设计的数据集。为了填补这一空白，我们利用无人飞行器（UAV）拍摄的2022年飓风伊安过境区域的空中视频构建了一个专门的3D数据集，并采用基于运动的结构重建（Structure-from-Motion, SfM）和多视角立体视觉（Multi-View Stereo, MVS）技术来重构3D点云。我们在这个数据集上评估了最先进的三维语义分割模型，包括Fast Point Transformer (FPT)，Point Transformer v3 (PTv3) 和 OA-CNNs，揭示了现有方法在受灾地区应用中的显著局限性。这些发现强调了改进三维分割技术以及开发专门的三维基准数据集以提高灾后场景理解和响应的紧迫需求。
URL
https://arxiv.org/abs/2512.24593
PDF
-
MambaSeg: Harnessing Mamba for Accurate and Efficient Image-Event Semantic Segmentation
2025-12-30 14:09:17Fuqiang Gu, Yuanke Li, Xianlei Long, Kangping Ji, Chao Chen, Qingyi Gu, Zhenliang NiAbstract
Semantic segmentation is a fundamental task in computer vision with wide-ranging applications, including autonomous driving and robotics. While RGB-based methods have achieved strong performance with CNNs and Transformers, their effectiveness degrades under fast motion, low-light, or high dynamic range conditions due to limitations of frame cameras. Event cameras offer complementary advantages such as high temporal resolution and low latency, yet lack color and texture, making them insufficient on their own. To address this, recent research has explored multimodal fusion of RGB and event data; however, many existing approaches are computationally expensive and focus primarily on spatial fusion, neglecting the temporal dynamics inherent in event streams. In this work, we propose MambaSeg, a novel dual-branch semantic segmentation framework that employs parallel Mamba encoders to efficiently model RGB images and event streams. To reduce cross-modal ambiguity, we introduce the Dual-Dimensional Interaction Module (DDIM), comprising a Cross-Spatial Interaction Module (CSIM) and a Cross-Temporal Interaction Module (CTIM), which jointly perform fine-grained fusion along both spatial and temporal dimensions. This design improves cross-modal alignment, reduces ambiguity, and leverages the complementary properties of each modality. Extensive experiments on the DDD17 and DSEC datasets demonstrate that MambaSeg achieves state-of-the-art segmentation performance while significantly reducing computational cost, showcasing its promise for efficient, scalable, and robust multimodal perception.
Abstract (translated)
语义分割是计算机视觉中的基本任务，具有广泛的应用，包括自动驾驶和机器人技术。基于RGB的方法在使用卷积神经网络（CNN）和变压器的情况下已经取得了强大的性能表现，但在快速运动、低光或高动态范围条件下，由于帧摄像头的局限性，其效果会下降。事件相机提供了诸如高时间分辨率和低延迟等互补优势，但缺乏颜色和纹理信息，在单独使用时是不够的。为了解决这个问题，最近的研究探索了RGB与事件数据的多模态融合；然而，许多现有的方法计算成本高昂，并主要集中在空间融合上，忽视了事件流中的固有时间动态特性。 在本工作中，我们提出了MambaSeg，这是一种新颖的双分支语义分割框架，它使用并行的Mamba编码器来有效地建模RGB图像和事件流。为了减少跨模态模糊性，我们引入了双维交互模块（DDIM），包括一个跨空间交互模块（CSIM）和一个跨时间交互模块（CTIM）。这两个模块共同沿空间和时间维度执行细粒度融合。这种设计提高了跨模式对齐能力，减少了混淆，并利用了每种模态的互补特性。在DDD17和DSEC数据集上进行的一系列实验表明，MambaSeg能够实现最先进的分割性能并显著降低计算成本，展示了其在高效、可扩展且稳健的多模态感知中的潜力。
URL
https://arxiv.org/abs/2512.24243
PDF
-
ARM: A Learnable, Plug-and-Play Module for CLIP-based Open-vocabulary Semantic Segmentation
2025-12-30 13:38:30Ziquan Liu, Zhewei Zhu, Xuyang ShiAbstract
Open-vocabulary semantic segmentation (OVSS) is fundamentally hampered by the coarse, image-level representations of CLIP, which lack precise pixel-level details. Existing training-free methods attempt to resolve this by either importing priors from costly external foundation models (e.g., SAM, DINO) or by applying static, hand-crafted heuristics to CLIP's internal features. These approaches are either computationally expensive or sub-optimal. We propose the Attention Refinement Module (ARM), a lightweight, learnable module that effectively unlocks and refines CLIP's internal potential. Unlike static-fusion methods, ARM learns to adaptively fuse hierarchical features. It employs a semantically-guided cross-attention block, using robust deep features (K, V) to select and refine detail-rich shallow features (Q), followed by a self-attention block. The key innovation lies in a ``train once, use anywhere" paradigm. Trained once on a general-purpose dataset (e.g., COCO-Stuff), ARM acts as a universal plug-and-play post-processor for diverse training-free frameworks. Extensive experiments show that ARM consistently boosts baseline performance on multiple benchmarks with negligible inference overhead, establishing an efficient and effective paradigm for training-free OVSS.
Abstract (translated)
开放词汇语义分割（OVSS）的基本问题是，CLIP的粗略、图像级表示缺乏精确的像素级细节。现有的无训练方法试图通过从昂贵的外部基础模型（如SAM、DINO）导入先验知识或应用静态的手工设计启发式来解决这一问题，以利用CLIP内部特征。这些方法要么计算成本高昂，要么次优。我们提出了注意力精炼模块（ARM），这是一个轻量级的学习模块，能够有效解锁和细化CLIP的内在潜力。 与静态融合方法不同，ARM学习自适应地融合分层特征。它采用语义引导的交叉注意块，利用强大的深度特征（K, V）来选择和丰富细节丰富的浅层特征（Q），随后是自我注意块。关键创新在于“训练一次，随处使用”的理念：在通用数据集（如COCO-Stuff）上训练一次后，ARM作为无训练框架的通用即插即用后期处理模块。 广泛的实验表明，在多个基准测试中，ARM能显著提升基线性能，并且几乎没有推理开销。这为无训练OVSS建立了一个高效有效的范式。
URL
https://arxiv.org/abs/2512.24224
PDF
-
Targeted Semantic Segmentation of Himalayan Glacial Lakes Using Time-Series SAR: Towards Automated GLOF Early Warning
2025-12-30 09:53:24Pawan Adhikari, Satish Raj Regmi, Hari Ram ShresthaAbstract
Glacial Lake Outburst Floods (GLOFs) are one of the most devastating climate change induced hazards. Existing remote monitoring approaches often prioritise maximising spatial coverage to train generalistic models or rely on optical imagery hampered by persistent cloud coverage. This paper presents an end-to-end, automated deep learning pipeline for the targeted monitoring of high-risk Himalayan glacial lakes using time-series Sentinel-1 SAR. We introduce a "temporal-first" training strategy, utilising a U-Net with an EfficientNet-B3 backbone trained on a curated dataset of a cohort of 4 lakes (Tsho Rolpa, Chamlang Tsho, Tilicho and Gokyo Lake). The model achieves an IoU of 0.9130 validating the success and efficacy of the "temporal-first" strategy required for transitioning to Early Warning Systems. Beyond the model, we propose an operational engineering architecture: a Dockerised pipeline that automates data ingestion via the ASF Search API and exposes inference results via a RESTful endpoint. This system shifts the paradigm from static mapping to dynamic and automated early warning, providing a scalable architectural foundation for future development in Early Warning Systems.
Abstract (translated)
冰川湖突发洪水（GLOFs）是气候变化引发的最具破坏性的灾害之一。现有的远程监测方法往往优先考虑最大化空间覆盖范围以训练通用模型，或者依赖于受持续云层遮挡限制的光学图像。本文介绍了一种端到端、自动化的深度学习管道，用于利用时间序列Sentinel-1 SAR对喜马拉雅高危冰川湖进行有针对性的监控。我们引入了“时间优先”的训练策略，使用EfficientNet-B3作为骨干网络的U-Net，在一组4个湖泊（Tsho Rolpa、Chamlang Tsho、Tilicho和Gokyo Lake）的精心策划的数据集上进行了训练。该模型实现了0.9130的IoU值，验证了“时间优先”策略的有效性，这对于向早期预警系统过渡是必要的。 除了模型之外，我们还提出了一种操作工程架构：一个Docker化的管道，通过ASF Search API自动摄取数据，并通过RESTful端点公开推断结果。该系统将范式从静态制图转向动态和自动化早期预警，为未来在早期预警系统的开发中提供可扩展的架构基础。
URL
https://arxiv.org/abs/2512.24117
PDF
-
Bridging Structure and Appearance: Topological Features for Robust Self-Supervised Segmentation
2025-12-30 05:34:28Haotang Li, Zhenyu Qi, Hao Qin, Huanrui Yang, Sen He, Kebin PengAbstract
Self-supervised semantic segmentation methods often fail when faced with appearance ambiguities. We argue that this is due to an over-reliance on unstable, appearance-based features such as shadows, glare, and local textures. We propose \textbf{GASeg}, a novel framework that bridges appearance and geometry by leveraging stable topological information. The core of our method is Differentiable Box-Counting (\textbf{DBC}) module, which quantifies multi-scale topological statistics from two parallel streams: geometric-based features and appearance-based features. To force the model to learn these stable structural representations, we introduce Topological Augmentation (\textbf{TopoAug}), an adversarial strategy that simulates real-world ambiguities by applying morphological operators to the input images. A multi-objective loss, \textbf{GALoss}, then explicitly enforces cross-modal alignment between geometric-based and appearance-based features. Extensive experiments demonstrate that GASeg achieves state-of-the-art performance on four benchmarks, including COCO-Stuff, Cityscapes, and PASCAL, validating our approach of bridging geometry and appearance via topological information.
Abstract (translated)
自监督语义分割方法在面对外观模糊性时往往表现不佳。我们认为这是由于过度依赖诸如阴影、眩光和局部纹理等不稳定的外观特征所致。为此，我们提出了\textbf{GASeg}这一新颖框架，它通过利用稳定拓扑信息来弥合外观与几何之间的差距。我们的方法核心是可微分盒计数（\textbf{DBC}）模块，该模块从两个并行流中量化多尺度的拓扑统计：基于几何特征和基于外观特征。 为了促使模型学习这些稳定的结构表示，我们引入了拓扑增强（\textbf{TopoAug}），这是一种对抗策略，通过将形态学算子应用于输入图像来模拟现实世界的模糊性。随后，一个多目标损失函数 \textbf{GALoss} 明确地强制执行基于几何特征和外观特征之间的跨模态对齐。 大量的实验表明，GASeg 在包括 COCO-Stuff、Cityscapes 和 PASCAL 在内的四个基准数据集上达到了最先进的性能水平，验证了我们通过拓扑信息来连接几何与外观的方法的有效性。
URL
https://arxiv.org/abs/2512.23997
PDF
-
PCR-ORB: Enhanced ORB-SLAM3 with Point Cloud Refinement Using Deep Learning-Based Dynamic Object Filtering
2025-12-29 09:10:31Sheng-Kai Chen, Jie-Yu Chao, Jr-Yu Chang, Po-Lien Wu, Po-Chiang LinAbstract
Visual Simultaneous Localization and Mapping (vSLAM) systems encounter substantial challenges in dynamic environments where moving objects compromise tracking accuracy and map consistency. This paper introduces PCR-ORB (Point Cloud Refinement ORB), an enhanced ORB-SLAM3 framework that integrates deep learning-based point cloud refinement to mitigate dynamic object interference. Our approach employs YOLOv8 for semantic segmentation combined with CUDA-accelerated processing to achieve real-time performance. The system implements a multi-stage filtering strategy encompassing ground plane estimation, sky region removal, edge filtering, and temporal consistency validation. Comprehensive evaluation on the KITTI dataset (sequences 00-09) demonstrates performance characteristics across different environmental conditions and scene types. Notable improvements are observed in specific sequences, with sequence 04 achieving 25.9% improvement in ATE RMSE and 30.4% improvement in ATE median. However, results show mixed performance across sequences, indicating scenario-dependent effectiveness. The implementation provides insights into dynamic object filtering challenges and opportunities for robust navigation in complex environments.
Abstract (translated)
视觉同时定位与地图构建（vSLAM）系统在动态环境中面临重大挑战，因为移动物体会影响跟踪精度和地图一致性。本文介绍了PCR-ORB（点云精炼ORB），这是一种增强版的ORB-SLAM3框架，集成了基于深度学习的点云精炼技术，以减轻动态对象干扰的影响。我们的方法结合使用YOLOv8进行语义分割以及CUDA加速处理来实现实时性能。该系统实施了多级过滤策略，包括地面平面估计、天空区域移除、边缘滤波和时间一致性验证。 在KITTI数据集（序列00-09）上的全面评估显示，在不同环境条件和场景类型下系统的性能特征。特别是在某些序列中观察到了显著改进：序列04的ATE RMSE提高了25.9%，ATE中位数提高了30.4%。然而，结果显示在不同的序列上表现不一，表明其有效性取决于具体的应用场景。 该实现为动态物体过滤挑战提供了见解，并揭示了在复杂环境中进行稳健导航的机会。
URL
https://arxiv.org/abs/2512.23318
PDF
-
AVOID: The Adverse Visual Conditions Dataset with Obstacles for Driving Scene Understanding
2025-12-29 05:34:26Jongoh Jeong, Taek-Jin Song, Jong-Hwan Kim, Kuk-Jin YoonAbstract
Understanding road scenes for visual perception remains crucial for intelligent self-driving cars. In particular, it is desirable to detect unexpected small road hazards reliably in real-time, especially under varying adverse conditions (e.g., weather and daylight). However, existing road driving datasets provide large-scale images acquired in either normal or adverse scenarios only, and often do not contain the road obstacles captured in the same visual domain as for the other classes. To address this, we introduce a new dataset called AVOID, the Adverse Visual Conditions Dataset, for real-time obstacle detection collected in a simulated environment. AVOID consists of a large set of unexpected road obstacles located along each path captured under various weather and time conditions. Each image is coupled with the corresponding semantic and depth maps, raw and semantic LiDAR data, and waypoints, thereby supporting most visual perception tasks. We benchmark the results on high-performing real-time networks for the obstacle detection task, and also propose and conduct ablation studies using a comprehensive multi-task network for semantic segmentation, depth and waypoint prediction tasks.
Abstract (translated)
理解道路场景对于视觉感知来说，仍然是智能自动驾驶汽车的关键。特别是，在各种不利条件下（例如天气和光照）实时可靠地检测意外的小型路面障碍物非常必要。然而，现有的道路驾驶数据集仅提供了正常或恶劣条件下的大规模图像，并且通常不包含与其它类别的同一视觉域内的道路障碍物。为解决这一问题，我们引入了一个新的数据集——AVOID（不利视觉条件数据集），该数据集是在模拟环境中收集的实时障碍物检测数据。AVOID 包含在各种天气和时间条件下捕捉到的大规模意外道路障碍物图像集合。每张图片都配有相应的语义图、深度图、原始和语义激光雷达数据及路径点，从而支持大多数视觉感知任务。我们针对障碍物检测任务对高性能的实时网络进行了基准测试，并提出并开展了使用综合多任务网络进行语义分割、深度预测和路径点预测任务的消融研究。
URL
https://arxiv.org/abs/2512.23215
PDF
-
Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion
2025-12-28 18:24:19Yi Zhou, Xuechao Zou, Shun Zhang, Kai Li, Shiying Wang, Jingming Chen, Congyan Lang, Tengfei Cao, Pin Tao, Yuanchun ShiAbstract
Semi-supervised remote sensing (RS) image semantic segmentation offers a promising solution to alleviate the burden of exhaustive annotation, yet it fundamentally struggles with pseudo-label drift, a phenomenon where confirmation bias leads to the accumulation of errors during training. In this work, we propose Co2S, a stable semi-supervised RS segmentation framework that synergistically fuses priors from vision-language models and self-supervised models. Specifically, we construct a heterogeneous dual-student architecture comprising two distinct ViT-based vision foundation models initialized with pretrained CLIP and DINOv3 to mitigate error accumulation and pseudo-label drift. To effectively incorporate these distinct priors, an explicit-implicit semantic co-guidance mechanism is introduced that utilizes text embeddings and learnable queries to provide explicit and implicit class-level guidance, respectively, thereby jointly enhancing semantic consistency. Furthermore, a global-local feature collaborative fusion strategy is developed to effectively fuse the global contextual information captured by CLIP with the local details produced by DINOv3, enabling the model to generate highly precise segmentation results. Extensive experiments on six popular datasets demonstrate the superiority of the proposed method, which consistently achieves leading performance across various partition protocols and diverse scenarios. Project page is available at this https URL.
Abstract (translated)
半监督遥感（RS）图像语义分割提供了一种有前景的解决方案，以减轻详尽标注的工作负担。然而，它在训练过程中面临着伪标签漂移的问题，这是一种由确认偏差导致错误积累的现象。在这项工作中，我们提出了Co2S，这是一个稳定的半监督RS分割框架，该框架将视觉语言模型和自监督模型中的先验知识进行了协同融合。具体而言，我们构建了一个异构双学生架构，其中包括两个基于ViT的视觉基础模型，并分别用预训练的CLIP和DINOv3进行初始化，以减轻错误积累和伪标签漂移的问题。 为了有效整合这些不同的先验知识，引入了显式隐式语义协同引导机制。该机制利用文本嵌入为明确类别提供指导，同时通过可学习查询提供隐式的类别级指导，从而共同提升语义一致性。 此外，开发了一种全局局部特征协作融合策略，以有效地将CLIP捕获的全球上下文信息与DINOv3产生的本地细节进行融合。这使模型能够生成高度精确的分割结果。 在六个流行数据集上的广泛实验表明了所提出方法的优势，在各种分区协议和多样化场景中均取得了领先性能。项目页面可在提供的链接查看。
URL
https://arxiv.org/abs/2512.23035
PDF
-
Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting
2025-12-28 04:19:25Yiqian Li, Wen Jiang, Kostas DaniilidisAbstract
Understanding semantics and dynamics has been crucial for embodied agents in various tasks. Both tasks have much more data redundancy than the static scene understanding task. We formulate the view selection problem as an active learning problem, where the goal is to prioritize frames that provide the greatest information gain for model training. To this end, we propose an active learning algorithm with Fisher Information that quantifies the informativeness of candidate views with respect to both semantic Gaussian parameters and deformation networks. This formulation allows our method to jointly handle semantic reasoning and dynamic scene modeling, providing a principled alternative to heuristic or random strategies. We evaluate our method on large-scale static images and dynamic video datasets by selecting informative frames from multi-camera setups. Experimental results demonstrate that our approach consistently improves rendering quality and semantic segmentation performance, outperforming baseline methods based on random selection and uncertainty-based heuristics.
Abstract (translated)
理解语义和动态对于各种任务中的具身代理（embodied agents）至关重要。与静态场景理解任务相比，这些任务的数据冗余度更高。我们将视角选择问题形式化为一种主动学习问题，在这种情况下，目标是优先考虑那些能为模型训练提供最大信息增益的帧。为此，我们提出了一种基于Fisher Information的主动学习算法，该算法量化了候选视图在语义高斯参数和变形网络方面的相关信息性。这一方法使我们的技术能够同时处理语义推理和动态场景建模，并且为启发式或随机策略提供了一个有原则性的替代方案。 我们在大规模静态图像和动态视频数据集上通过从多摄像头设置中选择信息丰富的帧来评估我们提出的方法。实验结果表明，与基于随机选择和不确定性启发式的基线方法相比，我们的方法在渲染质量和语义分割性能方面始终表现出色。
URL
https://arxiv.org/abs/2512.22771
PDF
-
iOSPointMapper: RealTime Pedestrian and Accessibility Mapping with Mobile AI
2025-12-26 21:44:08Himanshu Naidu, Yuxiang Zhang, Sachin Mehta, Anat CaspiAbstract
Accurate, up-to-date sidewalk data is essential for building accessible and inclusive pedestrian infrastructure, yet current approaches to data collection are often costly, fragmented, and difficult to scale. We introduce iOSPointMapper, a mobile application that enables real-time, privacy-conscious sidewalk mapping on the ground, using recent-generation iPhones and iPads. The system leverages on-device semantic segmentation, LiDAR-based depth estimation, and fused GPS/IMU data to detect and localize sidewalk-relevant features such as traffic signs, traffic lights and poles. To ensure transparency and improve data quality, iOSPointMapper incorporates a user-guided annotation interface for validating system outputs before submission. Collected data is anonymized and transmitted to the Transportation Data Exchange Initiative (TDEI), where it integrates seamlessly with broader multimodal transportation datasets. Detailed evaluations of the system's feature detection and spatial mapping performance reveal the application's potential for enhanced pedestrian mapping. Together, these capabilities offer a scalable and user-centered approach to closing critical data gaps in pedestrian
Abstract (translated)
准确、最新的人行道数据对于建设无障碍和包容性的行人基础设施至关重要，然而目前的数据收集方法往往成本高昂、碎片化且难以扩展。我们引入了一款名为iOSPointMapper的移动应用，该应用利用最近几代iPhone和iPad设备实现实时、注重隐私的人行道地图绘制功能。系统结合了设备端语义分割技术、基于LiDAR（光探测与测距）的深度估计以及融合GPS/IMU（惯性测量单元）数据来检测并定位人行道相关的特征，如交通标志、信号灯和杆柱等。 为了确保透明度并提高数据质量，iOSPointMapper整合了一个用户引导式的注释界面，在提交前验证系统输出。收集的数据经过匿名化处理，并传输给交通运输数据交换倡议（TDEI），在那里它可以无缝地与更广泛的多模式运输数据集集成在一起。对系统的特征检测和空间映射性能进行详细评估，揭示了该应用在增强行人地图绘制方面的潜力。 这些功能共同提供了一种可扩展且以用户为中心的方法来填补行人基础设施中的关键数据缺口。
URL
https://arxiv.org/abs/2512.22392
PDF
-
Self-supervised Multiplex Consensus Mamba for General Image Fusion
2025-12-24 03:57:21Yingying Wang, Rongjin Zhuang, Hui Zheng, Xuanhua He, Ke Cao, Xiaotong Tu, Xinghao DingAbstract
Image fusion integrates complementary information from different modalities to generate high-quality fused images, thereby enhancing downstream tasks such as object detection and semantic segmentation. Unlike task-specific techniques that primarily focus on consolidating inter-modal information, general image fusion needs to address a wide range of tasks while improving performance without increasing complexity. To achieve this, we propose SMC-Mamba, a Self-supervised Multiplex Consensus Mamba framework for general image fusion. Specifically, the Modality-Agnostic Feature Enhancement (MAFE) module preserves fine details through adaptive gating and enhances global representations via spatial-channel and frequency-rotational scanning. The Multiplex Consensus Cross-modal Mamba (MCCM) module enables dynamic collaboration among experts, reaching a consensus to efficiently integrate complementary information from multiple modalities. The cross-modal scanning within MCCM further strengthens feature interactions across modalities, facilitating seamless integration of critical information from both sources. Additionally, we introduce a Bi-level Self-supervised Contrastive Learning Loss (BSCL), which preserves high-frequency information without increasing computational overhead while simultaneously boosting performance in downstream tasks. Extensive experiments demonstrate that our approach outperforms state-of-the-art (SOTA) image fusion algorithms in tasks such as infrared-visible, medical, multi-focus, and multi-exposure fusion, as well as downstream visual tasks.
Abstract (translated)
图像融合技术通过整合来自不同模态的互补信息，生成高质量的融合图像，从而提升下游任务如目标检测和语义分割的表现。与特定于任务的技术主要关注跨模态信息的集成不同，通用图像融合需要解决广泛的任务需求，在不增加复杂度的前提下提高性能。 为此，我们提出了SMC-Mamba框架，这是一种自监督多路径共识马玛（Self-supervised Multiplex Consensus Mamba）框架，用于通用图像融合。具体来说： - **模态不可知特征增强（Modality-Agnostic Feature Enhancement, MAFE）模块**：通过自适应门控保留精细细节，并使用空间-通道和频率旋转扫描来加强全局表示。 - **多路径共识跨模态马玛（Multiplex Consensus Cross-modal Mamba，MCCM）模块**：允许专家之间的动态协作，达成一致以高效地整合多个模态中的互补信息。MCCM内的跨模态扫描进一步增强了跨模态特征间的交互作用，促进了两种来源关键信息的无缝融合。 此外，我们还引入了双层自监督对比学习损失（Bi-level Self-supervised Contrastive Learning Loss, BSCL），该方法在不增加计算负担的情况下保留高频信息，并同时提高下游任务的表现。 通过广泛的实验表明，我们的方法在红外-可见光、医疗、多焦点和多曝光融合以及下游视觉任务等方面均超越了最先进的图像融合算法。
URL
https://arxiv.org/abs/2512.20921
PDF
