Semantic_Segmentation
-
Learning to Sense for Driving: Joint Optics-Sensor-Model Co-Design for Semantic Segmentation
2025-12-23 22:28:30Reeshad Khan amd John GauchAbstract
Traditional autonomous driving pipelines decouple camera design from downstream perception, relying on fixed optics and handcrafted ISPs that prioritize human viewable imagery rather than machine semantics. This separation discards information during demosaicing, denoising, or quantization, while forcing models to adapt to sensor artifacts. We present a task-driven co-design framework that unifies optics, sensor modeling, and lightweight semantic segmentation networks into a single end-to-end RAW-to-task pipeline. Building on DeepLens[19], our system integrates realistic cellphone-scale lens models, learnable color filter arrays, Poisson-Gaussian noise processes, and quantization, all optimized directly for segmentation objectives. Evaluations on KITTI-360 show consistent mIoU improvements over fixed pipelines, with optics modeling and CFA learning providing the largest gains, especially for thin or low-light-sensitive classes. Importantly, these robustness gains are achieved with a compact ~1M-parameter model running at ~28 FPS, demonstrating edge deployability. Visual and quantitative analyses further highlight how co-designed sensors adapt acquisition to semantic structure, sharpening boundaries and maintaining accuracy under blur, noise, and low bit-depth. Together, these findings establish full-stack co-optimization of optics, sensors, and networks as a principled path toward efficient, reliable, and deployable perception in autonomous systems.
Abstract (translated)
传统的自动驾驶管道将相机设计与下游感知模块分离，依赖于固定的光学系统和优先考虑人类可视性而非机器语义的手动ISP。这种隔离在去马赛克化、降噪或量化过程中会丢失信息，并且迫使模型适应传感器的特性。我们提出了一种任务驱动的协同设计框架，将光学、传感器建模与轻量级语义分割网络统一为一个从原始数据（RAW）到特定任务的整体端到端管道。基于DeepLens[19]系统，我们的方法集成了真实的手机级别镜头模型、可学习的颜色滤波阵列、泊松-高斯噪声过程以及量化技术，并且所有这些都直接针对语义分割的目标进行优化。 在KITTI-360数据集上的评估表明，与固定管道相比，我们的模型在平均交并比（mIoU）方面持续取得改进。光学建模和颜色滤波阵列学习对这种改善贡献最大，尤其是在处理薄或低光敏感类别的场景时效果更加明显。 尤为重要的是，这些性能的提升是通过一个大约100万参数的小型模型实现的，该模型能够以约28帧每秒的速度运行，展示了其在边缘设备部署中的可行性。从视觉和量化分析中进一步可以看出，协同设计传感器是如何根据语义结构调整采集过程的，从而提高边界清晰度，并且即使在模糊、噪声或低位深度的情况下也能保持准确性。 这些研究结果共同证明了光学系统、传感器与网络之间的全栈协同优化是一种有效、可靠并可部署于自动驾驶系统的合理路径。
URL
https://arxiv.org/abs/2512.20815
PDF
-
BiCoR-Seg: Bidirectional Co-Refinement Framework for High-Resolution Remote Sensing Image Segmentation
2025-12-23 11:13:01Jinghao Shi, Jianing SongAbstract
High-resolution remote sensing image semantic segmentation (HRSS) is a fundamental yet critical task in the field of Earth observation. However, it has long faced the challenges of high inter-class similarity and large intra-class variability. Existing approaches often struggle to effectively inject abstract yet strongly discriminative semantic knowledge into pixel-level feature learning, leading to blurred boundaries and class confusion in complex scenes. To address these challenges, we propose Bidirectional Co-Refinement Framework for HRSS (BiCoR-Seg). Specifically, we design a Heatmap-driven Bidirectional Information Synergy Module (HBIS), which establishes a bidirectional information flow between feature maps and class embeddings by generating class-level heatmaps. Based on HBIS, we further introduce a hierarchical supervision strategy, where the interpretable heatmaps generated by each HBIS module are directly utilized as low-resolution segmentation predictions for supervision, thereby enhancing the discriminative capacity of shallow features. In addition, to further improve the discriminability of the embedding representations, we propose a cross-layer class embedding Fisher Discriminative Loss to enforce intra-class compactness and enlarge inter-class separability. Extensive experiments on the LoveDA, Vaihingen, and Potsdam datasets demonstrate that BiCoR-Seg achieves outstanding segmentation performance while offering stronger interpretability. The released code is available at this https URL.
Abstract (translated)
高分辨率遥感图像语义分割（HRSS）是地球观测领域的一项基本而关键的任务。然而，长期以来它一直面临着类间相似度高和类内变异性大的挑战。现有方法往往难以有效地将抽象但具有强区分性的语义知识注入到像素级特征学习中，导致复杂场景中的边界模糊和类别混淆问题。 为了应对这些挑战，我们提出了一种双向协同精炼框架（BiCoR-Seg）。具体而言，我们设计了热图驱动的双向信息协同模块（HBIS），该模块通过生成类级别热图在特征图与类别嵌入之间建立了双向的信息流。基于HBIS，我们进一步引入了一个分层监督策略，在此策略中，每个HBIS模块产生的可解释热图直接作为低分辨率分割预测用于监督，从而增强了浅层特征的区分能力。 此外，为了进一步提高嵌入表示的区分性，我们提出了一种跨层类别嵌入费舍尔判别损失（Cross-layer Class Embedding Fisher Discriminative Loss），以强制类内紧致性和扩大类间可分性。在LoveDA、Vaihingen和Potsdam数据集上的广泛实验表明，BiCoR-Seg实现了卓越的分割性能，并提供了更强的解释能力。 相关代码已公开发布在此链接中（https://...）供参考使用。
URL
https://arxiv.org/abs/2512.20255
PDF
-
Uncertainty-Gated Region-Level Retrieval for Robust Semantic Segmentation
2025-12-19 21:39:22Shreshth Rajan, Raymond LiuAbstract
Semantic segmentation of outdoor street scenes plays a key role in applications such as autonomous driving, mobile robotics, and assistive technology for visually-impaired pedestrians. For these applications, accurately distinguishing between key surfaces and objects such as roads, sidewalks, vehicles, and pedestrians is essential for maintaining safety and minimizing risks. Semantic segmentation must be robust to different environments, lighting and weather conditions, and sensor noise, while being performed in real-time. We propose a region-level, uncertainty-gated retrieval mechanism that improves segmentation accuracy and calibration under domain shift. Our best method achieves an 11.3% increase in mean intersection-over-union while reducing retrieval cost by 87.5%, retrieving for only 12.5% of regions compared to 100% for always-on baseline.
Abstract (translated)
户外街道场景的语义分割在自动驾驶、移动机器人和视觉障碍行人辅助技术等应用中起着关键作用。对于这些应用而言，准确地区分道路、人行道、车辆和行人等关键表面和物体是确保安全性和减少风险所必需的。为了适应不同的环境条件、光照和天气状况以及传感器噪声，并实现实时处理，语义分割必须具备鲁棒性。我们提出了一种基于区域级别的不确定性门控检索机制，该机制在领域偏移（domain shift）情况下提高了分割精度并改善了校准性能。我们的最佳方法实现了11.3%的平均交并比（mean intersection-over-union, mIoU）提升，并将检索成本降低了87.5%，仅对12.5%的区域进行检索，而基线系统则为100%全时段运行。
URL
https://arxiv.org/abs/2512.18082
PDF
-
SAVeD: A First-Person Social Media Video Dataset for ADAS-equipped vehicle Near-Miss and Crash Event Analyses
2025-12-19 15:58:52Shaoyan Zhai, Mohamed Abdel-Aty, Chenzhu Wang, Rodrigo Vena GarciaAbstract
The advancement of safety-critical research in driving behavior in ADAS-equipped vehicles require real-world datasets that not only include diverse traffic scenarios but also capture high-risk edge cases such as near-miss events and system failures. However, existing datasets are largely limited to either simulated environments or human-driven vehicle data, lacking authentic ADAS (Advanced Driver Assistance System) vehicle behavior under risk conditions. To address this gap, this paper introduces SAVeD, a large-scale video dataset curated from publicly available social media content, explicitly focused on ADAS vehicle-related crashes, near-miss incidents, and disengagements. SAVeD features 2,119 first-person videos, capturing ADAS vehicle operations in diverse locations, lighting conditions, and weather scenarios. The dataset includes video frame-level annotations for collisions, evasive maneuvers, and disengagements, enabling analysis of both perception and decision-making failures. We demonstrate SAVeD's utility through multiple analyses and contributions: (1) We propose a novel framework integrating semantic segmentation and monocular depth estimation to compute real-time Time-to-Collision (TTC) for dynamic objects. (2) We utilize the Generalized Extreme Value (GEV) distribution to model and quantify the extreme risk in crash and near-miss events across different roadway types. (3) We establish benchmarks for state-of-the-art VLLMs (VideoLLaMA2 and InternVL2.5 HiCo R16), showing that SAVeD's detailed annotations significantly enhance model performance through domain adaptation in complex near-miss scenarios.
Abstract (translated)
在配备了高级驾驶辅助系统（ADAS）的车辆中，推进安全关键性的驾驶行为研究需要包含多样化的交通场景以及捕捉高风险边缘情况的真实世界数据集，例如险情和系统故障。然而，现有的数据集主要局限于模拟环境或人类驾驶员的数据，缺乏在危险条件下真实ADAS车辆行为的信息。为了填补这一空白，本文介绍了SAVED，这是一个大规模的视频数据集，从公开可用的社会媒体内容中精选而来，重点关注与ADAS车辆相关的碰撞、险情和系统退出情况。 SAVED包含2,119个第一视角视频，涵盖了不同地点、光照条件和天气场景下的ADAS车辆操作。该数据集包括了针对碰撞、规避动作以及系统退出的逐帧注释，支持对感知和决策失败的分析。我们通过多种分析和贡献展示了SAVED的实用性： (1) 我们提出了一种新的框架，将语义分割与单目深度估计相结合，用于计算动态物体的实际时间到碰撞（TTC）。 (2) 我们使用广义极值分布来建模并量化不同道路类型中的严重风险在碰撞和险情事件中的发生情况。 (3) 对于当前最先进的视频语言模型VLLMs（VideoLLaMA2 和 InternVL2.5 HiCo R16），我们建立了基准，表明通过复杂险境场景下的领域适应性，SAVED的详细注释显著提升了这些模型的表现。
URL
https://arxiv.org/abs/2512.17724
PDF
-
MULTIAQUA: A multimodal maritime dataset and robust training strategies for multimodal semantic segmentation
2025-12-19 11:06:46Jon Muhovi\v{c}, Janez Per\v{s}Abstract
Unmanned surface vehicles can encounter a number of varied visual circumstances during operation, some of which can be very difficult to interpret. While most cases can be solved only using color camera images, some weather and lighting conditions require additional information. To expand the available maritime data, we present a novel multimodal maritime dataset MULTIAQUA (Multimodal Aquatic Dataset). Our dataset contains synchronized, calibrated and annotated data captured by sensors of different modalities, such as RGB, thermal, IR, LIDAR, etc. The dataset is aimed at developing supervised methods that can extract useful information from these modalities in order to provide a high quality of scene interpretation regardless of potentially poor visibility conditions. To illustrate the benefits of the proposed dataset, we evaluate several multimodal methods on our difficult nighttime test set. We present training approaches that enable multimodal methods to be trained in a more robust way, thus enabling them to retain reliable performance even in near-complete darkness. Our approach allows for training a robust deep neural network only using daytime images, thus significantly simplifying data acquisition, annotation, and the training process.
Abstract (translated)
无人水面车辆在运行过程中会遇到多种多样的视觉情况，其中一些情况下很难进行解读。虽然大多数案例仅使用彩色相机图像就可以解决，但某些天气和光照条件则需要额外的信息。为了扩充可用的海洋数据，我们提出了一种新的多模态海事数据集MULTIAQUA（多模态水下数据集）。我们的数据集中包含了由不同类型的传感器同步捕获、校准并标注的数据，如RGB、热成像、红外、激光雷达等。该数据集旨在开发能够从这些模式中提取有用信息的监督方法，从而在可能存在的低能见度条件下也能提供高质量的场景解读。 为了展示所提出数据集的优势，我们评估了几种多模态方法在我们的夜间测试集中难以处理的情况下的表现。我们还提出了训练方法，使多模态方法能够以更稳健的方式进行训练，从而使它们即使在近乎完全黑暗的情况下也能保持可靠的性能。我们的方法仅通过使用白天的图像就可以训练出一个鲁棒的深度神经网络，从而大大简化了数据采集、标注和训练过程。
URL
https://arxiv.org/abs/2512.17450
PDF
-
AIFloodSense: A Global Aerial Imagery Dataset for Semantic Segmentation and Understanding of Flooded Environments
2025-12-19 10:34:45Georgios Simantiris, Konstantinos Bacharidis, Apostolos Papanikolaou, Petros Giannakakis, Costas PanagiotakisAbstract
Accurate flood detection from visual data is a critical step toward improving disaster response and risk assessment, yet datasets for flood segmentation remain scarce due to the challenges of collecting and annotating large-scale imagery. Existing resources are often limited in geographic scope and annotation detail, hindering the development of robust, generalized computer vision methods. To bridge this gap, we introduce AIFloodSense, a comprehensive, publicly available aerial imagery dataset comprising 470 high-resolution images from 230 distinct flood events across 64 countries and six continents. Unlike prior benchmarks, AIFloodSense ensures global diversity and temporal relevance (2022-2024), supporting three complementary tasks: (i) Image Classification with novel sub-tasks for environment type, camera angle, and continent recognition; (ii) Semantic Segmentation providing precise pixel-level masks for flood, sky, and buildings; and (iii) Visual Question Answering (VQA) to enable natural language reasoning for disaster assessment. We establish baseline benchmarks for all tasks using state-of-the-art architectures, demonstrating the dataset's complexity and its value in advancing domain-generalized AI tools for climate resilience.
Abstract (translated)
从视觉数据中准确检测洪水是提高灾害应对和风险评估的关键步骤，然而由于收集和标注大规模图像的困难，用于洪水分割的数据集仍然稀缺。现有资源通常在地理范围和注释细节上受到限制，阻碍了稳健且通用的计算机视觉方法的发展。为了弥合这一差距，我们引入了AIFloodSense，这是一个全面、公开可用的航空影像数据集，包含来自64个国家和地区（六大洲）230次不同洪水事件的470张高分辨率图像。与之前的基准相比，AIFloodSense确保了全球多样性和时间相关性（2022-2024年），支持三项互补任务：(i) 图像分类，包括环境类型、拍摄角度和大陆识别的新子任务；(ii) 语义分割，提供洪水、天空和建筑物的精确像素级掩码；以及(iii) 视觉问答（VQA），以实现灾难评估中的自然语言推理。我们使用最先进的架构为所有任务建立了基准，并展示了该数据集的复杂性及其在推动气候适应性领域泛化人工智能工具发展方面的价值。
URL
https://arxiv.org/abs/2512.17432
PDF
-
Next-Embedding Prediction Makes Strong Vision Learners
2025-12-18 18:59:58Sihan Xu, Ziqiao Ma, Wenhao Chai, Xuweiyi Chen, Weiyang Jin, Joyce Chai, Saining Xie, Stella X. YuAbstract
Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.
Abstract (translated)
受自然语言生成预训练成功的启发，我们探讨了同样的原则是否可以培养出强大的自监督视觉学习者。与其训练模型输出用于下游任务的特征，我们将它们训练成能够直接执行预测性任务的嵌入生成器。这项工作探索了一种从学习表示转向学习模型的方法。具体而言，模型学会了根据过去的嵌入来预测未来的补丁(embedding)信息，使用因果屏蔽和停止梯度技术，我们称之为“Next-Embedding 预测自回归”(NEPA)。 研究表明，一个仅通过预测下一个嵌入作为其唯一训练目标的简单Transformer，在ImageNet-1k上进行预训练是有效的——这种方法不需要像素重建、离散令牌(token)、对比损失或特定任务的头部设计。这种形式保持了架构上的简洁性和可扩展性，并且无需额外的设计复杂度。 NEPA在多项任务中取得了优异的成绩，例如使用ViT-B和ViT-L基础模型对ImageNet-1K进行微调后分别达到了83.8%和85.3%的top-1准确率；此外，在ADE20K数据集上的语义分割转移学习也表现良好。我们认为从嵌入生成的预训练提供了一种简单、可扩展且可能跨模态(即适用于多种类型的数据输入)的方法，来替代现有的视觉自监督学习方法。
URL
https://arxiv.org/abs/2512.16922
PDF
-
Task-Oriented Data Synthesis and Control-Rectify Sampling for Remote Sensing Semantic Segmentation
2025-12-18 16:37:39Yunkai Yang, Yudong Zhang, Kunquan Zhang, Jinxiao Zhang, Xinying Chen, Haohuan Fu, Runmin DongAbstract
With the rapid progress of controllable generation, training data synthesis has become a promising way to expand labeled datasets and alleviate manual annotation in remote sensing (RS). However, the complexity of semantic mask control and the uncertainty of sampling quality often limit the utility of synthetic data in downstream semantic segmentation tasks. To address these challenges, we propose a task-oriented data synthesis framework (TODSynth), including a Multimodal Diffusion Transformer (MM-DiT) with unified triple attention and a plug-and-play sampling strategy guided by task feedback. Built upon the powerful DiT-based generative foundation model, we systematically evaluate different control schemes, showing that a text-image-mask joint attention scheme combined with full fine-tuning of the image and mask branches significantly enhances the effectiveness of RS semantic segmentation data synthesis, particularly in few-shot and complex-scene scenarios. Furthermore, we propose a control-rectify flow matching (CRFM) method, which dynamically adjusts sampling directions guided by semantic loss during the early high-plasticity stage, mitigating the instability of generated images and bridging the gap between synthetic data and downstream segmentation tasks. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art controllable generation methods, producing more stable and task-oriented synthetic data for RS semantic segmentation.
Abstract (translated)
随着可控生成技术的迅速发展，训练数据合成已成为扩大标注数据集和减轻远程 sensing（RS）中人工注释负担的一种有前景的方法。然而，语义掩码控制的复杂性和采样质量的不确定性往往限制了合成数据在下游语义分割任务中的实用性。为了解决这些挑战，我们提出了一种任务导向的数据合成框架（TODSynth），该框架包括一个具有统一三元注意力机制的多模态扩散变换器（MM-DiT）和一种由任务反馈引导的即插即用采样策略。 基于强大的DiT基础生成模型，我们系统地评估了不同的控制方案，发现结合文本-图像-掩码联合注意机制以及对图像和掩码分支进行完全微调的方案显著提高了RS语义分割数据合成的有效性，特别是在少量样本和复杂场景的情况下尤为明显。此外，我们提出了一种基于语义损失引导动态调整采样方向的方法——控制校正流匹配（CRFM），这种方法在生成图像早期高可塑性的阶段缓解了生成图像的不稳定性，并且缩小了合成数据与下游分割任务之间的差距。 广泛的实验表明，我们的方法持续优于现有的可控生成技术，在产生更为稳定和面向任务的RS语义分割合成数据方面表现出色。
URL
https://arxiv.org/abs/2512.16740
PDF
-
Causal-Tune: Mining Causal Factors from Vision Foundation Models for Domain Generalized Semantic Segmentation
2025-12-18 14:10:46Yin Zhang, Yongqiang Zhang, Yaoyue Zheng, Bogdan Raducanu, Dan LiuAbstract
Fine-tuning Vision Foundation Models (VFMs) with a small number of parameters has shown remarkable performance in Domain Generalized Semantic Segmentation (DGSS). Most existing works either train lightweight adapters or refine intermediate features to achieve better generalization on unseen domains. However, they both overlook the fact that long-term pre-trained VFMs often exhibit artifacts, which hinder the utilization of valuable representations and ultimately degrade DGSS performance. Inspired by causal mechanisms, we observe that these artifacts are associated with non-causal factors, which usually reside in the low- and high-frequency components of the VFM spectrum. In this paper, we explicitly examine the causal and non-causal factors of features within VFMs for DGSS, and propose a simple yet effective method to identify and disentangle them, enabling more robust domain generalization. Specifically, we propose Causal-Tune, a novel fine-tuning strategy designed to extract causal factors and suppress non-causal ones from the features of VFMs. First, we extract the frequency spectrum of features from each layer using the Discrete Cosine Transform (DCT). A Gaussian band-pass filter is then applied to separate the spectrum into causal and non-causal components. To further refine the causal components, we introduce a set of causal-aware learnable tokens that operate in the frequency domain, while the non-causal components are discarded. Finally, refined features are transformed back into the spatial domain via inverse DCT and passed to the next layer. Extensive experiments conducted on various cross-domain tasks demonstrate the effectiveness of Causal-Tune. In particular, our method achieves superior performance under adverse weather conditions, improving +4.8% mIoU over the baseline in snow conditions.
Abstract (translated)
使用少量参数对视觉基础模型（VFMs）进行微调，在领域泛化语义分割（DGSS）中表现出卓越的性能。现有的大多数工作要么训练轻量级适配器，要么改进中间特征以实现在未见领域的更好泛化能力。然而，它们都忽视了这样一个事实：长期预训练的VFMs往往展现出一些阻碍有价值表征利用的艺术瑕疵，最终导致DGSS性能下降。受因果机制启发，我们观察到这些艺术瑕疵通常与非因果因素相关联，而后者通常存在于VFM频谱中的低频和高频部分中。 在本文中，我们明确地考察了用于DGSS的VFMs特征中的因果和非因果因素，并提出了一种简单但有效的方法来识别并分离它们，从而实现更稳健的领域泛化。具体来说，我们提出了Causal-Tune，这是一种新的微调策略，旨在从VFMs特征中提取因果因素并抑制非因果因素。首先，我们使用离散余弦变换（DCT）从每一层中提取频谱特性。然后应用高斯带通滤波器将频谱分为因果和非因果部分。为了进一步精炼因果部分，我们在频域中引入了一组具有因果感知的可学习令牌，而忽略非因果部分。最后，通过逆DCT将细化后的特征转换回空间域，并传递给下一层。 在各种跨领域任务上的广泛实验表明了Causal-Tune的有效性。特别是，在恶劣天气条件下，我们的方法比基线改进了+4.8%的mIoU（雪条件下的表现）。
URL
https://arxiv.org/abs/2512.16567
PDF
-
PixelArena: A benchmark for Pixel-Precision Visual Intelligence
2025-12-18 08:41:27Feng Liang, Sizhe Cheng, Chenqi YiAbstract
Multi-modal large language models that have image output are emerging. Many image generation benchmarks focus on aesthetics instead of fine-grained generation capabilities. In PixelArena, we propose using semantic segmentation tasks to objectively examine their fine-grained generative intelligence with pixel precision. We find the latest Gemini 3 Pro Image has emergent image generation capabilities that generate semantic masks with high fidelity under zero-shot settings, showcasing visual intelligence unseen before and true generalization in new image generation tasks. We further investigate its results, compare them qualitatively and quantitatively with those of other models, and present failure cases. The findings not only signal exciting progress in the field but also provide insights into future research related to multimodality, reasoning, interpretability and benchmarking.
Abstract (translated)
多模态大型语言模型（能够生成图像）正在兴起。许多图像生成基准测试主要关注美学效果，而忽略了对精细生成能力的考察。在PixelArena中，我们提出使用语义分割任务来客观地评估这些模型的细粒度生成智能，并以像素级别的精度进行衡量。我们发现最新的Gemini 3 Pro Image模型具备了零样本设置下的图像生成能力，在这种情况下，它能够产生高保真的语义掩码，展示出前所未有的视觉智能和在新图像生成任务中的真实泛化能力。 进一步地，我们对其结果进行了深入研究，并从定性和定量两个角度与其他模型的结果进行比较。此外，还展示了该模型的失败案例。这些发现不仅标志着领域内的令人兴奋的进步，而且为与多模态、推理、可解释性以及基准测试相关的未来研究提供了宝贵的见解。
URL
https://arxiv.org/abs/2512.16303
PDF
-
In Pursuit of Pixel Supervision for Visual Pre-training
2025-12-17 18:59:58Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Hengshuang Zhao, Hu XuAbstract
At the most basic level, pixels are the source of the visual information through which we perceive the world. Pixels contain information at all levels, ranging from low-level attributes to high-level concepts. Autoencoders represent a classical and long-standing paradigm for learning representations from pixels or other raw inputs. In this work, we demonstrate that autoencoder-based self-supervised learning remains competitive today and can produce strong representations for downstream tasks, while remaining simple, stable, and efficient. Our model, codenamed "Pixio", is an enhanced masked autoencoder (MAE) with more challenging pre-training tasks and more capable architectures. The model is trained on 2B web-crawled images with a self-curation strategy with minimal human curation. Pixio performs competitively across a wide range of downstream tasks in the wild, including monocular depth estimation (e.g., Depth Anything), feed-forward 3D reconstruction (i.e., MapAnything), semantic segmentation, and robot learning, outperforming or matching DINOv3 trained at similar scales. Our results suggest that pixel-space self-supervised learning can serve as a promising alternative and a complement to latent-space approaches.
Abstract (translated)
在最基本的层面上，像素是承载我们感知世界的视觉信息的来源。每个像素包含了从低级属性到高级概念的各种信息层次。自编码器是一种经典的、长期存在的范式，用于从像素或其它原始输入中学习表示方法。在这项工作中，我们展示了基于自编码器的自我监督学习在今天依然具有竞争力，并能为下游任务生成强大的表示方法，同时保持简单性、稳定性和效率。我们的模型名为“Pixio”，是一个增强版的掩码自动编码器（MAE），配备了更困难的预训练任务和更有能力的架构。该模型是在使用最小人工管理策略进行自我筛选的20亿张网络抓取图像上进行训练的。在各种下游任务中，包括单眼深度估计、前馈3D重建、语义分割以及机器人学习等方面，Pixio表现出了竞争力，并且在类似规模下训练的DINOv3模型上超过了或与之持平的表现。我们的结果表明，在像素空间中的自我监督学习可以作为潜在空间方法的一个有前途的替代和补充方案。
URL
https://arxiv.org/abs/2512.15715
PDF
-
SemanticBridge -- A Dataset for 3D Semantic Segmentation of Bridges and Domain Gap Analysis
2025-12-17 12:17:11Maximilian Kellner, Mariana Ferrandon Cervantes, Yuandong Pan, Ruodan Lu, Ioannis Brilakis, Alexander ReitererAbstract
We propose a novel dataset that has been specifically designed for 3D semantic segmentation of bridges and the domain gap analysis caused by varying sensors. This addresses a critical need in the field of infrastructure inspection and maintenance, which is essential for modern society. The dataset comprises high-resolution 3D scans of a diverse range of bridge structures from various countries, with detailed semantic labels provided for each. Our initial objective is to facilitate accurate and automated segmentation of bridge components, thereby advancing the structural health monitoring practice. To evaluate the effectiveness of existing 3D deep learning models on this novel dataset, we conduct a comprehensive analysis of three distinct state-of-the-art architectures. Furthermore, we present data acquired through diverse sensors to quantify the domain gap resulting from sensor variations. Our findings indicate that all architectures demonstrate robust performance on the specified task. However, the domain gap can potentially lead to a decline in the performance of up to 11.4% mIoU.
Abstract (translated)
我们提出了一套专门为桥梁的三维语义分割和由不同传感器引起的域差距分析而设计的新数据集。这满足了基础设施检测与维护领域的一个关键需求，这对于现代社会至关重要。该数据集包含来自多个国家各种不同类型桥梁的高分辨率3D扫描图像，并为每一种提供了详细的语义标签。我们的初步目标是促进桥梁组件的精确和自动化分割，从而推进结构健康监测实践的发展。 为了评估现有三维深度学习模型在这套新数据集上的有效性，我们对三种不同的最先进的架构进行了全面分析。此外，我们还通过不同类型的传感器获取了数据，以量化由于传感器变化引起的域差距。我们的研究结果表明，所有架构在指定任务上都表现出稳健的性能，但域差距可能导致高达11.4% mIoU（平均交并比）的性能下降。
URL
https://arxiv.org/abs/2512.15369
PDF
-
SynthSeg-Agents: Multi-Agent Synthetic Data Generation for Zero-Shot Weakly Supervised Semantic Segmentation
2025-12-17 10:58:38Wangyu Wu, Zhenhong Chen, Xiaowei Huang, Fei Ma, Jimin XiaoAbstract
Weakly Supervised Semantic Segmentation (WSSS) with image level labels aims to produce pixel level predictions without requiring dense annotations. While recent approaches have leveraged generative models to augment existing data, they remain dependent on real world training samples. In this paper, we introduce a novel direction, Zero Shot Weakly Supervised Semantic Segmentation (ZSWSSS), and propose SynthSeg Agents, a multi agent framework driven by Large Language Models (LLMs) to generate synthetic training data entirely without real images. SynthSeg Agents comprises two key modules, a Self Refine Prompt Agent and an Image Generation Agent. The Self Refine Prompt Agent autonomously crafts diverse and semantically rich image prompts via iterative refinement, memory mechanisms, and prompt space exploration, guided by CLIP based similarity and nearest neighbor diversity filtering. These prompts are then passed to the Image Generation Agent, which leverages Vision Language Models (VLMs) to synthesize candidate images. A frozen CLIP scoring model is employed to select high quality samples, and a ViT based classifier is further trained to relabel the entire synthetic dataset with improved semantic precision. Our framework produces high quality training data without any real image supervision. Experiments on PASCAL VOC 2012 and COCO 2014 show that SynthSeg Agents achieves competitive performance without using real training images. This highlights the potential of LLM driven agents in enabling cost efficient and scalable semantic segmentation.
Abstract (translated)
弱监督语义分割（WSSS）使用图像级标签的目标是无需密集注释就能生成像素级别的预测。尽管最近的研究已经利用了生成模型来增强现有数据，但它们仍然依赖于真实世界的训练样本。在本文中，我们提出了一种新颖的方向——零样本弱监督语义分割（ZSWSSS），并提出了SynthSeg Agents，这是一个由大型语言模型（LLMs）驱动的多代理框架，用于完全不使用真实图像生成合成训练数据。SynthSeg Agents 包括两个关键模块：自我优化提示代理和图像生成代理。 **自我优化提示代理**自主地通过迭代细化、记忆机制以及基于CLIP相似性和最近邻多样性过滤的提示空间探索来创建多样且语义丰富的图像提示。 这些提示随后传递给**图像生成代理**，该代理利用视觉语言模型（VLMs）合成候选图像。一个冻结的CLIP评分模型被用来选择高质量样本，并通过训练一个基于ViT的分类器进一步为整个合成数据集重新标记，以提高语义精度。 我们的框架能够在没有任何真实图像监督的情况下产生高质量的训练数据。在PASCAL VOC 2012和COCO 2014上的实验表明，SynthSeg Agents 在不使用任何实际训练图像的情况下也达到了竞争性的性能水平。这突显了LLM驱动代理在实现低成本且可扩展语义分割中的潜力。
URL
https://arxiv.org/abs/2512.15310
PDF
-
Segmental Attention Decoding With Long Form Acoustic Encodings
2025-12-16 18:12:37Pawel Swietojanski, Xinwei Li, Mingbin Xu, Takaaki Hori, Dogan Can, Xiaodan ZhuangAbstract
We address the fundamental incompatibility of attention-based encoder-decoder (AED) models with long-form acoustic encodings. AED models trained on segmented utterances learn to encode absolute frame positions by exploiting limited acoustic context beyond segment boundaries, but fail to generalize when decoding long-form segments where these cues vanish. The model loses ability to order acoustic encodings due to permutation invariance of keys and values in cross-attention. We propose four modifications: (1) injecting explicit absolute positional encodings into cross-attention for each decoded segment, (2) long-form training with extended acoustic context to eliminate implicit absolute position encoding, (3) segment concatenation to cover diverse segmentations needed during training, and (4) semantic segmentation to align AED-decoded segments with training segments. We show these modifications close the accuracy gap between continuous and segmented acoustic encodings, enabling auto-regressive use of the attention decoder.
Abstract (translated)
我们解决了基于注意力的编码器-解码器（AED）模型与长形式声学编码不兼容的基本问题。当在分割的话语上训练时，AED 模型会通过利用跨越段边界的一小部分声学上下文来学习对绝对帧位置进行编码，但在解码长形式片段时却无法泛化，因为在这些情况下这些线索消失。由于跨注意力中键和值的排列不变性，模型失去了按顺序排列声学编码的能力。我们提出了四项改进措施：(1) 在每次解码段的跨注意力中注入明确的绝对位置编码；(2) 通过使用扩展的声学上下文进行长形式训练来消除隐式的绝对位置编码；(3) 段连接以覆盖在训练过程中所需的各种分割情况；以及 (4) 语义分割，将 AED 解码片段与训练段对齐。我们展示了这些改进措施可以缩小连续和分段声学编码之间的准确度差距，使注意力解码器能够进行自回归使用。
URL
https://arxiv.org/abs/2512.14652
PDF
-
DriverGaze360: OmniDirectional Driver Attention with Object-Level Guidance
2025-12-16 10:23:00Shreedhar Govil, Didier Stricker, Jason RambachAbstract
Predicting driver attention is a critical problem for developing explainable autonomous driving systems and understanding driver behavior in mixed human-autonomous vehicle traffic scenarios. Although significant progress has been made through large-scale driver attention datasets and deep learning architectures, existing works are constrained by narrow frontal field-of-view and limited driving diversity. Consequently, they fail to capture the full spatial context of driving environments, especially during lane changes, turns, and interactions involving peripheral objects such as pedestrians or cyclists. In this paper, we introduce DriverGaze360, a large-scale 360$^\circ$ field of view driver attention dataset, containing $\sim$1 million gaze-labeled frames collected from 19 human drivers, enabling comprehensive omnidirectional modeling of driver gaze behavior. Moreover, our panoramic attention prediction approach, DriverGaze360-Net, jointly learns attention maps and attended objects by employing an auxiliary semantic segmentation head. This improves spatial awareness and attention prediction across wide panoramic inputs. Extensive experiments demonstrate that DriverGaze360-Net achieves state-of-the-art attention prediction performance on multiple metrics on panoramic driving images. Dataset and method available at this https URL.
Abstract (translated)
预测驾驶员的注意力是开发可解释的自动驾驶系统和理解混合有人类驾驶与自主车辆交通场景中驾驶员行为的关键问题。尽管通过大规模驾驶员注意数据集和深度学习架构取得了显著进展，但现有的研究工作受到狭窄前方视野和有限驾驶多样性的限制，从而无法捕捉到完整的驾驶环境空间背景，特别是在变道、转弯以及涉及行人或骑自行车者等周边物体互动时。 在本文中，我们介绍了DriverGaze360，这是一个大规模的360°视场驾驶员注意力数据集，包含由19名人类驾驶员收集的大约100万张带有注视标签的图像帧。这使得能够对驾驶员的视线行为进行全面的全方位建模。此外，我们的全景注意预测方法——DriverGaze360-Net 通过采用辅助语义分割头，共同学习注意力图和被关注的对象，从而改善了在宽广全景输入中的空间感知能力和注意预测。广泛的实验表明，DriverGaze360-Net 在多个指标上实现了全景驾驶图像中注意预测的最先进的性能。 数据集和方法可在以下链接获取：[请在此处插入URL]。
URL
https://arxiv.org/abs/2512.14266
PDF
-
Seeing the Whole Picture: Distribution-Guided Data-Free Distillation for Semantic Segmentation
2025-12-15 10:37:05Hongxuan Sun, Tao WuAbstract
Semantic segmentation requires a holistic understanding of the physical world, as it assigns semantic labels to spatially continuous and structurally coherent objects rather than to isolated pixels. However, existing data-free knowledge distillation (DFKD) methods-primarily designed for classification-often disregard this continuity, resulting in significant performance degradation when applied directly to segmentation tasks. In this paper, we introduce DFSS, a novel data-free distillation framework tailored for semantic segmentation. Unlike prior approaches that treat pixels independently, DFSS respects the structural and contextual continuity of real-world scenes. Our key insight is to leverage Batch Normalization (BN) statistics from a teacher model to guide Approximate Distribution Sampling (ADS), enabling the selection of data that better reflects the original training distribution-without relying on potentially misleading teacher predictions. Additionally, we propose Weighted Distribution Progressive Distillation (WDPD), which dynamically prioritizes reliable samples that are more closely aligned with the original data distribution early in training and gradually incorporates more challenging cases, mirroring the natural progression of learning in human perception. Extensive experiments on standard benchmarks demonstrate that DFSS consistently outperforms existing data-free distillation methods for semantic segmentation, achieving state-of-the-art results with significantly reduced reliance on auxiliary data.
Abstract (translated)
语义分割需要对物理世界有一个整体的理解，因为它为连续且结构连贯的对象分配语义标签，而不仅仅是孤立的像素。然而，现有的无数据知识蒸馏（DFKD）方法主要针对分类设计，常常忽视这种连续性，在直接应用于分割任务时会导致性能显著下降。在本文中，我们介绍了 DFSS，一种专为语义分割定制的新颖无数据蒸馏框架。与之前将像素独立处理的方法不同，DFSS 尊重现实场景中的结构和上下文连贯性。我们的核心见解是利用教师模型的批量归一化（BN）统计信息来指导近似分布采样（ADS），从而选择更好地反映原始训练分布的数据——而不依赖可能具有误导性的教师预测。 此外，我们提出了加权分布渐进式蒸馏（WDPD），该方法在训练初期优先考虑与原始数据分布更加一致的可靠样本，并逐渐纳入更具挑战性的情况，这反映了人类感知学习过程中的自然进展。在标准基准测试上的广泛实验表明，DFSS 一贯优于现有的无数据知识蒸馏方法，在语义分割中实现了最先进的结果，并且显著减少了对辅助数据的依赖。
URL
https://arxiv.org/abs/2512.13175
PDF
-
JoDiffusion: Jointly Diffusing Image with Pixel-Level Annotations for Semantic Segmentation Promotion
2025-12-15 06:21:30Haoyu Wang, Lei Zhang, Wenrui Liu, Dengyang Jiang, Wei Wei, Chen DingAbstract
Given the inherently costly and time-intensive nature of pixel-level annotation, the generation of synthetic datasets comprising sufficiently diverse synthetic images paired with ground-truth pixel-level annotations has garnered increasing attention recently for training high-performance semantic segmentation models. However, existing methods necessitate to either predict pseudo annotations after image generation or generate images conditioned on manual annotation masks, which incurs image-annotation semantic inconsistency or scalability problem. To migrate both problems with one stone, we present a novel dataset generative diffusion framework for semantic segmentation, termed JoDiffusion. Firstly, given a standard latent diffusion model, JoDiffusion incorporates an independent annotation variational auto-encoder (VAE) network to map annotation masks into the latent space shared by images. Then, the diffusion model is tailored to capture the joint distribution of each image and its annotation mask conditioned on a text prompt. By doing these, JoDiffusion enables simultaneously generating paired images and semantically consistent annotation masks solely conditioned on text prompts, thereby demonstrating superior scalability. Additionally, a mask optimization strategy is developed to mitigate the annotation noise produced during generation. Experiments on Pascal VOC, COCO, and ADE20K datasets show that the annotated dataset generated by JoDiffusion yields substantial performance improvements in semantic segmentation compared to existing methods.
Abstract (translated)
鉴于像素级标注固有的高成本和耗时特性，近期越来越多的研究关注于生成包含足够多样化合成图像及其真实像素级标注的合成数据集，以训练高性能的语义分割模型。然而，现有方法要么在图像生成后预测伪标签，要么根据手动注释掩码生成图像，这会导致图像-标注语义不一致或可扩展性问题。为了同时解决这两个问题，我们提出了一种新的针对语义分割的数据集生成扩散框架，名为JoDiffusion。 首先，给定标准的潜在扩散模型，JoDiffusion整合了一个独立的注释变分自动编码器（VAE）网络，将注释掩码映射到与图像共享的潜在空间中。然后，该扩散模型被定制以捕捉每个图像及其标注掩码在文本提示条件下的联合分布。通过这种方式，JoDiffusion能够仅根据文本提示同时生成配对的图像和语义一致性的标注掩码，从而表现出卓越的可扩展性。 此外，我们开发了一种掩码优化策略来减轻生成过程中的注释噪声。在Pascal VOC、COCO和ADE20K数据集上的实验表明，JoDiffusion生成的标注数据集相比现有方法，在语义分割性能上取得了显著提升。
URL
https://arxiv.org/abs/2512.13014
PDF
-
TWLR: Text-Guided Weakly-Supervised Lesion Localization and Severity Regression for Explainable Diabetic Retinopathy Grading
2025-12-15 06:08:16Xi Luo, Shixin Xu, Ying Xie, JianZhong Hu, Yuwei He, Yuhui Deng, Huaxiong HuangAbstract
Accurate medical image analysis can greatly assist clinical diagnosis, but its effectiveness relies on high-quality expert annotations Obtaining pixel-level labels for medical images, particularly fundus images, remains costly and time-consuming. Meanwhile, despite the success of deep learning in medical imaging, the lack of interpretability limits its clinical adoption. To address these challenges, we propose TWLR, a two-stage framework for interpretable diabetic retinopathy (DR) assessment. In the first stage, a vision-language model integrates domain-specific ophthalmological knowledge into text embeddings to jointly perform DR grading and lesion classification, effectively linking semantic medical concepts with visual features. The second stage introduces an iterative severity regression framework based on weakly-supervised semantic segmentation. Lesion saliency maps generated through iterative refinement direct a progressive inpainting mechanism that systematically eliminates pathological features, effectively downgrading disease severity toward healthier fundus appearances. Critically, this severity regression approach achieves dual benefits: accurate lesion localization without pixel-level supervision and providing an interpretable visualization of disease-to-healthy transformations. Experimental results on the FGADR, DDR, and a private dataset demonstrate that TWLR achieves competitive performance in both DR classification and lesion segmentation, offering a more explainable and annotation-efficient solution for automated retinal image analysis.
Abstract (translated)
准确的医学影像分析能够极大地辅助临床诊断，但其有效性依赖于高质量的专业标注。获取医学图像（尤其是眼底图像）的像素级标签既耗时又昂贵。尽管深度学习在医学成像领域的成功已经取得突破，但缺乏可解释性限制了其临床应用。为了解决这些问题，我们提出了TWLR，这是一种用于可解释的糖尿病视网膜病变（DR）评估的两阶段框架。 第一阶段中，视觉-语言模型整合特定领域的专业眼科知识到文本嵌入中，从而同时执行DR分级和病变分类。这有效地将语义医学概念与视觉特征联系起来。 第二阶段引入了一个基于弱监督语义分割的迭代严重程度回归框架。通过迭代细化生成的病变显著图引导渐进式填充机制，系统地消除病理特征，使疾病严重程度逐步向更健康的眼底外观退化。关键在于这一严重度回归方法实现了双重效果：在没有像素级监督的情况下实现准确的病变定位，并提供可解释的疾病到健康转变可视化。 实验结果表明，在FGADR、DDR以及一个私有数据集上，TWLR在DR分类和病变分割方面均达到了竞争性的性能，为自动视网膜图像分析提供了更加可解释且标注效率更高的解决方案。
URL
https://arxiv.org/abs/2512.13008
PDF
-
OMUDA: Omni-level Masking for Unsupervised Domain Adaptation in Semantic Segmentation
2025-12-13 12:01:23Yang Ou, Xiongwei Zhao, Xinye Yang, Yihan Wang, Yicheng Di, Rong Yuan, Xieyuanli Chen, Xu ZhuAbstract
Unsupervised domain adaptation (UDA) enables semantic segmentation models to generalize from a labeled source domain to an unlabeled target domain. However, existing UDA methods still struggle to bridge the domain gap due to cross-domain contextual ambiguity, inconsistent feature representations, and class-wise pseudo-label noise. To address these challenges, we propose Omni-level Masking for Unsupervised Domain Adaptation (OMUDA), a unified framework that introduces hierarchical masking strategies across distinct representation levels. Specifically, OMUDA comprises: 1) a Context-Aware Masking (CAM) strategy that adaptively distinguishes foreground from background to balance global context and local details; 2) a Feature Distillation Masking (FDM) strategy that enhances robust and consistent feature learning through knowledge transfer from pre-trained models; and 3) a Class Decoupling Masking (CDM) strategy that mitigates the impact of noisy pseudo-labels by explicitly modeling class-wise uncertainty. This hierarchical masking paradigm effectively reduces the domain shift at the contextual, representational, and categorical levels, providing a unified solution beyond existing approaches. Extensive experiments on multiple challenging cross-domain semantic segmentation benchmarks validate the effectiveness of OMUDA. Notably, on the SYNTHIA->Cityscapes and GTA5->Cityscapes tasks, OMUDA can be seamlessly integrated into existing UDA methods and consistently achieving state-of-the-art results with an average improvement of 7%.
Abstract (translated)
无监督领域适应（UDA）使语义分割模型能够从有标签的源域泛化到无标签的目标域。然而，现有的UDA方法仍然难以弥合因跨域上下文模糊性、特征表示不一致以及类别伪标签噪声等原因导致的领域差距。为了解决这些挑战，我们提出了Omni-level Masking for Unsupervised Domain Adaptation（OMUDA），这是一种统一框架，引入了在不同表示层次上的分层掩码策略。具体来说，OMUDA包括： 1. 上下文感知掩码（CAM）策略：该策略能够自适应地区分前景和背景，并平衡全局上下文与局部细节； 2. 特征蒸馏掩码（FDM）策略：通过从预训练模型的知识转移来增强鲁棒且一致的特征学习； 3. 类别解耦掩码（CDM）策略：该策略通过显式建模类别不确定性，减轻了噪声伪标签的影响。 这种分层掩码范例在上下文、表示和分类级别上有效地减少了领域偏移，并提供了一种超越现有方法的统一解决方案。多项具有挑战性的跨域语义分割基准实验验证了OMUDA的有效性。特别地，在SYNTHIA->Cityscapes和GTA5->Cityscapes任务中，OMUDA可以无缝集成到现有的UDA方法中，并始终取得最先进的结果，平均改进达到7%。
URL
https://arxiv.org/abs/2512.12303
PDF
-
Enhancing deep learning performance on burned area delineation from SPOT-6/7 imagery for emergency management
2025-12-12 21:54:20Maria Rodriguez, Minh-Tan Pham, Martin Sudmanns, Quentin Poterek, Oscar NarvaezAbstract
After a wildfire, delineating burned areas (BAs) is crucial for quantifying damages and supporting ecosystem recovery. Current BA mapping approaches rely on computer vision models trained on post-event remote sensing imagery, but often overlook their applicability to time-constrained emergency management scenarios. This study introduces a supervised semantic segmentation workflow aimed at boosting both the performance and efficiency of BA delineation. It targets SPOT-6/7 imagery due to its very high resolution and on-demand availability. Experiments are evaluated based on Dice score, Intersection over Union, and inference time. The results show that U-Net and SegFormer models perform similarly with limited training data. However, SegFormer requires more resources, challenging its practical use in emergencies. Incorporating land cover data as an auxiliary task enhances model robustness without increasing inference time. Lastly, Test-Time Augmentation improves BA delineation performance but raises inference time, which can be mitigated with optimization methods like Mixed Precision.
Abstract (translated)
在野火之后，界定烧毁区域（Burned Areas，BAs）对于量化损失和支持生态系统恢复至关重要。当前的BA制图方法依赖于基于事后遥感图像训练的计算机视觉模型，但通常忽略了这些方法在时间紧迫的应急管理工作中的适用性。本研究引入了一种监督语义分割工作流程，旨在提升BA界定的性能和效率。该工作流程针对SPOT-6/7影像，因其非常高的分辨率及按需获取的特点而被选定为目标数据源。实验评估基于Dice系数、交并比（Intersection over Union）以及推理时间进行。 研究结果显示，在训练数据有限的情况下，U-Net和SegFormer模型表现类似。然而，SegFormer需要更多资源，这对其在紧急情况下的实际应用构成了挑战。将土地覆盖数据作为辅助任务纳入模型可以增强模型的鲁棒性而不会增加推理时间。最后，测试时的数据增强（Test-Time Augmentation）能提升BA界定性能，但会增加推理时间，这一问题可通过混合精度等优化方法来缓解。
URL
https://arxiv.org/abs/2512.12056
PDF
