Current methods for 3D semantic segmentation propose training models with limited annotations to address the difficulty of annotating large, irregular, and unordered 3D point cloud data. They usually focus on the 3D domain only, without leveraging the complementary nature of 2D and 3D data. Besides, some methods extend original labels or generate pseudo labels to guide the training, but they often fail to fully use these labels or address the noise within them. Meanwhile, the emergence of comprehensive and adaptable foundation models has offered effective solutions for segmenting 2D data. Leveraging this advancement, we present a novel approach that maximizes the utility of sparsely available 3D annotations by incorporating segmentation masks generated by 2D foundation models. We further propagate the 2D segmentation masks into the 3D space by establishing geometric correspondences between 3D scenes and 2D views. We extend the highly sparse annotations to encompass the areas delineated by 3D masks, thereby substantially augmenting the pool of available labels. Furthermore, we apply confidence- and uncertainty-based consistency regularization on augmentations of the 3D point cloud and select the reliable pseudo labels, which are further spread on the 3D masks to generate more labels. This innovative strategy bridges the gap between limited 3D annotations and the powerful capabilities of 2D foundation models, ultimately improving the performance of 3D weakly supervised segmentation.
当前的三维语义分割方法通过使用有限注释的数据来训练模型,以解决对大量、不规则且无序的三维点云数据进行标注的困难。这些方法通常只专注于三维领域,而没有利用二维和三维数据之间的互补性。此外,一些方法扩展原始标签或生成伪标签以指导训练,但它们往往未能充分利用这些标签或有效处理其中的噪声。与此同时,全面且适应性强的基础模型的出现为分割二维数据提供了有效的解决方案。借助这一进展,我们提出了一种新方法,通过引入由二维基础模型生成的分割掩码来最大化稀疏可用三维注释的效用。进一步地,我们将二维分割掩码传播到三维空间中,通过在三维场景和二维视图之间建立几何对应关系实现这一点。我们将高度稀疏的标注扩展到由三维掩码界定的区域,从而大幅增加了可用标签的数量。此外,我们对三维点云的数据增强应用了基于置信度和不确定性的一致性正则化,并选择可靠的伪标签,这些伪标签进一步被传播在三维掩码上以生成更多标签。这一创新策略弥合了有限的三维注释与二维基础模型的强大能力之间的差距,最终提升了弱监督下三维语义分割的表现。
https://arxiv.org/abs/2508.19909
Fine-grained action localization in untrimmed sports videos presents a significant challenge due to rapid and subtle motion transitions over short durations. Existing supervised and weakly supervised solutions often rely on extensive annotated datasets and high-capacity models, making them computationally intensive and less adaptable to real-world scenarios. In this work, we introduce a lightweight and unsupervised skeleton-based action localization pipeline that leverages spatio-temporal graph neural representations. Our approach pre-trains an Attention-based Spatio-Temporal Graph Convolutional Network (ASTGCN) on a pose-sequence denoising task with blockwise partitions, enabling it to learn intrinsic motion dynamics without any manual labeling. At inference, we define a novel Action Dynamics Metric (ADM), computed directly from low-dimensional ASTGCN embeddings, which detects motion boundaries by identifying inflection points in its curvature profile. Our method achieves a mean Average Precision (mAP) of 82.66% and average localization latency of 29.09 ms on the DSV Diving dataset, matching state-of-the-art supervised performance while maintaining computational efficiency. Furthermore, it generalizes robustly to unseen, in-the-wild diving footage without retraining, demonstrating its practical applicability for lightweight, real-time action analysis systems in embedded or dynamic environments.
未剪辑体育视频中的精细动作定位由于在短时间内快速而微妙的运动过渡,带来了重大挑战。现有的监督学习和弱监督解决方案通常依赖于大量注释数据集和高容量模型,这使得它们计算成本高昂且不太适应实际场景。在这项工作中,我们提出了一种基于骨架、轻量级且无监督的动作定位流水线,该管道利用了空间-时间图神经表示。我们的方法通过在分块划分的姿势序列去噪任务上对Attention-based Spatio-Temporal Graph Convolutional Network (ASTGCN)进行预训练,使其能够学习内在运动动力学而无需任何人工标注。 在推断阶段,我们定义了一种新颖的动作动态度量(ADM),该度量直接从低维ASTGCN嵌入中计算得出,并通过识别其曲率轮廓中的拐点来检测运动边界。我们的方法在DSV跳水数据集上实现了82.66%的平均精度(mAP)和29.09毫秒的平均定位延迟,同时匹配了最先进的监督性能并保持了计算效率。此外,该方法还能稳健地推广到未见过的真实场景中的潜水视频,无需重新训练,证明其在嵌入式或动态环境中实现轻量级、实时动作分析系统的实际应用性。
https://arxiv.org/abs/2508.19647
Surface defect detection is a critical task across numerous industries, aimed at efficiently identifying and localising imperfections or irregularities on manufactured components. While numerous methods have been proposed, many fail to meet industrial demands for high performance, efficiency, and adaptability. Existing approaches are often constrained to specific supervision scenarios and struggle to adapt to the diverse data annotations encountered in real-world manufacturing processes, such as unsupervised, weakly supervised, mixed supervision, and fully supervised settings. To address these challenges, we propose SuperSimpleNet, a highly efficient and adaptable discriminative model built on the foundation of SimpleNet. SuperSimpleNet incorporates a novel synthetic anomaly generation process, an enhanced classification head, and an improved learning procedure, enabling efficient training in all four supervision scenarios, making it the first model capable of fully leveraging all available data annotations. SuperSimpleNet sets a new standard for performance across all scenarios, as demonstrated by its results on four challenging benchmark datasets. Beyond accuracy, it is very fast, achieving an inference time below 10 ms. With its ability to unify diverse supervision paradigms while maintaining outstanding speed and reliability, SuperSimpleNet represents a promising step forward in addressing real-world manufacturing challenges and bridging the gap between academic research and industrial applications. Code: this https URL
表面缺陷检测是众多行业中的一项关键任务,旨在高效识别和定位制造组件上的瑕疵或不规则现象。尽管已经提出了许多方法,但很多方法未能满足工业对高性能、高效率以及灵活性的需求。现有方法通常局限于特定的监督场景,并且难以适应真实世界制造业中遇到的各种数据标注情况,如无监督、弱监督、混合监督及全监督设置等。为了解决这些挑战,我们提出了一种基于SimpleNet基础构建的高度高效和灵活的判别模型——SuperSimpleNet。SuperSimpleNet整合了新颖的人工异常生成过程、增强型分类头以及改进的学习程序,能够适应所有四种监督场景下的有效训练,使其成为首个能全面利用所有可用数据标注的模型。在四个具有挑战性的基准数据集上,SuperSimpleNet展示了其卓越性能,并证明了它可以在所有情景下设定新的标准。除了准确性之外,它还非常快速,在推理时间方面低于10毫秒。凭借统一各种监督范式的能力并同时保持出色的高速度和可靠性,SuperSimpleNet代表了一种应对现实世界制造挑战以及弥合学术研究与工业应用之间差距的有希望的进步。 代码地址:[此链接](https://this%20URL)
https://arxiv.org/abs/2508.19060
Monitoring coral reefs at large spatial scales remains an open challenge, essential for assessing ecosystem health and informing conservation efforts. While drone-based aerial imagery offers broad spatial coverage, its limited resolution makes it difficult to reliably distinguish fine-scale classes, such as coral morphotypes. At the same time, obtaining pixel-level annotations over large spatial extents is costly and labor-intensive, limiting the scalability of deep learning-based segmentation methods for aerial imagery. We present a multi-scale weakly supervised semantic segmentation framework that addresses this challenge by transferring fine-scale ecological information from underwater imagery to aerial data. Our method enables large-scale coral reef mapping from drone imagery with minimal manual annotation, combining classification-based supervision, spatial interpolation and self-distillation techniques. We demonstrate the efficacy of the approach, enabling large-area segmentation of coral morphotypes and demonstrating flexibility for integrating new classes. This study presents a scalable, cost-effective methodology for high-resolution reef monitoring, combining low-cost data collection, weakly supervised deep learning and multi-scale remote sensing.
监测大型空间范围内的珊瑚礁仍然是一个开放性的挑战,这对于评估生态系统健康和指导保护工作至关重要。虽然基于无人机的空中影像提供了广阔的地理覆盖范围,但其有限的分辨率使得难以可靠地区分如珊瑚形态等细小尺度类别。同时,在大规模范围内获取像素级标注既昂贵又耗时,这限制了深度学习分割方法的大规模应用。 我们提出了一种多尺度弱监督语义分割框架,通过将水下影像中的精细生态信息转移到空中数据中来解决这一挑战。我们的方法能够利用无人机影像对珊瑚礁进行大范围测绘,并且只需要少量的人工标注,结合分类驱动的监督、空间插值和自我蒸馏技术。我们展示了这种方法的有效性,使其能够在大面积范围内分割出珊瑚形态类型,并证明了其在整合新类别方面的灵活性。 本研究提供了一种可扩展且成本效益高的方法来实现高分辨率礁石监测,该方法结合了低成本的数据收集、弱监督的深度学习和多尺度遥感技术。
https://arxiv.org/abs/2508.18958
Weakly Supervised Semantic Segmentation (WSSS) with image-level labels has gained attention for its cost-effectiveness. Most existing methods emphasize inter-class separation, often neglecting the shared semantics among related categories and lacking fine-grained discrimination. To address this, we propose Contrastive Prompt Clustering (CPC), a novel WSSS framework. CPC exploits Large Language Models (LLMs) to derive category clusters that encode intrinsic inter-class relationships, and further introduces a class-aware patch-level contrastive loss to enforce intra-class consistency and inter-class separation. This hierarchical design leverages clusters as coarse-grained semantic priors while preserving fine-grained boundaries, thereby reducing confusion among visually similar categories. Experiments on PASCAL VOC 2012 and MS COCO 2014 demonstrate that CPC surpasses existing state-of-the-art methods in WSSS.
弱监督语义分割(WSSS)使用图像级标签已经因其成本效益而受到关注。大多数现有的方法侧重于类间分离,往往忽略了相关类别之间的共享语义,并且缺乏细粒度的区分能力。为了解决这些问题,我们提出了对比提示聚类(Contrastive Prompt Clustering, CPC),这是一种新颖的WSSS框架。CPC利用大型语言模型(LLMs)来推导编码固有类间关系的类别簇,并进一步引入了一种以类别为中心的patch级对比损失,用以强化类内一致性并保持类间分离。这种分层设计利用簇作为粗粒度语义先验的同时保留细粒度边界,从而减少视觉上相似类别之间的混淆。在PASCAL VOC 2012和MS COCO 2014数据集上的实验表明,CPC超越了现有的WSSS方法的最先进水平。
https://arxiv.org/abs/2508.17009
Meta-learning aims to uniformly sample homogeneous support-query pairs, characterized by the same categories and similar attributes, and extract useful inductive biases through identical network architectures. However, this identical network design results in over-semantic homogenization. To address this, we propose a novel homologous but heterogeneous network. By treating support-query pairs as dual perspectives, we introduce heterogeneous visual aggregation (HA) modules to enhance complementarity while preserving semantic commonality. To further reduce semantic noise and amplify the uniqueness of heterogeneous semantics, we design a heterogeneous transfer (HT) module. Finally, we propose heterogeneous CLIP (HC) textual information to enhance the generalization capability of multimodal models. In the weakly-supervised few-shot semantic segmentation (WFSS) task, with only 1/24 of the parameters of existing state-of-the-art models, TLG achieves a 13.2\% improvement on Pascal-5\textsuperscript{i} and a 9.7\% improvement on COCO-20\textsuperscript{i}. To the best of our knowledge, TLG is also the first weakly supervised (image-level) model that outperforms fully supervised (pixel-level) models under the same backbone architectures. The code is available at this https URL.
元学习的目标是均匀采样同质的支持-查询对,这些对具有相同的类别和相似的属性,并通过相同的网络架构提取有用的归纳偏差。然而,这种相同的设计导致了过度语义同质化。为了解决这个问题,我们提出了一种新型但异构的网络设计。通过将支持-查询对视为双重视角,我们引入了异构视觉聚合(HA)模块来增强互补性同时保留语义共性。为了进一步减少语义噪声并放大异构语义的独特性,我们设计了一个异构传输(HT)模块。最后,我们提出了一种异构CLIP (HC) 文本信息以提升多模态模型的泛化能力。 在弱监督下的少样本语义分割(WFSS)任务中,TLG模型仅使用现有最先进模型参数量的1/24,在Pascal-5i上实现了13.2%的改进,并在COCO-20i上实现了9.7%的改进。据我们所知,TLG也是第一个超越完全监督(像素级别)模型的弱监督(图像级别)模型,在相同的骨干架构下表现出色。 代码可在提供的链接处获取:[此URL](https://this-url.com) (请将"this https URL"替换为实际可用的具体URL地址)。
https://arxiv.org/abs/2508.16159
This manuscript presents a series of my selected contributions to the topic of label-efficient learning in computer vision and remote sensing. The central focus of this research is to develop and adapt methods that can learn effectively from limited or partially annotated data, and can leverage abundant unlabeled data in real-world applications. The contributions span both methodological developments and domain-specific adaptations, in particular addressing challenges unique to Earth observation data such as multi-modality, spatial resolution variability, and scene heterogeneity. The manuscript is organized around four main axes including (1) weakly supervised learning for object discovery and detection based on anomaly-aware representations learned from large amounts of background images; (2) multi-task learning that jointly trains on multiple datasets with disjoint annotations to improve performance on object detection and semantic segmentation; (3) self-supervised and supervised contrastive learning with multimodal data to enhance scene classification in remote sensing; and (4) few-shot learning for hierarchical scene classification using both explicit and implicit modeling of class hierarchies. These contributions are supported by extensive experimental results across natural and remote sensing datasets, reflecting the outcomes of several collaborative research projects. The manuscript concludes by outlining ongoing and future research directions focused on scaling and enhancing label-efficient learning for real-world applications.
这份手稿展示了我为计算机视觉和遥感领域中标签高效学习主题所选的一系列贡献。这项研究的核心在于开发并调整能够从有限或部分标注的数据中有效学习的方法,并且能够在实际应用中利用丰富的未标注数据。这些贡献涵盖了方法论的发展以及特定领域的适应,特别是针对地球观测数据的多模态、空间分辨率变化和场景异质性等独特挑战。手稿围绕四个主要方面组织: 1. 弱监督学习:基于从大量背景图像中学到的异常感知表示来进行对象发现与检测。 2. 多任务学习:在同一套不同标注的数据集上联合训练,以提高目标检测和语义分割的性能。 3. 自监督及监督对比学习:利用多模态数据增强遥感中的场景分类。 4. 少量样本学习:通过显式和隐式的类层级模型来进行分层场景分类。 这些贡献得到了跨自然与遥感数据集的大量实验结果的支持,反映了多个合作研究项目的成果。手稿最后概述了正在进行和未来的研究方向,重点在于扩大并提升标签高效学习在实际应用中的规模和效果。
https://arxiv.org/abs/2508.15973
Tree instance segmentation of airborne laser scanning (ALS) data is of utmost importance for forest monitoring, but remains challenging due to variations in the data caused by factors such as sensor resolution, vegetation state at acquisition time, terrain characteristics, etc. Moreover, obtaining a sufficient amount of precisely labeled data to train fully supervised instance segmentation methods is expensive. To address these challenges, we propose a weakly supervised approach where labels of an initial segmentation result obtained either by a non-finetuned model or a closed form algorithm are provided as a quality rating by a human operator. The labels produced during the quality assessment are then used to train a rating model, whose task is to classify a segmentation output into the same classes as specified by the human operator. Finally, the segmentation model is finetuned using feedback from the rating model. This in turn improves the original segmentation model by 34\% in terms of correctly identified tree instances while considerably reducing the number of non-tree instances predicted. Challenges still remain in data over sparsely forested regions characterized by small trees (less than two meters in height) or within complex surroundings containing shrubs, boulders, etc. which can be confused as trees where the performance of the proposed method is reduced.
基于航空激光扫描(ALS)数据的树实例分割对于森林监测至关重要,但由于传感器分辨率、采集时植被状态、地形特征等因素的影响,这一任务仍然具有挑战性。此外,获取足够数量的精确标注数据以训练完全监督的实例分割方法是非常昂贵的。为了解决这些难题,我们提出了一种弱监督的方法,在这种方法中,初始分割结果(通过未经微调的模型或封闭形式算法获得)由人工操作员提供质量评分。在质量评估过程中产生的标签被用来训练一个评级模型,其任务是将分割输出分类为人工作业所指定的相同类别。最后,使用评级模型提供的反馈对分割模型进行微调,这反过来又使原始分割模型正确识别树实例的数量提高了34%,同时显著减少了误判为非树实例的数量。 然而,在稀疏森林区域(树木高度小于两米)或复杂环境中(如灌木、巨石等),数据仍然存在挑战。在这些情况下,由于与周围环境的混淆问题,所提出的方法性能会有所下降。
https://arxiv.org/abs/2508.15646
The detection and segmentation of white blood cells in blood smear images is a key step in medical diagnostics, supporting various downstream tasks such as automated blood cell counting, morphological analysis, cell classification, and disease diagnosis and monitoring. Training robust and accurate models requires large amounts of labeled data, which is both time-consuming and expensive to acquire. In this work, we propose a novel approach for weakly supervised segmentation using neural cellular automata (NCA-WSS). By leveraging the feature maps generated by NCA during classification, we can extract segmentation masks without the need for retraining with segmentation labels. We evaluate our method on three white blood cell microscopy datasets and demonstrate that NCA-WSS significantly outperforms existing weakly supervised approaches. Our work illustrates the potential of NCA for both classification and segmentation in a weakly supervised framework, providing a scalable and efficient solution for medical image analysis.
在血涂片图像中检测和分割白细胞是医学诊断中的关键步骤,支持自动血液计数、形态分析、细胞分类以及疾病诊断与监测等下游任务。训练稳健且准确的模型需要大量的标注数据,而这既耗时又昂贵。为此,我们提出了一种基于神经元细胞自动机(Neural Cellular Automata, NCA)的弱监督分割方法(NCA-WSS)。通过利用NCA在分类过程中生成的特征图,我们可以无需使用分割标签重新训练模型即可提取出分割掩模。我们在三个白血细胞显微镜数据集上评估了该方法,并证明NCA-WSS显著优于现有的弱监督方法。我们的研究展示了NCA在弱监督框架下同时用于分类和分割任务的巨大潜力,为医学图像分析提供了一种可扩展且高效的解决方案。
https://arxiv.org/abs/2508.12322
The recent growth of large foundation models that can easily generate pseudo-labels for huge quantity of unlabeled data makes unsupervised Zero-Shot Cross-Domain Image Retrieval (UZS-CDIR) less relevant. In this paper, we therefore turn our attention to weakly supervised ZS-CDIR (WSZS-CDIR) with noisy pseudo labels generated by large foundation models such as CLIP. To this end, we propose CLAIR to refine the noisy pseudo-labels with a confidence score from the similarity between the CLIP text and image features. Furthermore, we design inter-instance and inter-cluster contrastive losses to encode images into a class-aware latent space, and an inter-domain contrastive loss to alleviate domain discrepancies. We also learn a novel cross-domain mapping function in closed-form, using only CLIP text embeddings to project image features from one domain to another, thereby further aligning the image features for retrieval. Finally, we enhance the zero-shot generalization ability of our CLAIR to handle novel categories by introducing an extra set of learnable prompts. Extensive experiments are carried out using TUBerlin, Sketchy, Quickdraw, and DomainNet zero-shot datasets, where our CLAIR consistently shows superior performance compared to existing state-of-the-art methods.
近期,能够轻易为大量未标记数据生成伪标签的大规模基础模型(如CLIP)的发展使得无监督的零样本跨域图像检索(UZS-CDIR)变得不那么相关了。因此,在这篇论文中,我们转向研究由大型基础模型(如CLIP)生成的带有噪声伪标签的弱监督零样本跨域图像检索(WSZS-CDIR)。为此,我们提出了CLAIR方法,该方法通过计算CLIP文本特征与图像特征之间的相似性来赋予每个伪标签一个置信度分数,并以此精炼掉噪声伪标签。此外,我们设计了实例间的和簇间的对比损失,将图像编码到类感知的潜在空间中,并引入跨域对比损失以减轻领域差异。同时,我们还学习了一种新颖的闭式形式跨域映射函数,仅使用CLIP文本嵌入来将一个领域的图像特征投影到另一个领域,从而进一步对齐用于检索的图像特征。最后,为了增强CLAIR处理新类别的零样本泛化能力,我们在系统中引入了额外的一组可学习的提示词。 通过TUBerlin、Sketchy、Quickdraw和DomainNet零样本数据集进行了广泛的实验,结果显示我们的CLAIR方法在性能上始终优于现有的最先进方法。
https://arxiv.org/abs/2508.12290
Clinical guidelines recommend performing left ventricular (LV) linear measurements in B-mode echocardiographic images at the basal level -- typically at the mitral valve leaflet tips -- and aligned perpendicular to the LV long axis along a virtual scanline (SL). However, most automated methods estimate landmarks directly from B-mode images for the measurement task, where even small shifts in predicted points along the LV walls can lead to significant measurement errors, reducing their clinical reliability. A recent semi-automatic method, EnLVAM, addresses this limitation by constraining landmark prediction to a clinician-defined SL and training on generated Anatomical Motion Mode (AMM) images to predict LV landmarks along the same. To enable full automation, a contour-aware SL placement approach is proposed in this work, in which the LV contour is estimated using a weakly supervised B-mode landmark detector. SL placement is then performed by inferring the LV long axis and the basal level-mimicking clinical guidelines. Building on this foundation, we introduce \textit{WiseLVAM} -- a novel, fully automated yet manually adaptable framework for automatically placing the SL and then automatically performing the LV linear measurements in the AMM mode. \textit{WiseLVAM} utilizes the structure-awareness from B-mode images and the motion-awareness from AMM mode to enhance robustness and accuracy with the potential to provide a practical solution for the routine clinical application.
临床指南推荐在B型超声心动图图像的基础水平(通常是在二尖瓣叶尖)进行左心室(LV)线性测量,并且这些测量应该沿着与LV长轴垂直的虚拟扫描线(SL)进行。然而,大多数自动化方法直接从B模式图像估计地标点以完成测量任务,即使预测点在LV壁上的微小偏移也可能导致显著的测量误差,从而降低其临床可靠性。 最近的一种半自动方法EnLVAM通过限制地标预测在由临床医生定义的扫描线上,并使用生成的解剖运动模式(AMM)图像进行训练来解决这一问题。这种方法可以预测沿着相同路径的心室地标位置。为了实现完全自动化,在本研究中提出了一种基于轮廓感知的SL放置方法,其中LV轮廓通过弱监督B模式地标检测器进行估计。然后根据推断出的LV长轴和模仿临床指南中的基础水平来进行扫描线(SL)定位。 在此基础上,我们介绍了\textit{WiseLVAM}——一种全新的、完全自动化的但又可手动调整的框架,用于在AMM模式下自动放置扫描线并进行LV线性测量。该方法利用B模式图像中的结构信息和AMM模式中的运动信息来增强其鲁棒性和准确性,并具有提供实用解决方案以供常规临床应用的潜力。
https://arxiv.org/abs/2508.12023
Graph Neural Networks (GNNs) have emerged as a powerful approach for graph-based machine learning tasks. Previous work applied GNNs to image-derived graph representations for various downstream tasks such as classification or anomaly detection. These transformations include segmenting images, extracting features from segments, mapping them to nodes, and connecting them. However, to the best of our knowledge, no study has rigorously compared the effectiveness of the numerous potential image-to-graph transformation approaches for GNN-based graph-level anomaly detection (GLAD). In this study, we systematically evaluate the efficacy of multiple segmentation schemes, edge construction strategies, and node feature sets based on color, texture, and shape descriptors to produce suitable image-derived graph representations to perform graph-level anomaly detection. We conduct extensive experiments on dermoscopic images using state-of-the-art GLAD models, examining performance and efficiency in purely unsupervised, weakly supervised, and fully supervised regimes. Our findings reveal, for example, that color descriptors contribute the best standalone performance, while incorporating shape and texture features consistently enhances detection efficacy. In particular, our best unsupervised configuration using OCGTL achieves a competitive AUC-ROC score of up to 0.805 without relying on pretrained backbones like comparable image-based approaches. With the inclusion of sparse labels, the performance increases substantially to 0.872 and with full supervision to 0.914 AUC-ROC.
图神经网络(GNNs)已成为基于图的机器学习任务的一种强大方法。之前的研究将GNN应用于各种下游任务中的图像衍生图表示,例如分类或异常检测。这些转换包括分割图像、从片段中提取特征、将其映射为节点并进行连接等步骤。然而,据我们所知,没有研究严谨地比较了针对基于GNN的图级异常检测(GLAD)的各种潜在图像到图转换方法的有效性。 在这项研究中,我们系统地评估了多种分割方案、边构建策略以及基于颜色、纹理和形状描述符的节点特征集的效果,以生成适合进行图级异常检测的图像衍生图表示。我们在皮肤镜像数据上进行了广泛的实验,使用最先进的GLAD模型,在完全无监督、弱监督和全监督环境中考察性能和效率。 我们的研究发现例如:单靠颜色描述符可以获得最佳独立表现,而结合形状和纹理特征则会持续提升检测效果。特别是,我们使用的最优无监督配置OCGTL在不依赖预训练骨干网络的情况下达到了最高0.805的AUC-ROC得分,这与类似图像基方法相比具有竞争力。加入稀疏标签后,性能显著提高至0.872 AUC-ROC,在完全监督环境下进一步提升到0.914 AUC-ROC。
https://arxiv.org/abs/2508.11826
Automatic speech recognition (ASR) plays a vital role in enabling natural human-machine interaction across applications such as virtual assistants, industrial automation, customer support, and real-time transcription. However, developing accurate ASR systems for low-resource languages like Arabic remains a significant challenge due to limited labeled data and the linguistic complexity introduced by diverse dialects. In this work, we present a scalable training pipeline that combines weakly supervised learning with supervised fine-tuning to develop a robust Arabic ASR model. In the first stage, we pretrain the model on 15,000 hours of weakly labeled speech covering both Modern Standard Arabic (MSA) and various Dialectal Arabic (DA) variants. In the subsequent stage, we perform continual supervised fine-tuning using a mixture of filtered weakly labeled data and a small, high-quality annotated dataset. Our approach achieves state-of-the-art results, ranking first in the multi-dialectal Arabic ASR challenge. These findings highlight the effectiveness of weak supervision paired with fine-tuning in overcoming data scarcity and delivering high-quality ASR for low-resource, dialect-rich languages.
自动语音识别(ASR)在虚拟助手、工业自动化、客户支持和实时转录等应用中发挥着关键作用,以实现自然的人机交互。然而,对于像阿拉伯语这样资源较少的语言来说,开发准确的ASR系统仍然面临巨大挑战,原因在于标注数据有限以及由多种方言引起的语言复杂性增加。在这项工作中,我们提出了一种可扩展的训练管道,结合了弱监督学习和有监督微调来开发一个稳健的阿拉伯语ASR模型。 在第一阶段,我们在涵盖现代标准阿拉伯语(MSA)和各种方言阿拉伯语(DA)变体的15,000小时弱标注语音数据上预训练模型。随后,在第二阶段中,我们使用过滤后的弱标注数据与小规模高质量注释数据集进行持续有监督微调。 我们的方法取得了最先进的结果,并在多方言阿拉伯语ASR挑战赛中排名第一。这些发现突显了弱监督结合微调的有效性,能够克服数据稀缺问题并为低资源、方言丰富的语言提供高质量的ASR系统。
https://arxiv.org/abs/2508.08912
Quality Assurance (QA) for radiology reports refers to judging whether the junior reports (written by junior doctors) are qualified. The QA scores of one junior report are given by the senior doctor(s) after reviewing the image and junior report. This process requires intensive labor costs for senior doctors. Additionally, the QA scores may be inaccurate for reasons like diagnosis bias, the ability of senior doctors, and so on. To address this issue, we propose a Span-level Quality Assurance EvaluaTOR (Sqator) to mark QA scores automatically. Unlike the common document-level semantic comparison method, we try to analyze the semantic difference by exploring more fine-grained text spans. Unlike the common document-level semantic comparison method, we try to analyze the semantic difference by exploring more fine-grained text spans. Specifically, Sqator measures QA scores by measuring the importance of revised spans between junior and senior reports, and outputs the final QA scores by merging all revised span scores. We evaluate Sqator using a collection of 12,013 radiology reports. Experimental results show that Sqator can achieve competitive QA scores. Moreover, the importance scores of revised spans can be also consistent with the judgments of senior doctors.
放射学报告的质量保证(QA)是指判断由初级医生撰写的报告是否合格。一份初级报告的QA评分是由资深医生在审查影像资料和报告后给出的,这需要大量资深医生的工作成本。此外,由于诊断偏见、资深医生的能力等因素,可能导致QA评分不准确。为解决这一问题,我们提出了一种名为“Span-level Quality Assurance EvaluaTOR”(Sqator)的方法,旨在自动标注QA评分。 不同于常见的文档级语义比较方法,我们尝试通过探索更细粒度的文本片段来分析语义差异。具体来说,Sqator通过测量初级和资深报告之间修订部分的重要性来衡量QA分数,并将所有修订部分的得分合并输出最终的QA分数。我们在12,013份放射学报告的数据集上评估了Sqator的效果。实验结果表明,Sqator能够实现具有竞争力的QA评分效果。此外,修订片段的重要程度评分与资深医生的评判也保持了一致性。
https://arxiv.org/abs/2508.08876
Conversational systems relying on text-based large language models (LLMs) often overlook paralinguistic cues, essential for understanding emotions and intentions. Speech-language models (SLMs), which use speech as input, are emerging as a promising solution. However, SLMs built by extending frozen LLMs struggle to capture paralinguistic information and exhibit reduced context understanding. We identify entangled information and improper training strategies as key issues. To address these issues, we propose two heterogeneous adapters and suggest a weakly supervised training strategy. Our approach disentangles paralinguistic and linguistic information, enabling SLMs to interpret speech through structured representations. It also preserves contextual understanding by avoiding the generation of task-specific vectors through controlled randomness. This approach trains only the adapters on common datasets, ensuring parameter and data efficiency. Experiments demonstrate competitive performance in emotional conversation tasks, showcasing the model's ability to effectively integrate both paralinguistic and linguistic information within contextual settings.
依赖文本的大规模语言模型(LLM)构建的对话系统往往忽视了对情感和意图理解至关重要的副语言线索。以语音为输入的语音-语言模型(SLMs)正逐渐成为解决这一问题的有前景方案。然而,基于冻结状态下的LLM扩展而来的SLMs难以捕捉到副语言信息,并且在上下文理解方面表现较差。我们发现纠缠的信息和不当的训练策略是主要问题所在。 为了应对这些问题,我们提出两种异构适配器并建议采用弱监督式训练策略。我们的方法能够分离出副语言与语言信息,使得SLMs可以通过结构化表示来解读语音,并通过控制随机性避免生成特定任务向量的方式保持上下文理解能力。这一方法仅在通用数据集上对适配器进行训练,从而确保参数和数据的高效使用。 实验结果表明,在情感对话等任务中我们的模型能表现出色,这证明了该模型能够有效地将副语言与语言信息融入到特定的上下文中去。
https://arxiv.org/abs/2508.08095
Facilitating an entity's interaction with objects requires accurately identifying parts that afford specific actions. Weakly supervised affordance grounding (WSAG) seeks to imitate human learning from third-person demonstrations, where humans intuitively grasp functional parts without needing pixel-level annotations. To achieve this, grounding is typically learned using a shared classifier across images from different perspectives, along with distillation strategies incorporating part discovery process. However, since affordance-relevant parts are not always easily distinguishable, models primarily rely on classification, often focusing on common class-specific patterns that are unrelated to affordance. To address this limitation, we move beyond isolated part-level learning by introducing selective prototypical and pixel contrastive objectives that adaptively learn affordance-relevant cues at both the part and object levels, depending on the granularity of the available information. Initially, we find the action-associated objects in both egocentric (object-focused) and exocentric (third-person example) images by leveraging CLIP. Then, by cross-referencing the discovered objects of complementary views, we excavate the precise part-level affordance clues in each perspective. By consistently learning to distinguish affordance-relevant regions from affordance-irrelevant background context, our approach effectively shifts activation from irrelevant areas toward meaningful affordance cues. Experimental results demonstrate the effectiveness of our method. Codes are available at this http URL.
促进实体与物体互动需要准确识别能够执行特定操作的部分。弱监督功能定位(WSAG)旨在模仿人类从第三人称演示中学习的方式,即人类可以直观地理解功能性部分,而无需像素级别的标注信息。为了实现这一目标,通常使用跨不同视角图像的共享分类器来学习定位,并结合发现部位的过程中的蒸馏策略。然而,由于与功能相关的部分并不总是易于区分,模型主要依赖于分类,往往集中于无关紧要的类别特定模式而非功能性相关模式。 为了解决这个局限性,我们通过引入选择性的原型和像素对比目标超越了孤立的部分级学习方法,这些目标可以根据可用信息的粒度自适应地在部件级别和物体级别上学习功能相关的提示。首先,我们利用CLIP(一种视觉语言模型)来确定第一人称视角(以对象为中心)和第三人称视角图像中的与动作关联的对象。然后,通过将不同视图中发现的对象相互参照,我们在每个视角中挖掘精确的功能部分线索。通过持续学习区分功能相关的区域和非相关背景上下文,我们的方法有效地将激活从无关区域转移到有意义的功能提示上。 实验结果证明了我们方法的有效性。代码可在上述提供的链接处获取。
https://arxiv.org/abs/2508.07877
Accurate segmentation of subcutaneous vessels from clinical images is hampered by scarce, expensive ground truth and by low contrast, noisy appearance of vessels across patients and modalities. We present a novel weakly supervised training framework tailored for subcutaneous vessel segmentation that leverages inexpensive sparse annotations (e.g., centerline traces, dot markers, or short scribbles). Sparse labels are expanded into dense, probabilistic supervision via a differentiable random walk label propagation model whose transition weights incorporate image driven vesselness cues and tubular continuity priors. The propagation yields per-pixel hitting probabilities together with calibrated uncertainty estimates; these are incorporated into an uncertainty weighted loss to avoid over fitting to ambiguous regions. Crucially, the label-propagator is learned jointly with a CNN based segmentation predictor, enabling the system to discover vessel edges and continuity constraints without explicit edge supervision. We further introduce a topology aware regularizer that encourages centerline connectivity and penalizes spurious branches, improving clinical usability. In experiments on clinical subcutaneous imaging datasets, our method consistently outperforms naive training on sparse labels and conventional dense pseudo-labeling, producing more complete vascular maps and better calibrated uncertainty for downstream decision making. The approach substantially reduces annotation burden while preserving clinically relevant vessel topology.
从临床图像中精确分割皮下血管受到稀缺且昂贵的真值标注和患者间及模态间的低对比度、噪声外观的影响。我们提出了一种新颖的弱监督训练框架,专门用于皮下血管分割,该框架利用廉价稀疏注释(例如,中心线轨迹、点标记或简短涂鸦)。这些稀疏标签通过一种可微分随机游走标签传播模型扩展为密集且概率化的监督信号,该模型在传输权重中整合了图像驱动的血管特征和管状连续性先验。这种传播产生了每个像素的命中概率以及校准过的不确定性估计;这些信息被纳入一个不确定性加权损失函数中以避免过度拟合到模棱两可区域。关键在于标签传播器与基于 CNN 的分割预测器共同学习,从而使系统能够发现血管边缘和连续性约束而无需明确的边缘监督。此外,我们引入了一种拓扑感知正则化器,鼓励中心线连通并惩罚虚假分支,从而提高临床实用性。在对临床皮下成像数据集进行实验时,我们的方法始终优于基于稀疏标签的简单训练以及传统的密集伪标注,生成了更完整的血管图,并且为下游决策提供了更好的校准不确定性估计。该方法大幅减少了注释负担,同时保留了临床上相关的血管拓扑结构。
https://arxiv.org/abs/2508.06819
Urbanization, climate change, and agricultural stress are increasing the demand for precise and timely environmental monitoring. Land Surface Temperature (LST) is a key variable in this context and is retrieved from remote sensing satellites. However, these systems face a trade-off between spatial and temporal resolution. While spatio-temporal fusion methods offer promising solutions, few have addressed the estimation of daily LST at 10 m resolution. In this study, we present WGAST, a Weakly-Supervised Generative Network for Daily 10 m LST Estimation via Spatio-Temporal Fusion of Terra MODIS, Landsat 8, and Sentinel-2. WGAST is the first end-to-end deep learning framework designed for this task. It adopts a conditional generative adversarial architecture, with a generator composed of four stages: feature extraction, fusion, LST reconstruction, and noise suppression. The first stage employs a set of encoders to extract multi-level latent representations from the inputs, which are then fused in the second stage using cosine similarity, normalization, and temporal attention mechanisms. The third stage decodes the fused features into high-resolution LST, followed by a Gaussian filter to suppress high-frequency noise. Training follows a weakly supervised strategy based on physical averaging principles and reinforced by a PatchGAN discriminator. Experiments demonstrate that WGAST outperforms existing methods in both quantitative and qualitative evaluations. Compared to the best-performing baseline, on average, WGAST reduces RMSE by 17.18% and improves SSIM by 11.00%. Furthermore, WGAST is robust to cloud-induced LST and effectively captures fine-scale thermal patterns, as validated against 33 ground-based sensors. The code is available at this https URL.
城市化、气候变化和农业压力正在增加对精确且及时的环境监测需求。地表温度(Land Surface Temperature,LST)是这一背景下的关键变量,并通过遥感卫星获取数据。然而,这些系统在空间分辨率与时间分辨率之间存在权衡。尽管时空融合方法提供了有前景的解决方案,但鲜少有关于如何估算每日10米分辨率LST的研究。在这项研究中,我们提出了WGAST——一种基于Terra MODIS、Landsat 8和Sentinel-2卫星数据时空融合的弱监督生成网络,用于估计每日10米分辨率的地表温度。WGAST是第一个为这一任务设计的端到端深度学习框架。它采用了一种条件生成对抗架构,其中生成器包含四个阶段:特征提取、融合、LST重建和噪声抑制。第一阶段利用一组编码器从输入中提取多层潜在表示,然后在第二阶段使用余弦相似度、归一化以及时间注意力机制进行融合。第三阶段将融合的特征解码为高分辨率LST,并通过高斯滤波来消除高频噪声。训练采用基于物理平均原理的弱监督策略,并由PatchGAN鉴别器加强。实验表明,WGAST在定量和定性评估中都优于现有方法。相比最佳基准线,在平均情况下,WGAST使RMSE降低了17.18%,SSIM提升了11%。此外,WGAST对云引起的LST具有鲁棒性,并有效捕捉到了细尺度的热模式,通过33个地面传感器进行了验证。代码可在以下网址获取:[提供链接]。
https://arxiv.org/abs/2508.06485
Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image, requiring both precise intra-class localization to pinpoint objects and effective inter-class reasoning to model complex category dependencies. While Vision-Language Pre-training (VLP) models offer a strong open-vocabulary foundation, they often struggle with fine-grained localization under weak supervision and typically fail to explicitly leverage structured relational knowledge beyond basic semantics, limiting performance especially for unseen classes. To overcome these limitations, we propose the Dual Adaptive Refinement Transfer (DART) framework. DART enhances a frozen VLP backbone via two synergistic adaptive modules. For intra-class refinement, an Adaptive Refinement Module (ARM) refines patch features adaptively, coupled with a novel Weakly Supervised Patch Selecting (WPS) loss that enables discriminative localization using only image-level labels. Concurrently, for inter-class transfer, an Adaptive Transfer Module (ATM) leverages a Class Relationship Graph (CRG), constructed using structured knowledge mined from a Large Language Model (LLM), and employs graph attention network to adaptively transfer relational information between class representations. DART is the first framework, to our knowledge, to explicitly integrate external LLM-derived relational knowledge for adaptive inter-class transfer while simultaneously performing adaptive intra-class refinement under weak supervision for OV-MLR. Extensive experiments on challenging benchmarks demonstrate that our DART achieves new state-of-the-art performance, validating its effectiveness.
开放词汇多标签识别(OV-MLR)的目标是在图像中识别多种已知和未知的对象类别,这需要精确的类内定位以确定对象的位置,并有效地进行跨类推理以建模复杂的类别依赖关系。虽然视觉语言预训练(VLP)模型提供了强大的开放词汇基础,但它们在弱监督下的细粒度定位方面常常遇到困难,并且通常无法显式地利用超出基本语义的结构化关联知识,这限制了其在未见过类别的表现能力。 为了克服这些局限性,我们提出了双自适应细化传输(DART)框架。DART通过两个协同作用的自适应模块增强了冻结的VLP骨干网络。对于类内精炼,自适应精炼模块(ARM)能够根据图像级别的标签自适应地精炼块特征,并且结合了一种新颖的弱监督块选择损失(WPS),仅使用图像级别标签即可实现有区分性的定位。与此同时,在跨类别迁移方面,自适应传输模块(ATM)利用了一个基于大型语言模型(LLM)挖掘出的结构化知识构建的类关系图(CRG),并通过图形注意力网络来适应性地在类表示之间传递关联信息。 据我们所知,DART是第一个显式集成外部由LLM产生的关系知识进行自适应跨类别迁移的同时,在弱监督条件下执行自适应内类别精炼以实现OV-MLR的框架。在具有挑战性的基准测试中进行的一系列实验表明,我们的DART达到了新的最先进性能水平,验证了其有效性。 这个介绍展示了DART框架如何通过结合视觉语言预训练模型和大型语言模型的知识来改进开放词汇多标签识别任务中的定位精度和类别关系理解能力。
https://arxiv.org/abs/2508.05585
Skin carcinoma is the most prevalent form of cancer globally, accounting for over $8 billion in annual healthcare expenditures. Early diagnosis, accurate and timely treatment are critical to improving patient survival rates. In clinical settings, physicians document patient visits using detailed SOAP (Subjective, Objective, Assessment, and Plan) notes. However, manually generating these notes is labor-intensive and contributes to clinician burnout. In this work, we propose skin-SOAP, a weakly supervised multimodal framework to generate clinically structured SOAP notes from limited inputs, including lesion images and sparse clinical text. Our approach reduces reliance on manual annotations, enabling scalable, clinically grounded documentation while alleviating clinician burden and reducing the need for large annotated data. Our method achieves performance comparable to GPT-4o, Claude, and DeepSeek Janus Pro across key clinical relevance metrics. To evaluate this clinical relevance, we introduce two novel metrics MedConceptEval and Clinical Coherence Score (CCS) which assess semantic alignment with expert medical concepts and input features, respectively.
皮肤癌是全球最常见的癌症形式,每年的医疗保健支出超过80亿美元。早期诊断、准确及时的治疗对于提高患者生存率至关重要。在临床环境中,医生通过详细的SOAP(主诉、客观检查、评估和计划)记录来记录患者的就诊情况。然而,手动生成这些记录既费时又增加了医务人员的工作负担,导致职业倦怠。为此,我们提出了skin-SOAP,这是一种弱监督多模态框架,可以从有限的输入(包括病变图像和稀疏的临床文本)生成结构化的SOAP记录。我们的方法减少了对人工注释的依赖,使文档制作更具规模性和临床实用性,同时减轻了医务人员的工作负担,并减少了对大规模标注数据的需求。我们提出的方法在关键临床相关性指标上达到了与GPT-4o、Claude和DeepSeek Janus Pro相当的性能水平。为了评估其临床相关性,我们引入了两个新的评价指标:MedConceptEval 和临床连贯性评分(CCS),分别用于评估语义与专家医学概念及输入特征的一致性。
https://arxiv.org/abs/2508.05019