Weakly_Supervised
-
Cross-Modal Prototype Allocation: Unsupervised Slide Representation Learning via Patch-Text Contrast in Computational Pathology
2025-03-26 03:31:07Yuxuan Chen, Jiawen Li, Jiali Hu, Xitong Ling, Tian Guan, Anjia Han, Yonghong HeAbstract
With the rapid advancement of pathology foundation models (FMs), the representation learning of whole slide images (WSIs) attracts increasing attention. Existing studies develop high-quality patch feature extractors and employ carefully designed aggregation schemes to derive slide-level representations. However, mainstream weakly supervised slide representation learning methods, primarily based on multiple instance learning (MIL), are tailored to specific downstream tasks, which limits their generalizability. To address this issue, some studies explore unsupervised slide representation learning. However, these approaches focus solely on the visual modality of patches, neglecting the rich semantic information embedded in textual data. In this work, we propose ProAlign, a cross-modal unsupervised slide representation learning framework. Specifically, we leverage a large language model (LLM) to generate descriptive text for the prototype types present in a WSI, introducing patch-text contrast to construct initial prototype embeddings. Furthermore, we propose a parameter-free attention aggregation strategy that utilizes the similarity between patches and these prototypes to form unsupervised slide embeddings applicable to a wide range of downstream tasks. Extensive experiments on four public datasets show that ProAlign outperforms existing unsupervised frameworks and achieves performance comparable to some weakly supervised models.
Abstract (translated)
随着病理基础模型（FMs）的迅速发展，全滑片图像（WSIs）的表示学习引起了越来越多的关注。现有研究开发了高质量的补丁特征提取器，并采用了精心设计的聚合方案以推导出样本级别的表示。然而，主流的弱监督下的滑片表示学习方法主要基于多实例学习（MIL），这些方法是为特定下游任务量身定制的，这限制了它们的泛化能力。为了解决这个问题，一些研究探索了无监督的滑片表示学习方法。但是，这些方法仅关注补丁的视觉模式，忽略了嵌入在文本数据中的丰富语义信息。 在此项工作中，我们提出了一种跨模态无监督滑片表示学习框架ProAlign。具体而言，我们利用大型语言模型（LLM）为WSI中存在的原型类型生成描述性文本，并引入了补丁-文本对比以构建初始的原型嵌入。此外，我们还提出了一个参数自由的注意力聚合策略，该策略利用补丁与这些原型之间的相似度来形成适用于广泛下游任务的无监督滑片嵌入。 在四个公开数据集上进行的大量实验表明，ProAlign优于现有的无监督框架，并且其性能可媲美一些弱监督模型。
URL
https://arxiv.org/abs/2503.20190
PDF
-
FG$^2$: Fine-Grained Cross-View Localization by Fine-Grained Feature Matching
2025-03-24 14:34:20Zimin Xia, Alexandre AlahiAbstract
We propose a novel fine-grained cross-view localization method that estimates the 3 Degrees of Freedom pose of a ground-level image in an aerial image of the surroundings by matching fine-grained features between the two images. The pose is estimated by aligning a point plane generated from the ground image with a point plane sampled from the aerial image. To generate the ground points, we first map ground image features to a 3D point cloud. Our method then learns to select features along the height dimension to pool the 3D points to a Bird's-Eye-View (BEV) plane. This selection enables us to trace which feature in the ground image contributes to the BEV representation. Next, we sample a set of sparse matches from computed point correspondences between the two point planes and compute their relative pose using Procrustes alignment. Compared to the previous state-of-the-art, our method reduces the mean localization error by 28% on the VIGOR cross-area test set. Qualitative results show that our method learns semantically consistent matches across ground and aerial views through weakly supervised learning from the camera pose.
Abstract (translated)
我们提出了一种新颖的细粒度跨视图定位方法，通过匹配两张图像（地面图像和空中俯瞰图像）之间的细粒度特征来估计地面图像在空中图像中的3个自由度姿态。该姿态是通过对生成自地面图像的点平面与从空中图像中采样得到的点平面进行对齐来估算的。 为了生成地面点，我们首先将地面图像的特征映射到一个三维点云上。然后，我们的方法学习在高度维度选择特征以聚合3D点至鸟瞰图（BEV）平面。这种选择使我们能够追踪哪个地面图像中的特征贡献于BEV表示。 接下来，我们在计算出两个点平面之间的对应关系后采样一组稀疏匹配，并使用普罗克鲁斯特斯对齐法计算它们的相对姿态。 与最新的方法相比，在VIGOR跨区域测试集上，我们的方法将平均定位误差降低了28%。定性结果显示，通过从相机姿态中进行弱监督学习，该方法能够学会在地面视图和空中视图之间找到语义一致的匹配关系。
URL
https://arxiv.org/abs/2503.18725
PDF
-
Neuro-symbolic Weak Supervision: Theory and Semantics
2025-03-24 10:02:51Nijesh Upreti, Vaishak BelleAbstract
Weak supervision allows machine learning models to learn from limited or noisy labels, but it introduces challenges in interpretability and reliability - particularly in multi-instance partial label learning (MI-PLL), where models must resolve both ambiguous labels and uncertain instance-label mappings. We propose a semantics for neuro-symbolic framework that integrates Inductive Logic Programming (ILP) to improve MI-PLL by providing structured relational constraints that guide learning. Within our semantic characterization, ILP defines a logical hypothesis space for label transitions, clarifies classifier semantics, and establishes interpretable performance standards. This hybrid approach improves robustness, transparency, and accountability in weakly supervised settings, ensuring neural predictions align with domain knowledge. By embedding weak supervision into a logical framework, we enhance both interpretability and learning, making weak supervision more suitable for real-world, high-stakes applications.
Abstract (translated)
弱监督允许机器学习模型在有限或有噪声的标签下进行学习，但这也带来了可解释性和可靠性方面的挑战——尤其是在多实例部分标记学习（MI-PLL）中，模型必须解决模棱两可的标签和不确定的实例-标签映射问题。我们提出了一种神经符号框架的语义表达方式，该方式结合了归纳逻辑编程（ILP），以通过提供结构化的关系约束来指导学习从而改进 MI-PLL。在我们的语义描述中，ILP 定义了一个逻辑假设空间用于标签转换，明确了分类器的意义，并确立了可解释性的性能标准。这种混合方法提高了弱监督环境下的稳健性、透明度和问责制，确保神经预测与领域知识相一致。通过将弱监督嵌入到逻辑框架中，我们增强了其可解释性和学习能力，使弱监督更适合于现实世界中的高风险应用。
URL
https://arxiv.org/abs/2503.18509
PDF
-
LiDAR Remote Sensing Meets Weak Supervision: Concepts, Methods, and Perspectives
2025-03-24 06:51:38Yuan Gao, Shaobo Xia, Pu Wang, Xiaohuan Xi, Sheng Nie, Cheng WangAbstract
LiDAR (Light Detection and Ranging) enables rapid and accurate acquisition of three-dimensional spatial data, widely applied in remote sensing areas such as surface mapping, environmental monitoring, urban modeling, and forestry inventory. LiDAR remote sensing primarily includes data interpretation and LiDAR-based inversion. However, LiDAR interpretation typically relies on dense and precise annotations, which are costly and time-consuming. Similarly, LiDAR inversion depends on scarce supervisory signals and expensive field surveys for annotations. To address this challenge, weakly supervised learning has gained significant attention in recent years, with many methods emerging to tackle LiDAR remote sensing tasks using incomplete, inaccurate, and inexact annotations, as well as annotations from other domains. Existing review articles treat LiDAR interpretation and inversion as separate tasks. This review, for the first time, adopts a unified weakly supervised learning perspective to systematically examine research on both LiDAR interpretation and inversion. We summarize the latest advancements, provide a comprehensive review of the development and application of weakly supervised techniques in LiDAR remote sensing, and discuss potential future research directions in this field.
Abstract (translated)
LiDAR（激光探测与测距）技术能够快速且精确地获取三维空间数据，在遥感领域，如地形测绘、环境监测、城市建模和森林资源调查等方面得到广泛应用。LiDAR遥感主要包括数据解释和基于LiDAR的反演两个方面。然而，传统的LiDAR数据解释通常依赖于密集而精准的手动标注，这既耗时又成本高昂。同时，基于LiDAR的反演也需要稀缺且昂贵的现场调查来获取必要的监督信号。 为了解决这些问题，在过去几年中，弱监督学习方法得到了广泛的关注，并且出现了许多使用不完整、不准确或来自其他领域的注释数据的方法，以解决LiDAR遥感任务。现有的综述文章通常将LiDAR解释和反演视为两个独立的任务进行讨论。本文首次从统一的弱监督学习视角出发，系统地考察了有关这两个方面的研究工作。我们总结最新的进展，并全面回顾在LiDAR遥感中应用弱监督技术的发展与应用情况，同时探讨该领域的潜在未来研究方向。
URL
https://arxiv.org/abs/2503.18384
PDF
-
DEPT: Deep Extreme Point Tracing for Ultrasound Image Segmentation
2025-03-19 14:32:14Lei Shi, Xi Fang, Naiyu Wang, Junxing ZhangAbstract
Automatic medical image segmentation plays a crucial role in computer aided diagnosis. However, fully supervised learning approaches often require extensive and labor-intensive annotation efforts. To address this challenge, weakly supervised learning methods, particularly those using extreme points as supervisory signals, have the potential to offer an effective solution. In this paper, we introduce Deep Extreme Point Tracing (DEPT) integrated with Feature-Guided Extreme Point Masking (FGEPM) algorithm for ultrasound image segmentation. Notably, our method generates pseudo labels by identifying the lowest-cost path that connects all extreme points on the feature map-based cost matrix. Additionally, an iterative training strategy is proposed to refine pseudo labels progressively, enabling continuous network improvement. Experimental results on two public datasets demonstrate the effectiveness of our proposed method. The performance of our method approaches that of the fully supervised method and outperforms several existing weakly supervised methods.
Abstract (translated)
自动医学图像分割在计算机辅助诊断中扮演着关键角色。然而，完全监督的学习方法通常需要大量的标注工作，这既耗时又费力。为了解决这一挑战，采用极值点作为监督信号的弱监督学习方法具有提供有效解决方案的潜力。在这篇论文中，我们引入了一种结合特征引导极值点掩码（Feature-Guided Extreme Point Masking, FGEPM）算法的深度极值点追踪（Deep Extreme Point Tracing, DEPT）方法，用于超声图像分割。特别地，我们的方法通过在基于特征图的成本矩阵上找到连接所有极值点的最低成本路径来生成伪标签。此外，我们还提出了一种迭代训练策略，逐步精炼伪标签，从而实现网络性能的持续改进。实验结果表明，在两个公开数据集上的表现证明了我们所提方法的有效性。我们的方法在性能上接近完全监督的方法，并且优于几种现有的弱监督方法。
URL
https://arxiv.org/abs/2503.15260
PDF
-
A Comprehensive Survey on Architectural Advances in Deep CNNs: Challenges, Applications, and Emerging Research Directions
2025-03-19 08:41:06Saddam Hussain Khan (Artificial Intelligence Lab, Department of Computer Systems Engineering, University of Engineering,Applied Sciences), Rashid Iqbal (Artificial Intelligence Lab, Department of Computer Systems Engineering, University of Engineering,Applied Sciences)arXiv_AI Speech_Recognition CNN Recognition Weakly_Supervised Action_Recognition Detection Object_Detection Deep_Learning Review Survey Classification Face Face_Recognition Attention Text_Classification Transfer_Learning Pose_Estimation Inference Prediction Language_Model Sentiment Transformer Pose Action Few-Shot Zero-Shot Medical 3D Enhancement SpeechAbstract
Deep Convolutional Neural Networks (CNNs) have significantly advanced deep learning, driving breakthroughs in computer vision, natural language processing, medical diagnosis, object detection, and speech recognition. Architectural innovations including 1D, 2D, and 3D convolutional models, dilated and grouped convolutions, depthwise separable convolutions, and attention mechanisms address domain-specific challenges and enhance feature representation and computational efficiency. Structural refinements such as spatial-channel exploitation, multi-path design, and feature-map enhancement contribute to robust hierarchical feature extraction and improved generalization, particularly through transfer learning. Efficient preprocessing strategies, including Fourier transforms, structured transforms, low-precision computation, and weight compression, optimize inference speed and facilitate deployment in resource-constrained environments. This survey presents a unified taxonomy that classifies CNN architectures based on spatial exploitation, multi-path structures, depth, width, dimensionality expansion, channel boosting, and attention mechanisms. It systematically reviews CNN applications in face recognition, pose estimation, action recognition, text classification, statistical language modeling, disease diagnosis, radiological analysis, cryptocurrency sentiment prediction, 1D data processing, video analysis, and speech recognition. In addition to consolidating architectural advancements, the review highlights emerging learning paradigms such as few-shot, zero-shot, weakly supervised, federated learning frameworks and future research directions include hybrid CNN-transformer models, vision-language integration, generative learning, etc. This review provides a comprehensive perspective on CNN's evolution from 2015 to 2025, outlining key innovations, challenges, and opportunities.
Abstract (translated)
深度卷积神经网络（CNNs）在深度学习领域取得了重大进展，推动了计算机视觉、自然语言处理、医学诊断、目标检测和语音识别等领域的突破。包括一维、二维及三维卷积模型、扩张卷积与分组卷积、逐深可分离卷积以及注意力机制在内的架构创新解决了特定领域的挑战，并增强了特征表示能力及计算效率。空间-通道利用、多路径设计及特征图增强等结构改进有助于稳健的层级特征提取和泛化性能提升，特别是在迁移学习中表现尤为突出。有效的预处理策略，如傅里叶变换、结构转换、低精度运算及权重压缩，则优化了推理速度并支持在资源受限环境中部署应用。 本文综述提出了一种统一分类体系，根据空间利用、多路径架构、深度、宽度、维度扩展、通道增强和注意力机制对CNN架构进行分类。系统性地回顾了CNN在人脸识别、姿态估计、动作识别、文本分类、统计语言建模、疾病诊断、放射学分析、加密货币情感预测、一维数据处理、视频分析及语音识别等领域的应用。除了总结结构上的进步，还强调了新兴的学习范式如小样本学习（few-shot learning）、零样本学习（zero-shot learning）、弱监督学习和联邦学习框架，并展望了未来研究方向包括混合CNN-Transformer模型、视觉语言整合以及生成性学习等。 本文提供了2015年至2025年间CNN进化历程的全面视角，概述了关键创新、挑战及机遇。
URL
https://arxiv.org/abs/2503.16546
PDF
-
Weakly Supervised Spatial Implicit Neural Representation Learning for 3D MRI-Ultrasound Deformable Image Registration in HDR Prostate Brachytherapy
2025-03-18 16:30:08Jing Wang, Ruirui Liu, Yu Lei, Michael J. Baine, Tian Liu, Yang LeiAbstract
Purpose: Accurate 3D MRI-ultrasound (US) deformable registration is critical for real-time guidance in high-dose-rate (HDR) prostate brachytherapy. We present a weakly supervised spatial implicit neural representation (SINR) method to address modality differences and pelvic anatomy challenges. Methods: The framework uses sparse surface supervision from MRI/US segmentations instead of dense intensity matching. SINR models deformations as continuous spatial functions, with patient-specific surface priors guiding a stationary velocity field for biologically plausible deformations. Validation included 20 public Prostate-MRI-US-Biopsy cases and 10 institutional HDR cases, evaluated via Dice similarity coefficient (DSC), mean surface distance (MSD), and 95% Hausdorff distance (HD95). Results: The proposed method achieved robust registration. For the public dataset, prostate DSC was $0.93 \pm 0.05$, MSD $0.87 \pm 0.10$ mm, and HD95 $1.58 \pm 0.37$ mm. For the institutional dataset, prostate CTV achieved DSC $0.88 \pm 0.09$, MSD $1.21 \pm 0.38$ mm, and HD95 $2.09 \pm 1.48$ mm. Bladder and rectum performance was lower due to ultrasound's limited field of view. Visual assessments confirmed accurate alignment with minimal discrepancies. Conclusion: This study introduces a novel weakly supervised SINR-based approach for 3D MRI-US deformable registration. By leveraging sparse surface supervision and spatial priors, it achieves accurate, robust, and computationally efficient registration, enhancing real-time image guidance in HDR prostate brachytherapy and improving treatment precision.
Abstract (translated)
目的：在高剂量率（HDR）前列腺组织间放射治疗中，精确的3D MRI-超声波（US）可变形配准对于实时引导至关重要。本文介绍了一种基于弱监督的空间隐式神经表示（SINR）方法来解决模态差异和骨盆解剖结构挑战。 方法：该框架使用来自MRI/US分割的稀疏表面监督，而不是密集强度匹配。SINR模型将变形视为连续空间函数，并通过特定于患者的表面先验指导一个静态速度场以实现生物合理的变形。验证包括20个公开的前列腺-MRI-US-活检案例和10个机构HDR案例，在Dice相似系数（DSC）、平均面距离（MSD）和95% Hausdorff距离（HD95）方面进行评估。 结果：所提出的方法实现了稳健的配准。对于公共数据集，前列腺DSC为$0.93 \pm 0.05$，MSD为$0.87 \pm 0.10$毫米，HD95为$1.58 \pm 0.37$毫米。对于机构数据集，前列腺CTV的DSC为$0.88 \pm 0.09$，MSD为$1.21 \pm 0.38$毫米，HD95为$2.09 \pm 1.48$毫米。由于超声波有限的视野范围，膀胱和直肠的表现较低。视觉评估确认了准确对齐且差异很小。 结论：本研究介绍了一种基于弱监督SINR的新方法用于3D MRI-US可变形配准。通过利用稀疏表面监督和空间先验信息，该方法实现了准确、稳健且计算效率高的配准，从而提高了HDR前列腺组织间放射治疗中的实时图像引导精度，并改善了治疗的精确度。
URL
https://arxiv.org/abs/2503.14395
PDF
-
Exploiting Inherent Class Label: Towards Robust Scribble Supervised Semantic Segmentation
2025-03-18 04:43:07Xinliang Zhang, Lei Zhu, Shuang Zeng, Hangzhou He, Ourui Fu, Zhengjian Yao, Zhaoheng Xie, Yanye LuAbstract
Scribble-based weakly supervised semantic segmentation leverages only a few annotated pixels as labels to train a segmentation model, presenting significant potential for reducing the human labor involved in the annotation process. This approach faces two primary challenges: first, the sparsity of scribble annotations can lead to inconsistent predictions due to limited supervision; second, the variability in scribble annotations, reflecting differing human annotator preferences, can prevent the model from consistently capturing the discriminative regions of objects, potentially leading to unstable predictions. To address these issues, we propose a holistic framework, the class-driven scribble promotion network, for robust scribble-supervised semantic segmentation. This framework not only utilizes the provided scribble annotations but also leverages their associated class labels to generate reliable pseudo-labels. Within the network, we introduce a localization rectification module to mitigate noisy labels and a distance perception module to identify reliable regions surrounding scribble annotations and pseudo-labels. In addition, we introduce new large-scale benchmarks, ScribbleCOCO and ScribbleCityscapes, accompanied by a scribble simulation algorithm that enables evaluation across varying scribble styles. Our method demonstrates competitive performance in both accuracy and robustness, underscoring its superiority over existing approaches. The datasets and the codes will be made publicly available.
Abstract (translated)
基于涂鸦的弱监督语义分割方法仅利用少量标注像素作为标签来训练分割模型，从而大大减少了人工标注过程中的劳动量。然而，这种方法面临两大挑战：首先，由于涂鸦标注稀疏导致有限的指导性，可能会产生不一致的预测；其次，不同的人工标注者偏好可能导致涂鸦标注的变化，这可能使模型难以稳定地捕捉到物体的判别区域，进而造成不稳定预测。为了解决这些问题，我们提出了一种全面框架——类别驱动涂鸦推广网络，用于鲁棒的基于涂鸦标签的语义分割。该框架不仅利用给定的涂鸦注释，还通过它们关联的类标签生成可靠的伪标签。在网络内部，我们引入了一个定位修正模块来减少噪声标签的影响，并设计了距离感知模块以识别围绕涂鸦标注和伪标签的可靠区域。此外，我们提出了新的大规模基准测试数据集——ScribbleCOCO 和 ScribbleCityscapes，并附带了一种涂鸦模拟算法，能够在不同风格的涂鸦下进行评估。我们的方法在准确性和鲁棒性方面均表现出色，证明了其相对于现有方法的优势。我们将公开发布这些数据集和代码。
URL
https://arxiv.org/abs/2503.13895
PDF
-
Stitch-a-Recipe: Video Demonstration from Multistep Descriptions
2025-03-18 01:57:48Chi Hsuan Wu, Kumar Ashutosh, Kristen GraumanAbstract
When obtaining visual illustrations from text descriptions, today's methods take a description with-a single text context caption, or an action description-and retrieve or generate the matching visual context. However, prior work does not permit visual illustration of multistep descriptions, e.g. a cooking recipe composed of multiple steps. Furthermore, simply handling each step description in isolation would result in an incoherent demonstration. We propose Stitch-a-Recipe, a novel retrieval-based method to assemble a video demonstration from a multistep description. The resulting video contains clips, possibly from different sources, that accurately reflect all the step descriptions, while being visually coherent. We formulate a training pipeline that creates large-scale weakly supervised data containing diverse and novel recipes and injects hard negatives that promote both correctness and coherence. Validated on in-the-wild instructional videos, Stitch-a-Recipe achieves state-of-the-art performance, with quantitative gains up to 24% as well as dramatic wins in a human preference study.
Abstract (translated)
在从文本描述获取视觉图解时，当今的方法通常采用单一的文本背景说明或动作描述来检索或生成相应的视觉内容。然而，现有的方法并不能处理多步骤描述的可视化展示，例如由多个步骤组成的烹饪食谱。此外，仅仅孤立地处理每个步骤的描述会导致不连贯的表现形式。为此，我们提出了Stitch-a-Recipe这一创新性的基于检索的方法，用于从多步描述中组装视频演示。该方法生成的视频包含来自不同来源的片段，并能够准确反映所有步骤描述的同时保持视觉上的连贯性。 为了训练这个模型，我们设计了一个训练流程，创建了大规模弱监督数据集，其中包含了多样且新颖的食谱，并引入难例负样本以促进正确性和连贯性的提升。在现实世界中的教学视频上进行验证后，Stitch-a-Recipe取得了最先进的性能，在定量评估中最多提高了24%，并且在人类偏好的研究中也有显著的优势。
URL
https://arxiv.org/abs/2503.13821
PDF
-
UCF-Crime-DVS: A Novel Event-Based Dataset for Video Anomaly Detection with Spiking Neural Networks
2025-03-17 08:11:26Yuanbin Qian, Shuhan Ye, Chong Wang, Xiaojie Cai, Jiangbo Qian, Jiafei WuAbstract
Video anomaly detection plays a significant role in intelligent surveillance systems. To enhance model's anomaly recognition ability, previous works have typically involved RGB, optical flow, and text features. Recently, dynamic vision sensors (DVS) have emerged as a promising technology, which capture visual information as discrete events with a very high dynamic range and temporal resolution. It reduces data redundancy and enhances the capture capacity of moving objects compared to conventional camera. To introduce this rich dynamic information into the surveillance field, we created the first DVS video anomaly detection benchmark, namely UCF-Crime-DVS. To fully utilize this new data modality, a multi-scale spiking fusion network (MSF) is designed based on spiking neural networks (SNNs). This work explores the potential application of dynamic information from event data in video anomaly detection. Our experiments demonstrate the effectiveness of our framework on UCF-Crime-DVS and its superior performance compared to other models, establishing a new baseline for SNN-based weakly supervised video anomaly detection.
Abstract (translated)
视频异常检测在智能监控系统中扮演着重要角色。为了提升模型的异常识别能力，以往的研究通常会利用RGB图像、光流和文本特征。最近，动态视觉传感器（DVS）作为一种有前景的技术崭露头角，它能够以离散事件的形式捕捉具有极高动态范围和时间分辨率的视觉信息。这相较于传统摄像头而言，可以减少数据冗余并增强对移动物体的捕获能力。为了将这种丰富的动态信息引入监控领域，我们创建了首个DVS视频异常检测基准——UCF-Crime-DVS。 为了充分利用这一新的数据模式，基于脉冲神经网络（SNNs）设计了一种多尺度脉冲融合网络（MSF）。这项工作探索了事件数据中的动态信息在视频异常检测中的潜在应用。我们的实验展示了该框架在UCF-Crime-DVS上的有效性，并且与其它模型相比表现出更优异的性能，为基于SNN的弱监督视频异常检测建立了新的基准。
URL
https://arxiv.org/abs/2503.12905
PDF
-
Prototype-Based Image Prompting for Weakly Supervised Histopathological Image Segmentation
2025-03-15 09:55:31Qingchen Tang, Lei Fan, Maurice Pagnucco, Yang SongAbstract
Weakly supervised image segmentation with image-level labels has drawn attention due to the high cost of pixel-level annotations. Traditional methods using Class Activation Maps (CAMs) often highlight only the most discriminative regions, leading to incomplete masks. Recent approaches that introduce textual information struggle with histopathological images due to inter-class homogeneity and intra-class heterogeneity. In this paper, we propose a prototype-based image prompting framework for histopathological image segmentation. It constructs an image bank from the training set using clustering, extracting multiple prototype features per class to capture intra-class heterogeneity. By designing a matching loss between input features and class-specific prototypes using contrastive learning, our method addresses inter-class homogeneity and guides the model to generate more accurate CAMs. Experiments on four datasets (LUAD-HistoSeg, BCSS-WSSS, GCSS, and BCSS) show that our method outperforms existing weakly supervised segmentation approaches, setting new benchmarks in histopathological image segmentation.
Abstract (translated)
弱监督下的图像分割，特别是仅使用图像级标签的情况，因像素级标注成本高昂而受到了广泛关注。传统的基于类激活图（CAMs）的方法往往只能突出最具辨识力的区域，导致生成的掩码不完整。近期引入文本信息以改进分割效果的方法，在处理组织病理学图像时遇到了类别间同质性和类别内异质性的问题。 本文提出了一种用于组织病理学图像分割的基于原型的图像提示框架。该方法通过聚类训练集来构建一个图像库，并为每个类别提取多个原型特征，以此捕捉类别内的异质性。设计输入特征与特定类别的原型之间的匹配损失，采用对比学习的方法，以解决类别间同质性问题并引导模型生成更准确的CAMs。 在四个数据集（LUAD-HistoSeg、BCSS-WSSS、GCSS 和 BCSS）上的实验表明，我们的方法超越了现有的弱监督分割方法，在组织病理学图像分割领域设立了新的性能基准。
URL
https://arxiv.org/abs/2503.12068
PDF
-
Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches
2025-03-14 13:21:59Panggih Kusuma Ningrum, Philipp Mayr, Nina Smirnova, Iana AtanassovaAbstract
UnScientify, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking. This approach streamlines the labeling and annotation processes essential for identifying scientific uncertainty, covering a variety of uncertainty expression types to support diverse applications including information retrieval, text mining and scientific document processing. The evaluation results highlight the trade-offs between modern large language models (LLMs) and the UnScientify system. UnScientify, which employs more traditional techniques, achieved superior performance in the scientific uncertainty detection task, attaining an accuracy score of 0.808. This finding underscores the continued relevance and efficiency of UnScientify's simple rule-based and pattern matching strategy for this specific application. The results demonstrate that in scenarios where resource efficiency, interpretability, and domain-specific adaptability are critical, traditional methods can still offer significant advantages.
Abstract (translated)
UnScientify 是一个设计用于检测学术全文中科学不确定性的系统。该系统利用弱监督技术来识别科学文本和作者引用中的口头表达的不确定性。UnScientify 的核心方法基于一个多方面的管道，整合了跨度模式匹配、复杂句子分析以及作者引用检查等功能。这种策略简化了标记和注释过程，有助于识别科学不确定性，并涵盖了各种不确定性的表达类型，支持信息检索、文本挖掘和科学文档处理等多种应用。 评估结果强调了现代大型语言模型（LLMs）与 UnScientify 系统之间的权衡取舍。采用传统技术的 UnScientify 在检测科学不确定性任务中表现出色，准确率达到 0.808。这一发现突显了对于此类特定应用而言，简单规则基础和模式匹配策略的相关性和效率。结果表明，在资源效率、可解释性以及领域特定适应能力至关重要的场景下，传统方法仍能提供显著优势。
URL
https://arxiv.org/abs/2503.11376
PDF
-
Weakly Supervised Contrastive Adversarial Training for Learning Robust Features from Semi-supervised Data
2025-03-14 03:01:10Lilin Zhang, Chengpei Wu, Ning YangAbstract
Existing adversarial training (AT) methods often suffer from incomplete perturbation, meaning that not all non-robust features are perturbed when generating adversarial examples (AEs). This results in residual correlations between non-robust features and labels, leading to suboptimal learning of robust features. However, achieving complete perturbation, i.e., perturbing as many non-robust features as possible, is challenging due to the difficulty in distinguishing robust and non-robust features and the sparsity of labeled data. To address these challenges, we propose a novel approach called Weakly Supervised Contrastive Adversarial Training (WSCAT). WSCAT ensures complete perturbation for improved learning of robust features by disrupting correlations between non-robust features and labels through complete AE generation over partially labeled data, grounded in information theory. Extensive theoretical analysis and comprehensive experiments on widely adopted benchmarks validate the superiority of WSCAT.
Abstract (translated)
现有的对抗训练（AT）方法常常受到不完全扰动的困扰，这意味着在生成对抗样本（AEs）时，并非所有的非鲁棒特征都会被扰动。这导致了非鲁棒特征与标签之间的残余相关性，从而导致对鲁棒特征的学习效果不佳。然而，实现完全扰动，即尽可能多地扰动非鲁棒特征，由于难以区分鲁棒和非鲁棒特征以及标注数据的稀疏性而变得具有挑战性。 为了解决这些挑战，我们提出了一种名为弱监督对比对抗训练（WSCAT）的新方法。通过在部分标记的数据上进行完全AE生成来打破非鲁棒特征与标签之间的相关性，并基于信息理论确保完全扰动以改善对鲁棒特征的学习效果，WSCAT能够应对这一难题。 广泛的理论分析和在广泛采用的基准上的全面实验验证了WSCAT的优越性。
URL
https://arxiv.org/abs/2503.11032
PDF
-
MACS: Multi-source Audio-to-image Generation with Contextual Significance and Semantic Alignment
2025-03-13 11:56:25Hao Zhou, Xiaobao Guo, Yuzhe Zhu, Adams Wai-Kin KongAbstract
Propelled by the breakthrough in deep generative models, audio-to-image generation has emerged as a pivotal cross-model task that converts complex auditory signals into rich visual representations. However, previous works only focus on single-source audio inputs for image generation, ignoring the multi-source characteristic in natural auditory scenes, thus limiting the performance in generating comprehensive visual content. To bridge this gap, a method called MACS is proposed to conduct multi-source audio-to-image generation. This is the first work that explicitly separates multi-source audio to capture the rich audio components before image generation. MACS is a two-stage method. In the first stage, multi-source audio inputs are separated by a weakly supervised method, where the audio and text labels are semantically aligned by casting into a common space using the large pre-trained CLAP model. We introduce a ranking loss to consider the contextual significance of the separated audio signals. In the second stage, efficient image generation is achieved by mapping the separated audio signals to the generation condition using only a trainable adapter and a MLP layer. We preprocess the LLP dataset as the first full multi-source audio-to-image generation benchmark. The experiments are conducted on multi-source, mixed-source, and single-source audio-to-image generation tasks. The proposed MACS outperforms the current state-of-the-art methods in 17 of the 21 evaluation indexes on all tasks and delivers superior visual quality. The code will be publicly available.
Abstract (translated)
深度生成模型的突破推动了音频到图像生成任务的发展，这一任务将复杂的听觉信号转换为丰富的视觉表示，并在跨模态任务中占据重要地位。然而，之前的研究仅关注单一来源音频输入的图像生成，忽略了自然听觉场景中的多源特性，从而限制了全面视觉内容生成的表现力。为了弥补这一不足，提出了一种名为MACS的方法来实现多源音频到图像的生成。这是第一个明确分离多源音频以捕捉丰富音频成分并在图像生成前进行处理的工作。 MACS是一种两阶段方法。在第一阶段中，使用弱监督方式将多源音频输入进行分离，并通过将音频和文本标签投射到一个共同空间内（利用大型预训练的CLAP模型）来实现语义对齐。我们引入了一个排名损失函数以考虑分离后音频信号中的上下文重要性。在第二阶段中，仅通过可训练适配器和MLP层映射分离后的音频信号到生成条件，从而实现了高效的图像生成。 我们将LLP数据集预处理为首个完整的多源音频到图像生成基准测试数据集，并在多源、混合源及单一来源的音频到图像生成任务上进行了实验。所提出的MACS方法在所有21个评估指标中的17项中超越了当前最先进的方法，同时提供了更优的视觉质量。代码将公开发布。
URL
https://arxiv.org/abs/2503.10287
PDF
-
WeakSupCon: Weakly Supervised Contrastive Learning for Encoder Pre-training
2025-03-06 07:25:43Bodong Zhang, Hamid Manoochehri, Beatrice S. Knudsen, Tolga TasdizenAbstract
Weakly supervised multiple instance learning (MIL) is a challenging task given that only bag-level labels are provided, while each bag typically contains multiple instances. This topic has been extensively studied in histopathological image analysis, where labels are usually available only at the whole slide image (WSI) level, while each whole slide image can be divided into thousands of small image patches for training. The dominant MIL approaches take fixed patch features as inputs to address computational constraints and ensure model stability. These features are commonly generated by encoders pre-trained on ImageNet, foundation encoders pre-trained on large datasets, or through self-supervised learning on local datasets. While the self-supervised encoder pre-training on the same dataset as downstream MIL tasks helps mitigate domain shift and generate better features, the bag-level labels are not utilized during the process, and the features of patches from different categories may cluster together, reducing classification performance on MIL tasks. Recently, pre-training with supervised contrastive learning (SupCon) has demonstrated superior performance compared to self-supervised contrastive learning and even end-to-end training on traditional image classification tasks. In this paper, we propose a novel encoder pre-training method for downstream MIL tasks called Weakly Supervised Contrastive Learning (WeakSupCon) that utilizes bag-level labels. In our method, we employ multi-task learning and define distinct contrastive learning losses for samples with different bag labels. Our experiments demonstrate that the features generated using WeakSupCon significantly enhance MIL classification performance compared to self-supervised approaches across three datasets.
Abstract (translated)
弱监督多实例学习（MIL）是一个具有挑战性的任务，因为仅提供了袋子级别的标签，而每个袋子通常包含多个实例。这一主题在组织病理图像分析领域得到了广泛研究，在该领域中，标签通常是针对整个切片图像（WSI）级别提供的，然而每个完整的切片可以被分割成数千个小图像块用于训练。目前主导的MIL方法使用固定的补丁特征作为输入以解决计算限制并确保模型稳定性。这些特征通常通过在ImageNet上预训练的编码器、大型数据集上的基础编码器或本地数据集上的自监督学习生成。 尽管在相同的数据集上进行自我监督编码器预训练有助于缓解领域转移，并生成更好的特征，但袋子级别的标签并未在此过程中被利用；来自不同类别的补丁可能聚集在一起，从而降低了MIL任务中的分类性能。最近，使用监督对比学习（SupCon）的预训练方法在传统图像分类任务上展示出优于自监督对比学习甚至端到端训练的表现。在本文中，我们提出了一种针对下游MIL任务的新颖编码器预训练方法——弱监督对比学习（WeakSupCon），这种方法利用了袋子级别的标签。我们的方法采用了多任务学习，并为具有不同袋子标签的样本定义了不同的对比学习损失。 实验结果表明，在三个数据集上使用WeakSupCon生成的特征显著提升了MIL分类性能，优于自监督方法。
URL
https://arxiv.org/abs/2503.04165
PDF
-
CA-W3D: Leveraging Context-Aware Knowledge for Weakly Supervised Monocular 3D Detection
2025-03-06 07:02:13Chupeng Liu, Runkai Zhao, Weidong CaiAbstract
Weakly supervised monocular 3D detection, while less annotation-intensive, often struggles to capture the global context required for reliable 3D reasoning. Conventional label-efficient methods focus on object-centric features, neglecting contextual semantic relationships that are critical in complex scenes. In this work, we propose a Context-Aware Weak Supervision for Monocular 3D object detection, namely CA-W3D, to address this limitation in a two-stage training paradigm. Specifically, we first introduce a pre-training stage employing Region-wise Object Contrastive Matching (ROCM), which aligns regional object embeddings derived from a trainable monocular 3D encoder and a frozen open-vocabulary 2D visual grounding model. This alignment encourages the monocular encoder to discriminate scene-specific attributes and acquire richer contextual knowledge. In the second stage, we incorporate a pseudo-label training process with a Dual-to-One Distillation (D2OD) mechanism, which effectively transfers contextual priors into the monocular encoder while preserving spatial fidelity and maintaining computational efficiency during inference. Extensive experiments conducted on the public KITTI benchmark demonstrate the effectiveness of our approach, surpassing the SoTA method over all metrics, highlighting the importance of contextual-aware knowledge in weakly-supervised monocular 3D detection.
Abstract (translated)
弱监督单目3D检测虽然减少了标注需求，但在捕捉可靠3D推理所需的全局上下文方面往往表现不佳。传统的标签高效方法侧重于对象中心特征，忽视了复杂场景中至关重要的语义关系。为此，我们提出了一种针对单目3D物体检测的上下文感知弱监督方法（Context-Aware Weak Supervision for Monocular 3D object detection），简称CA-W3D，在两阶段训练范式下解决这一局限性。 具体来说，我们在预训练阶段引入了基于区域对象对比匹配（Region-wise Object Contrastive Matching, ROCM）的方法。该方法使可调单目3D编码器与固定的开放词汇2D视觉定位模型中提取的区域对象嵌入对齐。这种对齐鼓励单目编码器区分场景特定属性，并获得更丰富的上下文知识。 在第二阶段，我们采用了一种伪标签训练过程，结合了双向到单一蒸馏机制（Dual-to-One Distillation, D2OD），该机制有效将上下文先验传递给单目编码器，同时保持空间精度和推理时的计算效率。 我们在公共KITTI基准上进行了广泛的实验，验证了我们方法的有效性，在所有指标上超越了现有最佳方法。这突显了在弱监督下的单目3D检测中，上下文感知知识的重要性。
URL
https://arxiv.org/abs/2503.04154
PDF
-
Cross-modal Causal Relation Alignment for Video Question Grounding
2025-03-05 01:36:32Weixing Chen, Yang Liu, Binglin Chen, Jiandong Su, Yongsen Zheng, Liang LinAbstract
Video question grounding (VideoQG) requires models to answer the questions and simultaneously infer the relevant video segments to support the answers. However, existing VideoQG methods usually suffer from spurious cross-modal correlations, leading to a failure to identify the dominant visual scenes that align with the intended question. Moreover, vision-language models exhibit unfaithful generalization performance and lack robustness on challenging downstream tasks such as VideoQG. In this work, we propose a novel VideoQG framework named Cross-modal Causal Relation Alignment (CRA), to eliminate spurious correlations and improve the causal consistency between question-answering and video temporal grounding. Our CRA involves three essential components: i) Gaussian Smoothing Grounding (GSG) module for estimating the time interval via cross-modal attention, which is de-noised by an adaptive Gaussian filter, ii) Cross-Modal Alignment (CMA) enhances the performance of weakly supervised VideoQG by leveraging bidirectional contrastive learning between estimated video segments and QA features, iii) Explicit Causal Intervention (ECI) module for multimodal deconfounding, which involves front-door intervention for vision and back-door intervention for language. Extensive experiments on two VideoQG datasets demonstrate the superiority of our CRA in discovering visually grounded content and achieving robust question reasoning. Codes are available at this https URL.
Abstract (translated)
视频问题定位（Video Question Grounding，简称VideoQG）要求模型能够回答问题并同时推断出支持答案的相关视频片段。然而，现有的VideoQG方法通常会遇到虚假跨模态相关性的问题，导致难以识别与预期问题一致的主要视觉场景。此外，视觉-语言模型在诸如VideoQG等具有挑战性的下游任务中表现出泛化性能不佳且不够稳健的特点。 针对这些问题，在这项工作中我们提出了一种名为“跨模态因果关系对齐”（Cross-modal Causal Relation Alignment, CRA）的新框架，旨在消除虚假相关性，并提高问题回答与视频时间定位之间的因果一致性。CRA包括三个核心组件： 1. **高斯平滑接地(Gaussian Smoothing Grounding, GSG)模块**：通过跨模态注意力机制估计时间段，并利用自适应的高斯滤波器进行去噪。 2. **跨模态对齐(Cross-Modal Alignment, CMA)**：通过视频片段和问答特征之间的双向对比学习，增强弱监督VideoQG的表现。 3. **明确因果干预(Explicit Causal Intervention, ECI)模块**：用于多模态的反混淆处理，包括视觉的前门干预以及语言的后门干预。 在两个VideoQG数据集上的广泛实验表明，我们的CRA框架在发现视觉接地内容和实现稳健的问题推理方面具有显著优势。相关代码可以在提供的URL地址获取。
URL
https://arxiv.org/abs/2503.07635
PDF
-
Learning from Noisy Labels with Contrastive Co-Transformer
2025-03-04 22:48:43Yan Han, Soumava Kumar Roy, Mehrtash Harandi, Lars PeterssonAbstract
Deep learning with noisy labels is an interesting challenge in weakly supervised learning. Despite their significant learning capacity, CNNs have a tendency to overfit in the presence of samples with noisy labels. Alleviating this issue, the well known Co-Training framework is used as a fundamental basis for our work. In this paper, we introduce a Contrastive Co-Transformer framework, which is simple and fast, yet able to improve the performance by a large margin compared to the state-of-the-art approaches. We argue the robustness of transformers when dealing with label noise. Our Contrastive Co-Transformer approach is able to utilize all samples in the dataset, irrespective of whether they are clean or noisy. Transformers are trained by a combination of contrastive loss and classification loss. Extensive experimental results on corrupted data from six standard benchmark datasets including Clothing1M, demonstrate that our Contrastive Co-Transformer is superior to existing state-of-the-art methods.
Abstract (translated)
带有噪声标签的深度学习是弱监督学习中的一个有趣挑战。尽管卷积神经网络（CNNs）具有强大的学习能力，但在存在带噪标签样本的情况下，它们倾向于过拟合。为缓解这一问题，本工作以著名的Co-Training框架作为基础。在本文中，我们引入了一种对比共变换器（Contrastive Co-Transformer）框架，该框架简单且快速，能够大幅优于现有方法的表现。我们认为，当处理标签噪声时，变压器具有鲁棒性。我们的对比共变换器方法可以利用数据集中所有的样本，无论它们是干净的还是带噪的。通过结合对比损失和分类损失，对变压器进行训练。在六个标准基准数据集（包括Clothing1M）上的大量实验结果表明，我们提出的对比共变换器优于现有的最先进方法。
URL
https://arxiv.org/abs/2503.03042
PDF
-
Multimodal Distillation-Driven Ensemble Learning for Long-Tailed Histopathology Whole Slide Images Analysis
2025-03-02 14:31:45Xitong Ling, Yifeng Ping, Jiawen Li, Jing Peng, Yuxuan Chen, Minxi Ouyang, Yizhi Wang, Yonghong He, Tian Guan, Xiaoping Liu, Lianghui ZhuAbstract
Multiple Instance Learning (MIL) plays a significant role in computational pathology, enabling weakly supervised analysis of Whole Slide Image (WSI) datasets. The field of WSI analysis is confronted with a severe long-tailed distribution problem, which significantly impacts the performance of classifiers. Long-tailed distributions lead to class imbalance, where some classes have sparse samples while others are abundant, making it difficult for classifiers to accurately identify minority class samples. To address this issue, we propose an ensemble learning method based on MIL, which employs expert decoders with shared aggregators and consistency constraints to learn diverse distributions and reduce the impact of class imbalance on classifier performance. Moreover, we introduce a multimodal distillation framework that leverages text encoders pre-trained on pathology-text pairs to distill knowledge and guide the MIL aggregator in capturing stronger semantic features relevant to class information. To ensure flexibility, we use learnable prompts to guide the distillation process of the pre-trained text encoder, avoiding limitations imposed by specific prompts. Our method, MDE-MIL, integrates multiple expert branches focusing on specific data distributions to address long-tailed issues. Consistency control ensures generalization across classes. Multimodal distillation enhances feature extraction. Experiments on Camelyon+-LT and PANDA-LT datasets show it outperforms state-of-the-art methods.
Abstract (translated)
多重实例学习（MIL）在计算病理学中扮演着重要角色，它能够对整个滑动图像（WSI）数据集进行弱监督分析。然而，WSI 分析领域面临着严重的长尾分布问题，这对分类器的性能产生了显著影响。长尾分布导致类别不平衡的问题，在这个问题中，某些类别的样本非常稀少而另一些类别则异常丰富，这使得分类器难以准确地识别少数类样本。为了解决这一问题，我们提出了一种基于MIL的集成学习方法，该方法使用共享聚合器和一致性的专家解码器来学习多样的分布，并减少类别不平衡对分类器性能的影响。 此外，我们还引入了一个多模态知识蒸馏框架，利用预先在病理学图像-文本对上训练的文本编码器来提取并指导MIL聚合器捕获与类信息相关的更强大的语义特征。为了确保灵活性，我们在预训练的文本编码器的知识蒸馏过程中使用了可学习提示，以避免特定提示的限制。 我们的方法 MDE-MIL 集成了多个专注于特定数据分布的专家分支来解决长尾问题。一致性控制保证了跨类别的泛化能力，而多模态知识蒸馏则增强了特征提取能力。实验结果表明，在Camelyon+-LT和PANDA-LT数据集上的测试中，我们的方法超过了现有的最先进的方法。
URL
https://arxiv.org/abs/2503.00915
PDF
-
Weakly Supervised Multiple Instance Learning for Whale Call Detection and Localization in Long-Duration Passive Acoustic Monitoring
2025-02-28 08:34:12Ragib Amin Nihal, Benjamin Yen, Runwu Shi, Kazuhiro NakadaiAbstract
Marine ecosystem monitoring via Passive Acoustic Monitoring (PAM) generates vast data, but deep learning often requires precise annotations and short segments. We introduce DSMIL-LocNet, a Multiple Instance Learning framework for whale call detection and localization using only bag-level labels. Our dual-stream model processes 2-30 minute audio segments, leveraging spectral and temporal features with attention-based instance selection. Tests on Antarctic whale data show longer contexts improve classification (F1: 0.8-0.9) while medium instances ensure localization precision (0.65-0.70). This suggests MIL can enhance scalable marine monitoring. Code: this https URL
Abstract (translated)
通过被动声学监测（PAM）进行海洋生态系统监测会产生大量数据，但深度学习通常需要精确的注释和较短的数据片段。我们介绍了DSMIL-LocNet框架，这是一个用于仅使用袋级标签检测和定位鲸鱼叫声的多实例学习方法。我们的双流模型处理2至30分钟的音频段，并利用谱图和时序特征以及基于注意力机制的选择实例进行处理。在南极鲸鱼数据上的测试表明，更长的时间上下文可以改善分类性能（F1：0.8-0.9），而中等长度的数据片段则确保了定位精度（0.65-0.70）。这表明多实例学习可以增强海洋监测的可扩展性。代码位于此链接：[此处插入实际链接]
URL
https://arxiv.org/abs/2502.20838
PDF
