Weakly_Supervised
-
Weakly Supervised Object Segmentation by Background Conditional Divergence
2025-06-25 16:46:46Hassan Baker, Matthew S. Emigh, Austin J. BrockmeierAbstract
As a computer vision task, automatic object segmentation remains challenging in specialized image domains without massive labeled data, such as synthetic aperture sonar images, remote sensing, biomedical imaging, etc. In any domain, obtaining pixel-wise segmentation masks is expensive. In this work, we propose a method for training a masking network to perform binary object segmentation using weak supervision in the form of image-wise presence or absence of an object of interest, which provides less information but may be obtained more quickly from manual or automatic labeling. A key step in our method is that the segmented objects can be placed into background-only images to create realistic, images of the objects with counterfactual backgrounds. To create a contrast between the original and counterfactual background images, we propose to first cluster the background-only images, and then during learning create counterfactual images that blend objects segmented from their original source backgrounds to backgrounds chosen from a targeted cluster. One term in the training loss is the divergence between these counterfactual images and the real object images with backgrounds of the target cluster. The other term is a supervised loss for background-only images. While an adversarial critic could provide the divergence, we use sample-based divergences. We conduct experiments on side-scan and synthetic aperture sonar in which our approach succeeds compared to previous unsupervised segmentation baselines that were only tested on natural images. Furthermore, to show generality we extend our experiments to natural images, obtaining reasonable performance with our method that avoids pretrained networks, generative networks, and adversarial critics. The basecode for this work can be found at \href{GitHub}{this https URL}.
Abstract (translated)
作为计算机视觉任务，自动对象分割在缺乏大规模标注数据的专业图像领域中仍然具有挑战性，例如合成孔径声纳图像、遥感和生物医学成像等。在任何领域中，获取像素级的分割掩模都是昂贵的。在这项工作中，我们提出了一种方法，通过弱监督（以目标对象在图像中的存在与否的形式）来训练遮罩网络进行二元物体分割。这种形式提供的信息较少，但可以从手动或自动标记中更快地获得。 我们方法的关键步骤是将分割出的对象放置到仅包含背景的图片中，创建具有反事实背景的目标对象的真实图像。为了在原始和反事实背景图像之间形成对比，我们首先对仅含背景的图像进行聚类，在学习过程中生成混合来自目标聚类背景中的反事实图像。这些反事实图像是通过将从它们原出处背景分割出来的物体与从特定簇中选取的背景融合而来的。 训练损失的一个项是这些反事实图像与其具有目标群集背景的真实对象图像之间的散度。另一个项则是仅含背景图像的监督损失。尽管对抗性批评者可以提供这种散度，但我们使用基于样本的散度。我们在侧扫声纳和合成孔径声纳上进行了实验，在这些领域中我们的方法优于以往仅在自然图像上测试过的无监督分割基准模型。 为了展示其通用性，我们将实验扩展到自然图像，并且无需预训练网络、生成式网络或对抗性批评者即可获得合理性能。这项工作的基础代码可以在GitHub上的[此链接](https://github.com/your-repo-url-here)找到。
URL
https://arxiv.org/abs/2506.22505
PDF
-
Improving Weakly Supervised Temporal Action Localization by Exploiting Multi-resolution Information in Temporal Domain
2025-06-23 03:20:18Rui Su, Dong Xu, Luping Zhou, Wanli OuyangAbstract
Weakly supervised temporal action localization is a challenging task as only the video-level annotation is available during the training process. To address this problem, we propose a two-stage approach to fully exploit multi-resolution information in the temporal domain and generate high quality frame-level pseudo labels based on both appearance and motion streams. Specifically, in the first stage, we generate reliable initial frame-level pseudo labels, and in the second stage, we iteratively refine the pseudo labels and use a set of selected frames with highly confident pseudo labels to train neural networks and better predict action class scores at each frame. We fully exploit temporal information at multiple scales to improve temporal action localization performance. Specifically, in order to obtain reliable initial frame-level pseudo labels, in the first stage, we propose an Initial Label Generation (ILG) module, which leverages temporal multi-resolution consistency to generate high quality class activation sequences (CASs), which consist of a number of sequences with each sequence measuring how likely each video frame belongs to one specific action class. In the second stage, we propose a Progressive Temporal Label Refinement (PTLR) framework. In our PTLR framework, two networks called Network-OTS and Network-RTS, which are respectively used to generate CASs for the original temporal scale and the reduced temporal scales, are used as two streams (i.e., the OTS stream and the RTS stream) to refine the pseudo labels in turn. By this way, the multi-resolution information in the temporal domain is exchanged at the pseudo label level, and our work can help improve each stream (i.e., the OTS/RTS stream) by exploiting the refined pseudo labels from another stream (i.e., the RTS/OTS stream).
Abstract (translated)
弱监督时间动作定位是一项具有挑战性的任务，因为在训练过程中只有视频级别的注释可用。为了解决这个问题，我们提出了一种两阶段的方法来充分利用时域中的多分辨率信息，并基于外观和运动流生成高质量的帧级伪标签。具体来说，在第一阶段中，我们生成可靠的初始帧级伪标签；在第二阶段中，我们迭代地细化这些伪标签，并使用一组具有高置信度伪标签的选择性帧训练神经网络，以更好地预测每个帧的动作类别得分。 为了充分利用时域中的多分辨率信息来提高时间动作定位性能，我们在第一阶段提出了一个初始标签生成（ILG）模块。该模块利用了时间多分辨率一致性来生成高质量的类激活序列（CAS），这些序列由多个序列组成，每个序列衡量视频中每一帧属于特定动作类别的可能性。 在第二阶段，我们提出了一种渐进式时间标签细化（PTLR）框架，在此框架中使用两个称为Network-OTS和Network-RTS的网络分别生成原始时间和减少的时间尺度上的CAS。这两个网络作为两条流（即OTC流和RTC流），交替地用来细化伪标签。 通过这种方式，时域中的多分辨率信息以伪标签的形式进行交换，我们的工作可以通过利用另一条流中经过精炼的伪标签来改进每一条流（即OTC/RTC流）。
URL
https://arxiv.org/abs/2506.18261
PDF
-
CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images
2025-06-22 14:02:27Dongdong Meng, Sheng Li, Hao Wu, Suqing Tian, Wenjun Ma, Guoping Wang, Xueqing YanAbstract
Accurate automatic medical image segmentation relies on high-quality, dense annotations, which are costly and time-consuming. Weakly supervised learning provides a more efficient alternative by leveraging sparse and coarse annotations instead of dense, precise ones. However, segmentation performance degradation and overfitting caused by sparse annotations remain key challenges. To address these issues, we propose CmFNet, a novel 3D weakly supervised cross-modal medical image segmentation approach. CmFNet consists of three main components: a modality-specific feature learning network, a cross-modal feature learning network, and a hybrid-supervised learning strategy. Specifically, the modality-specific feature learning network and the cross-modal feature learning network effectively integrate complementary information from multi-modal images, enhancing shared features across modalities to improve segmentation performance. Additionally, the hybrid-supervised learning strategy guides segmentation through scribble supervision, intra-modal regularization, and inter-modal consistency, modeling spatial and contextual relationships while promoting feature alignment. Our approach effectively mitigates overfitting, delivering robust segmentation results. It excels in segmenting both challenging small tumor regions and common anatomical structures. Extensive experiments on a clinical cross-modal nasopharyngeal carcinoma (NPC) dataset (including CT and MR imaging) and the publicly available CT Whole Abdominal Organ dataset (WORD) show that our approach outperforms state-of-the-art weakly supervised methods. In addition, our approach also outperforms fully supervised methods when full annotation is used. Our approach can facilitate clinical therapy and benefit various specialists, including physicists, radiologists, pathologists, and oncologists.
Abstract (translated)
准确的自动医学图像分割依赖于高质量、密集的标注，这既耗时又昂贵。弱监督学习通过利用稀疏和粗略的标注提供了一种更高效的替代方案，而不是使用密集且精确的标注。然而，由于稀疏标注导致的分割性能下降以及过拟合问题仍然是关键挑战。为了解决这些问题，我们提出了CmFNet——一种新颖的3D弱监督跨模态医学图像分割方法。CmFNet由三个主要组成部分组成：特定模态特征学习网络、跨模态特征学习网络和混合监督学习策略。 具体来说，特定模态特征学习网络和跨模态特征学习网络有效整合了多模态图像中的互补信息，增强各模态之间的共享特性，从而提高分割性能。此外，混合监督学习策略通过涂鸦监督（scribble supervision）、同模态正则化（intra-modal regularization）以及异模态一致性（inter-modal consistency），建模空间和上下文关系，并促进特征对齐。我们的方法有效地缓解了过拟合问题，提供了鲁棒的分割结果。 它在分割具有挑战性的微小肿瘤区域和常见解剖结构方面表现出色。通过临床跨模态鼻咽癌（NPC）数据集（包括CT和MR成像）以及公开可用的CT全腹部器官数据集（WORD）进行广泛的实验表明，我们的方法优于现有的弱监督方法。此外，在使用完整标注时，我们的方法也超越了完全监督的方法。 该方法可以促进临床治疗，并有利于物理学家、放射科医生、病理学家和肿瘤学家等各专业人员。
URL
https://arxiv.org/abs/2506.18042
PDF
-
Class Agnostic Instance-level Descriptor for Visual Instance Search
2025-06-20 04:32:39Qi-Ying Sun, Wan-Lei Zhao, Yi-Bo Miao, Chong-Wah NgoAbstract
Despite the great success of the deep features in content-based image retrieval, the visual instance search remains challenging due to the lack of effective instance level feature representation. Supervised or weakly supervised object detection methods are not among the options due to their poor performance on the unknown object categories. In this paper, based on the feature set output from self-supervised ViT, the instance level region discovery is modeled as detecting the compact feature subsets in a hierarchical fashion. The hierarchical decomposition results in a hierarchy of feature subsets. The non-leaf nodes and leaf nodes on the hierarchy correspond to the various instance regions in an image of different semantic scales. The hierarchical decomposition well addresses the problem of object embedding and occlusions, which are widely observed in the real scenarios. The features derived from the nodes on the hierarchy make up a comprehensive representation for the latent instances in the image. Our instance-level descriptor remains effective on both the known and unknown object categories. Empirical studies on three instance search benchmarks show that it outperforms state-of-the-art methods considerably.
Abstract (translated)
尽管基于内容的图像检索中的深度特征取得了巨大成功，但由于缺乏有效的实例级别特征表示，视觉实例搜索仍然具有挑战性。由于在未知对象类别上的表现不佳，监督或弱监督的对象检测方法并不适用。本文基于自监督ViT输出的特征集，将实例级别的区域发现建模为以分层方式检测紧凑的特征子集的过程。这种层次化分解产生了一系列特征子集层级结构。该层级结构中的非叶节点和叶节点对应于图像中不同语义尺度的各种实例区域。这种层次化分解很好地解决了对象嵌入和遮挡的问题，这些问题在现实场景中普遍存在。从这些层级上的节点派生的特征构成了图像中潜在实例的整体表示。我们的实例级别描述符对已知和未知的对象类别都保持有效。在三个实例搜索基准数据集上的实证研究表明，它显著优于现有方法。
URL
https://arxiv.org/abs/2506.16745
PDF
-
CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision
2025-06-17 18:44:21Dyah Adila, Shuai Zhang, Boran Han, Bonan Min, Yuyang WangAbstract
The integration of contextual information has significantly enhanced the performance of large language models (LLMs) on knowledge-intensive tasks. However, existing methods often overlook a critical challenge: the credibility of context documents can vary widely, potentially leading to the propagation of unreliable information. In this paper, we introduce CrEst, a novel weakly supervised framework for assessing the credibility of context documents during LLM inference--without requiring manual annotations. Our approach is grounded in the insight that credible documents tend to exhibit higher semantic coherence with other credible documents, enabling automated credibility estimation through inter-document agreement. To incorporate credibility into LLM inference, we propose two integration strategies: a black-box approach for models without access to internal weights or activations, and a white-box method that directly modifies attention mechanisms. Extensive experiments across three model architectures and five datasets demonstrate that CrEst consistently outperforms strong baselines, achieving up to a 26.86% improvement in accuracy and a 3.49% increase in F1 score. Further analysis shows that CrEst maintains robust performance even under high-noise conditions.
Abstract (translated)
上下文信息的整合已显著提升了大型语言模型（LLM）在知识密集型任务上的表现。然而，现有方法往往忽视了一个关键挑战：上下文文档的可信度可能存在巨大差异，这可能导致不实信息的传播。本文介绍了CrEst，这是一种新颖的弱监督框架，在LLM推理过程中评估上下文文档的可信度——无需手动标注。我们的方法基于这样一个观点：可信的文档通常与其他可信文档在语义上更为一致，从而可以通过文档间的一致性实现自动化的可信度估计。 为了将可信度整合到LLM推理中，我们提出了两种集成策略：一种是黑盒方法，适用于无法访问内部权重或激活值的情况；另一种是白盒方法，直接修改注意力机制。在三种模型架构和五个数据集上的广泛实验表明，CrEst始终优于强大的基线方法，在准确性方面最多提高了26.86%，F1分数则提升了3.49%。进一步的分析显示，即使在高噪声条件下，CrEst也能保持其稳健的表现。
URL
https://arxiv.org/abs/2506.14912
PDF
-
Learning Event Completeness for Weakly Supervised Video Anomaly Detection
Abstract
Weakly supervised video anomaly detection (WS-VAD) is tasked with pinpointing temporal intervals containing anomalous events within untrimmed videos, utilizing only video-level annotations. However, a significant challenge arises due to the absence of dense frame-level annotations, often leading to incomplete localization in existing WS-VAD methods. To address this issue, we present a novel LEC-VAD, Learning Event Completeness for Weakly Supervised Video Anomaly Detection, which features a dual structure designed to encode both category-aware and category-agnostic semantics between vision and language. Within LEC-VAD, we devise semantic regularities that leverage an anomaly-aware Gaussian mixture to learn precise event boundaries, thereby yielding more complete event instances. Besides, we develop a novel memory bank-based prototype learning mechanism to enrich concise text descriptions associated with anomaly-event categories. This innovation bolsters the text's expressiveness, which is crucial for advancing WS-VAD. Our LEC-VAD demonstrates remarkable advancements over the current state-of-the-art methods on two benchmark datasets XD-Violence and UCF-Crime.
Abstract (translated)
弱监督视频异常检测（WS-VAD）的任务是在未经修剪的视频中识别包含异常事件的时间区间，仅使用视频级别的注释信息。然而，由于缺乏密集的帧级别注释，现有WS-VAD方法通常会遇到定位不完整的问题。为了解决这个问题，我们提出了一种新的LECVAD (Learning Event Completeness for Weakly Supervised Video Anomaly Detection)，该方法采用双层结构来编码视觉和语言之间的类别感知和非类别感知语义信息。在LECVAD中，我们设计了利用异常意识高斯混合体来学习精确事件边界的语义规律，从而提供更完整的事件实例。此外，我们开发了一种基于记忆库的原型学习机制，用于丰富与异常事件类别相关的简洁文本描述。这一创新增强了文本的表现力，在推进WS-VAD方面至关重要。我们的LECVAD在两个基准数据集（XD-Violence和UCF-Crime）上展示了相对于当前最先进方法的重大进步。
URL
https://arxiv.org/abs/2506.13095
PDF
-
Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models
2025-06-12 12:19:18Konstantinos Vilouras, Ilias Stogiannidis, Junyu Yan, Alison Q. O'Neil, Sotirios A. TsaftarisAbstract
Latent Diffusion Models have shown remarkable results in text-guided image synthesis in recent years. In the domain of natural (RGB) images, recent works have shown that such models can be adapted to various vision-language downstream tasks with little to no supervision involved. On the contrary, text-to-image Latent Diffusion Models remain relatively underexplored in the field of medical imaging, primarily due to limited data availability (e.g., due to privacy concerns). In this work, focusing on the chest X-ray modality, we first demonstrate that a standard text-conditioned Latent Diffusion Model has not learned to align clinically relevant information in free-text radiology reports with the corresponding areas of the given scan. Then, to alleviate this issue, we propose a fine-tuning framework to improve multi-modal alignment in a pre-trained model such that it can be efficiently repurposed for downstream tasks such as phrase grounding. Our method sets a new state-of-the-art on a standard benchmark dataset (MS-CXR), while also exhibiting robust performance on out-of-distribution data (VinDr-CXR). Our code will be made publicly available.
Abstract (translated)
近年来，潜在扩散模型（Latent Diffusion Models）在文本引导的图像合成领域取得了显著成果。在自然图像（RGB）领域，最近的研究表明，这类模型可以在几乎没有或不需要监督的情况下适应各种视觉-语言下游任务。然而，在医学成像领域，基于文本到图像的潜在扩散模型研究相对较少，主要原因是可用数据有限（例如，由于隐私问题）。在这项工作中，我们专注于胸部X光影像模式，并首先展示了标准的条件文本的潜在扩散模型未能学会将放射学报告中的临床相关信息与给定扫描中的相应区域对齐。为了解决这个问题，我们提出了一种微调框架，以改进预训练模型中的多模态对齐，使其可以高效地重新应用于下游任务（如短语定位）。我们的方法在标准基准数据集（MS-CXR）上建立了新的最先进水平，并且在分布外数据（VinDr-CXR）上也表现出稳健的性能。我们将公开发布代码。
URL
https://arxiv.org/abs/2506.10633
PDF
-
Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework
2025-06-12 03:33:46Sadia Kamal, Tim Oates, Joy WanAbstract
Skin carcinoma is the most prevalent form of cancer globally, accounting for over $8 billion in annual healthcare expenditures. In clinical settings, physicians document patient visits using detailed SOAP (Subjective, Objective, Assessment, and Plan) notes. However, manually generating these notes is labor-intensive and contributes to clinician burnout. In this work, we propose a weakly supervised multimodal framework to generate clinically structured SOAP notes from limited inputs, including lesion images and sparse clinical text. Our approach reduces reliance on manual annotations, enabling scalable, clinically grounded documentation while alleviating clinician burden and reducing the need for large annotated data. Our method achieves performance comparable to GPT-4o, Claude, and DeepSeek Janus Pro across key clinical relevance metrics. To evaluate clinical quality, we introduce two novel metrics MedConceptEval and Clinical Coherence Score (CCS) which assess semantic alignment with expert medical concepts and input features, respectively.
Abstract (translated)
皮肤癌是全球最常见的癌症类型，每年的医疗保健支出超过80亿美元。在临床环境中，医生使用详细的SOAP（主诉、客观检查、评估和计划）笔记来记录患者的就诊情况。然而，手动生成这些笔记耗时且繁重，导致了医师的职业倦怠。在这项工作中，我们提出了一种弱监督多模态框架，可以从有限的输入数据中自动生成结构化的临床SOAP笔记，包括病变图像和稀疏的临床文本信息。我们的方法减少了对人工标注的依赖，实现了可扩展、基于临床实践的文档记录，并减轻了医生的工作负担以及减少大规模注释数据的需求。我们的方法在关键临床相关性指标上达到了与GPT-4o、Claude和DeepSeek Janus Pro相当的表现水平。为了评估临床质量，我们引入了两个新颖的评价指标：MedConceptEval 和临床连贯度评分（CCS），分别用于评估语义与专家医学概念以及输入特征的一致性。
URL
https://arxiv.org/abs/2506.10328
PDF
-
Conditional diffusion models for guided anomaly detection in brain images using fluid-driven anomaly randomization
2025-06-11 23:27:00Ana Lawry Aguila, Peirong Liu, Oula Puonti, Juan Eugenio IglesiasAbstract
Supervised machine learning has enabled accurate pathology detection in brain MRI, but requires training data from diseased subjects that may not be readily available in some scenarios, for example, in the case of rare diseases. Reconstruction-based unsupervised anomaly detection, in particular using diffusion models, has gained popularity in the medical field as it allows for training on healthy images alone, eliminating the need for large disease-specific cohorts. These methods assume that a model trained on normal data cannot accurately represent or reconstruct anomalies. However, this assumption often fails with models failing to reconstruct healthy tissue or accurately reconstruct abnormal regions i.e., failing to remove anomalies. In this work, we introduce a novel conditional diffusion model framework for anomaly detection and healthy image reconstruction in brain MRI. Our weakly supervised approach integrates synthetically generated pseudo-pathology images into the modeling process to better guide the reconstruction of healthy images. To generate these pseudo-pathologies, we apply fluid-driven anomaly randomization to augment real pathology segmentation maps from an auxiliary dataset, ensuring that the synthetic anomalies are both realistic and anatomically coherent. We evaluate our model's ability to detect pathology, using both synthetic anomaly datasets and real pathology from the ATLAS dataset. In our extensive experiments, our model: (i) consistently outperforms variational autoencoders, and conditional and unconditional latent diffusion; and (ii) surpasses on most datasets, the performance of supervised inpainting methods with access to paired diseased/healthy images.
Abstract (translated)
监督式机器学习已经在脑部MRI中实现了准确的病理检测，但需要来自患病个体的数据进行训练，在某些情况下这些数据可能难以获得，例如在罕见病的情况下。基于重建的无监督异常检测方法，特别是使用扩散模型的方法，在医学领域变得越来越受欢迎，因为它允许仅通过健康图像进行训练，从而消除了对特定疾病的大规模患者群体的需求。然而，这种方法假设一个经过正常数据训练的模型无法准确表示或重构异常情况，这在实践中经常不成立，因为模型可能无法精确地重建健康的组织或者准确地重构异常区域。 在这项工作中，我们引入了一个新颖的条件扩散模型框架用于脑部MRI中的异常检测和健康图像重构。我们的弱监督方法将通过合成生成伪病理图像来更好地指导健康图像的重构过程。为了生成这些伪病理图像，我们将流体驱动的异常随机化应用于辅助数据集中真实病理分割图，确保合成的异常既现实又解剖结构上一致。 我们在使用合成异常数据集和ATLAS数据集中的真实病理学的情况下评估了我们模型的异常检测能力。在广泛的实验中，我们的模型：（i）始终优于变分自编码器、条件和非条件潜在扩散方法；并且（ii）在大多数数据集中超过了那些可以访问患病/健康图像配对的监督式图像修复方法的表现。 通过这种方式，该研究不仅提高了无监督异常检测技术的有效性，还为处理罕见疾病的MRI分析提供了新的途径。
URL
https://arxiv.org/abs/2506.10233
PDF
-
TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
2025-06-11 06:52:31Ayush Gupta, Anirban Roy, Rama Chellappa, Nathaniel D. Bastian, Alvaro Velasquez, Susmit JhaAbstract
We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup, without any temporal annotations. Given a video and a question, we generate an open-ended answer grounded with the start and end time. For this task, we propose TOGA: a vision-language model for Temporally Grounded Open-Ended Video QA with Weak Supervision. We instruct-tune TOGA to jointly generate the answer and the temporal grounding. We operate in a weakly supervised setup where the temporal grounding annotations are not available. We generate pseudo labels for temporal grounding and ensure the validity of these labels by imposing a consistency constraint between the question of a grounding response and the response generated by a question referring to the same temporal segment. We notice that jointly generating the answers with the grounding improves performance on question answering as well as grounding. We evaluate TOGA on grounded QA and open-ended QA tasks. For grounded QA, we consider the NExT-GQA benchmark which is designed to evaluate weakly supervised grounded question answering. For open-ended QA, we consider the MSVD-QA and ActivityNet-QA benchmarks. We achieve state-of-the-art performance for both tasks on these benchmarks.
Abstract (translated)
我们解决的是在弱监督环境下，视频问答（video QA）中的时间定位问题，而无需任何时间标注。给定一个视频和一个问题时，我们会生成基于开始时间和结束时间的开放式答案。为了解决这个任务，我们提出了TOGA：一种用于弱监督下的、具有时间定位的开放性视频问答的视觉-语言模型(Temporally Grounded Open-Ended Video QA)。通过指令微调，TOGA能够同时生成答案和时间定位信息。我们在时间标注不可用的弱监督环境中操作，并为时间定位生成伪标签，以确保这些标签的有效性，我们通过对同一时间段的问题回答之间的一致性约束来实现这一点。 我们注意到，与单独生成答案或时间定位相比，同时生成二者可以提高问答及定位性能。我们在有依据的问答和开放式问答任务上评估TOGA的表现。对于基于证据的问答，我们考虑NExT-GQA基准测试，该测试旨在评估弱监督下的基于证据的问题回答能力。而对于开放性问答，我们参考MSVD-QA和ActivityNet-QA这两个基准。在这些基准测试中，我们的方法在两项任务上均取得了最先进的性能表现。
URL
https://arxiv.org/abs/2506.09445
PDF
-
Do MIL Models Transfer?
2025-06-10 17:50:04Daniel Shao, Richard J. Chen, Andrew H. Song, Joel Runevic, Ming Y. Lu, Tong Ding, Faisal MahmoodAbstract
Multiple Instance Learning (MIL) is a cornerstone approach in computational pathology (CPath) for generating clinically meaningful slide-level embeddings from gigapixel tissue images. However, MIL often struggles with small, weakly supervised clinical datasets. In contrast to fields such as NLP and conventional computer vision, where transfer learning is widely used to address data scarcity, the transferability of MIL models remains poorly understood. In this study, we systematically evaluate the transfer learning capabilities of pretrained MIL models by assessing 11 models across 21 pretraining tasks for morphological and molecular subtype prediction. Our results show that pretrained MIL models, even when trained on different organs than the target task, consistently outperform models trained from scratch. Moreover, pretraining on pancancer datasets enables strong generalization across organs and tasks, outperforming slide foundation models while using substantially less pretraining data. These findings highlight the robust adaptability of MIL models and demonstrate the benefits of leveraging transfer learning to boost performance in CPath. Lastly, we provide a resource which standardizes the implementation of MIL models and collection of pretrained model weights on popular CPath tasks, available at this https URL
Abstract (translated)
多示例学习（MIL）是计算病理学（CPath）中的一个基石方法，用于从数十亿像素的组织图像中生成具有临床意义的幻灯片级别的嵌入。然而，在小规模、弱监督的临床数据集上，MIL常常面临挑战。与自然语言处理（NLP）和传统计算机视觉等领域不同，这些领域广泛使用迁移学习来解决数据稀缺问题，关于多示例学习模型的可迁移性知之甚少。 在本研究中，我们通过评估11个预训练模型在21项预训练任务中的表现来系统地评价了预训练MIL模型的迁移学习能力。这些任务涵盖形态和分子亚型预测。我们的结果显示，即使是在与目标任务不同的器官上进行训练，预训练过的MIL模型也始终优于从头开始训练的模型。此外，在泛癌症数据集上的预训练能够实现跨不同器官和任务的强大泛化性能，并且在使用显著较少的预训练数据的情况下超过了幻灯片基础模型。 这些发现强调了MIL模型的稳健适应性，展示了利用迁移学习提高CPath中性能的好处。最后，我们提供了一个资源，用于标准化流行CPath任务中的多示例学习模型实现和收集预训练模型权重，可在提供的网址上获取：[此链接]（请确保将实际URL替换为具体的可用链接）。
URL
https://arxiv.org/abs/2506.09022
PDF
-
FMaMIL: Frequency-Driven Mamba Multi-Instance Learning for Weakly Supervised Lesion Segmentation in Medical Images
2025-06-09 11:18:02Hangbei Cheng, Xiaorong Dong, Xueyu Liu, Jianan Zhang, Xuetao Ma, Mingqiang Wei, Liansheng Wang, Junxin Chen, Yongfei WuAbstract
Accurate lesion segmentation in histopathology images is essential for diagnostic interpretation and quantitative analysis, yet it remains challenging due to the limited availability of costly pixel-level annotations. To address this, we propose FMaMIL, a novel two-stage framework for weakly supervised lesion segmentation based solely on image-level labels. In the first stage, a lightweight Mamba-based encoder is introduced to capture long-range dependencies across image patches under the MIL paradigm. To enhance spatial sensitivity and structural awareness, we design a learnable frequency-domain encoding module that supplements spatial-domain features with spectrum-based information. CAMs generated in this stage are used to guide segmentation training. In the second stage, we refine the initial pseudo labels via a CAM-guided soft-label supervision and a self-correction mechanism, enabling robust training even under label noise. Extensive experiments on both public and private histopathology datasets demonstrate that FMaMIL outperforms state-of-the-art weakly supervised methods without relying on pixel-level annotations, validating its effectiveness and potential for digital pathology applications.
Abstract (translated)
在病理图像中准确地分割病变区域对于诊断解释和定量分析至关重要，但由于昂贵的像素级注释资源有限，这一任务仍然具有挑战性。为了应对这个问题，我们提出了FMaMIL，这是一种基于仅使用图像级别标签的弱监督学习框架的新颖两阶段方法。 **第一阶段**：在该阶段中，引入了一个轻量级的以Mamba为基础的编码器，在多实例学习（MIL）范式下捕捉图像块间的长程依赖性。为了增强空间敏感性和结构感知能力，我们设计了一个可学习的频域编码模块，通过基于谱的信息补充了空域特征。该阶段生成的类激活图（CAMs）用于引导分割训练。 **第二阶段**：在这一阶段中，我们通过使用CAM指导下的软标签监督和自我校正机制来优化初始伪标签，这使得即使在存在标签噪声的情况下也能实现稳健的训练。 广泛的实验表明，无论是在公开数据集还是私有病理图像数据集中，FMaMIL的表现都优于现有的弱监督方法，并且无需依赖像素级注释。这些结果验证了该框架的有效性和其在未来数字病理学应用中的潜力。
URL
https://arxiv.org/abs/2506.07652
PDF
-
Precise Information Control in Long-Form Text Generation
2025-06-06 23:42:42Jacqueline He, Howard Yen, Margaret Li, Shuyue Stella Li, Zhiyuan Zeng, Weijia Shi, Yulia Tsvetkov, Danqi Chen, Pang Wei Koh, Luke ZettlemoyerAbstract
A central challenge in modern language models (LMs) is intrinsic hallucination: the generation of information that is plausible but unsubstantiated relative to input context. To study this problem, we propose Precise Information Control (PIC), a new task formulation that requires models to generate long-form outputs grounded in a provided set of short self-contained statements, known as verifiable claims, without adding any unsupported ones. For comprehensiveness, PIC includes a full setting that tests a model's ability to include exactly all input claims, and a partial setting that requires the model to selectively incorporate only relevant claims. We present PIC-Bench, a benchmark of eight long-form generation tasks (e.g., summarization, biography generation) adapted to the PIC setting, where LMs are supplied with well-formed, verifiable input claims. Our evaluation of a range of open and proprietary LMs on PIC-Bench reveals that, surprisingly, state-of-the-art LMs still intrinsically hallucinate in over 70% of outputs. To alleviate this lack of faithfulness, we introduce a post-training framework, using a weakly supervised preference data construction method, to train an 8B PIC-LM with stronger PIC ability--improving from 69.1% to 91.0% F1 in the full PIC setting. When integrated into end-to-end factual generation pipelines, PIC-LM improves exact match recall by 17.1% on ambiguous QA with retrieval, and factual precision by 30.5% on a birthplace verification task, underscoring the potential of precisely grounded generation.
Abstract (translated)
现代语言模型（LM）的一个核心挑战是内在幻觉：生成的信息在输入上下文中看似合理但缺乏实质证据支持。为研究这一问题，我们提出了精确信息控制（PIC），这是一种新的任务形式化方法，要求模型基于一组提供的简短独立陈述——称为可验证声明——生成长文本输出，并且不允许添加任何未被证实的信息。为了全面评估，PIC 包含一个完整的设置来测试模型能否准确地包括所有的输入声明，以及一个部分设置以要求模型选择性地整合相关的声明。我们推出了 PIC-Bench，这是一个由八个改编至 PIC 设置的长文本生成任务（例如总结、人物生成）构成的基准，其中语言模型接收结构良好且可验证的输入陈述。 我们在 PIC-Bench 上评估了一系列开源和专有语言模型后发现，令人惊讶的是，最先进的语言模型仍然在超过 70% 的输出中产生内在幻觉。为了解决这一忠实度不足的问题，我们引入了一个后训练框架，利用弱监督偏好数据构造方法来训练一个具有更强 PIC 能力的8B参数PIC-LM——在完整的PIC设置下F1得分从69.1%提升到91.0%。 将PIC-LM集成进端到端事实生成流水线中，在带有检索的模糊问题回答任务中，精确匹配召回率提升了17.1%，而在出生地验证任务中，则实现了30.5%的事实精度提升。这些结果强调了精准基础性生成的潜力。
URL
https://arxiv.org/abs/2506.06589
PDF
-
Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models
2025-06-06 11:50:18Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. PontiAbstract
To what extent do vision-and-language foundation models possess a realistic world model (observation $\times$ action $\rightarrow$ observation) and a dynamics model (observation $\times$ observation $\rightarrow$ action), when actions are expressed through language? While open-source foundation models struggle with both, we find that fine-tuning them to acquire a dynamics model through supervision is significantly easier than acquiring a world model. In turn, dynamics models can be used to bootstrap world models through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, the dynamics model can annotate actions for unlabelled pairs of video frame observations to expand the training data. We further propose a new objective, where image tokens in observation pairs are weighted by their importance, as predicted by a recognition model. Secondly, the dynamics models can assign rewards to multiple samples of the world model to score them, effectively guiding search at inference time. We evaluate the world models resulting from both strategies through the task of action-centric image editing on Aurora-Bench. Our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin of $15\%$ on real-world subsets according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of Aurora-Bench.
Abstract (translated)
当动作通过语言表达时，视觉和语言基础模型在多大程度上具备现实世界的模型（观察 $\times$ 动作 $\rightarrow$ 观察）和动态模型（观察 $\times$ 观察 $\rightarrow$ 动作）？开源基础模型在这两方面都存在困难，但我们发现通过监督来微调它们以获取动态模型比获取世界模型要容易得多。反过来，可以通过两种主要策略利用动态模型来启动世界模型：1）从合成数据进行弱监督学习；2）在推理时间验证。首先，动态模型可以为未标记的视频帧观察对分配动作标签，以此扩展训练数据集。此外，我们还提出了一种新目标，在此目标中，通过识别模型预测的重要性加权观测配对中的图像令牌。其次，动态模型还可以将奖励分配给世界模型的多个样本进行评分，从而有效指导推理时间搜索。 我们在Aurora-Bench上通过对以行动为中心的图像编辑任务来评估两种策略产生的世界模型性能。我们最好的模型在与最先进的图像编辑模型的竞争中表现出具有竞争力的表现，在GPT4o-as-judge根据现实世界的子集评估时提高了15%，并且在整个Aurora-Bench所有子集中实现了最佳的人均评价结果。
URL
https://arxiv.org/abs/2506.06006
PDF
-
HuGeDiff: 3D Human Generation via Diffusion with Gaussian Splatting
2025-06-04 18:11:23Maksym Ivashechkin, Oscar Mendez, Richard BowdenAbstract
3D human generation is an important problem with a wide range of applications in computer vision and graphics. Despite recent progress in generative AI such as diffusion models or rendering methods like Neural Radiance Fields or Gaussian Splatting, controlling the generation of accurate 3D humans from text prompts remains an open challenge. Current methods struggle with fine detail, accurate rendering of hands and faces, human realism, and controlability over appearance. The lack of diversity, realism, and annotation in human image data also remains a challenge, hindering the development of a foundational 3D human model. We present a weakly supervised pipeline that tries to address these challenges. In the first step, we generate a photorealistic human image dataset with controllable attributes such as appearance, race, gender, etc using a state-of-the-art image diffusion model. Next, we propose an efficient mapping approach from image features to 3D point clouds using a transformer-based architecture. Finally, we close the loop by training a point-cloud diffusion model that is conditioned on the same text prompts used to generate the original samples. We demonstrate orders-of-magnitude speed-ups in 3D human generation compared to the state-of-the-art approaches, along with significantly improved text-prompt alignment, realism, and rendering quality. We will make the code and dataset available.
Abstract (translated)
三维人体生成是计算机视觉和图形学中一个具有广泛应用的重要问题。尽管在生成式AI（如扩散模型）以及渲染方法（例如神经辐射场或高斯点聚法）方面取得了近期进展，但从文本提示准确生成三维人体仍然是一个开放性的挑战。当前的方法在精细细节、手部和面部的精确渲染、人类的真实感以及外观控制性上存在困难。此外，由于人体图像数据中缺乏多样性、真实性和标注信息，开发基础的人体三维模型也面临阻碍。 我们提出了一种弱监督管道来解决这些挑战。首先，利用最先进的图像扩散模型生成包含可控属性（如外观、种族、性别等）的逼真人体图像数据集。接下来，我们提出了一个基于变压器架构的有效映射方法，用于将图像特征转换为三维点云。最后，通过训练条件相同的文本提示以生成原始样本为基础的点云扩散模型来闭合这一循环。 我们的方法在三维人体生成速度上相比现有最佳方法提高了几个数量级，并显著改善了文本提示的对齐性、真实感和渲染质量。我们将公开发布代码和数据集。
URL
https://arxiv.org/abs/2506.04351
PDF
-
FreePRM: Training Process Reward Models Without Ground Truth Process Labels
2025-06-04 04:33:53Lin Sun, Chuang Liu, Xiaofeng Ma, Tao Yang, Weijia Lu, Ning WuAbstract
Recent advancements in Large Language Models (LLMs) have demonstrated that Process Reward Models (PRMs) play a crucial role in enhancing model performance. However, training PRMs typically requires step-level labels, either manually annotated or automatically generated, which can be costly and difficult to obtain at scale. To address this challenge, we introduce FreePRM, a weakly supervised framework for training PRMs without access to ground-truth step-level labels. FreePRM first generates pseudo step-level labels based on the correctness of final outcome, and then employs Buffer Probability to eliminate impact of noise inherent in pseudo labeling. Experimental results show that FreePRM achieves an average F1 score of 53.0% on ProcessBench, outperforming fully supervised PRM trained on Math-Shepherd by +24.1%. Compared to other open-source PRMs, FreePRM outperforms upon RLHFlow-PRM-Mistral-8B (28.4%) by +24.6%, EurusPRM (31.3%) by +21.7%, and Skywork-PRM-7B (42.1%) by +10.9%. This work introduces a new paradigm in PRM training, significantly reducing reliance on costly step-level annotations while maintaining strong performance.
Abstract (translated)
最近在大型语言模型（LLMs）方面取得的进展表明，过程奖励模型（PRMs）在提高模型性能方面发挥着关键作用。然而，训练PRMs通常需要逐步骤的标签数据，这些标签要么是人工标注的，要么是由自动化工具生成的，这既昂贵又难以大规模获取。为了解决这一挑战，我们引入了FreePRM，这是一种无需访问真实步级标签就能训练PRM的弱监督框架。 FreePRM首先基于最终结果的正确性生成伪步骤级标签，然后利用缓存概率来消除伪标记中固有的噪声影响。实验结果显示，在ProcessBench数据集上，FreePRM实现了平均F1分数为53.0%，这一成绩优于完全监督训练下的Math-Shepherd PRM模型+24.1%。与开源的其他PRMs相比，FreePRM在性能上也取得了显著优势：相较于RLHFlow-PRM-Mistral-8B（28.4%），其表现高出+24.6%，优于EurusPRM（31.3%）+21.7%，以及Skywork-PRM-7B（42.1%）+10.9%。 这项工作开创了PRMs训练的新范式，显著减少了对昂贵的步骤级注释的依赖，同时保持了强大的性能表现。
URL
https://arxiv.org/abs/2506.03570
PDF
-
MSDA: Combining Pseudo-labeling and Self-Supervision for Unsupervised Domain Adaptation in ASR
2025-05-30 14:46:05Dimitrios Damianos, Georgios Paraskevopoulos, Alexandros PotamianosAbstract
In this work, we investigate the Meta PL unsupervised domain adaptation framework for Automatic Speech Recognition (ASR). We introduce a Multi-Stage Domain Adaptation pipeline (MSDA), a sample-efficient, two-stage adaptation approach that integrates self-supervised learning with semi-supervised techniques. MSDA is designed to enhance the robustness and generalization of ASR models, making them more adaptable to diverse conditions. It is particularly effective for low-resource languages like Greek and in weakly supervised scenarios where labeled data is scarce or noisy. Through extensive experiments, we demonstrate that Meta PL can be applied effectively to ASR tasks, achieving state-of-the-art results, significantly outperforming state-of-the-art methods, and providing more robust solutions for unsupervised domain adaptation in ASR. Our ablations highlight the necessity of utilizing a cascading approach when combining self-supervision with self-training.
Abstract (translated)
在这项工作中，我们研究了用于自动语音识别（ASR）的元PL无监督领域适应框架。我们引入了一种多阶段领域适应流水线（MSDA），这是一种样本效率高的两阶段适应方法，它将自监督学习与半监督技术相结合。MSDA旨在增强ASR模型的鲁棒性和泛化能力，使其更能适应各种条件。该方法特别适用于希腊语等资源匮乏的语言以及标签数据稀缺或嘈杂的弱监督场景。 通过广泛的实验，我们展示了Meta PL可以有效地应用于ASR任务，并取得了最先进的成果，在很大程度上超越了现有的最佳方法，为无监督领域适应在ASR中的应用提供了更稳健的解决方案。我们的消融研究突显了在结合自监督和自我训练时采用级联方法的重要性。
URL
https://arxiv.org/abs/2505.24656
PDF
-
Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors
2025-05-30 01:12:39Peiran Xu, Yadong MuAbstract
In this work, we focus on the task of weakly supervised affordance grounding, where a model is trained to identify affordance regions on objects using human-object interaction images and egocentric object images without dense labels. Previous works are mostly built upon class activation maps, which are effective for semantic segmentation but may not be suitable for locating actions and functions. Leveraging recent advanced foundation models, we develop a supervised training pipeline based on pseudo labels. The pseudo labels are generated from an off-the-shelf part segmentation model, guided by a mapping from affordance to part names. Furthermore, we introduce three key enhancements to the baseline model: a label refining stage, a fine-grained feature alignment process, and a lightweight reasoning module. These techniques harness the semantic knowledge of static objects embedded in off-the-shelf foundation models to improve affordance learning, effectively bridging the gap between objects and actions. Extensive experiments demonstrate that the performance of the proposed model has achieved a breakthrough improvement over existing methods. Our codes are available at this https URL .
Abstract (translated)
在这项工作中，我们专注于弱监督下的功能区域定位任务，即在没有密集标签的情况下，通过使用人类与物体交互的图像和第一人称视角的对象图像来训练模型识别对象的功能区域。以往的工作大多基于类激活图（CAMs），这类方法对语义分割非常有效，但对于定位动作和功能可能不太适用。我们利用最近出现的基础模型，开发了一个基于伪标签的监督训练流程。这些伪标签是由现成的部分分割模型生成，并通过功能到部件名称的映射进行引导。 此外，我们还为基准模型引入了三个关键改进：标签精炼阶段、细粒度特征对齐过程以及轻量级推理模块。这些技术利用嵌入在现成基础模型中的静态物体语义知识来增强功能学习能力，并有效地弥合了物体与动作之间的差距。广泛的实验表明，所提出的模型的性能在现有方法的基础上取得了重大突破性改进。 我们的代码可在 [此链接](https://this https URL) 获取。
URL
https://arxiv.org/abs/2505.24103
PDF
-
CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization
2025-05-29 15:03:59Rui Xia, Dan Jiang, Quan Zhang, Ke Zhang, Chun YuanAbstract
Temporal Action Localization (TAL) has garnered significant attention in information retrieval. Existing supervised or weakly supervised methods heavily rely on labeled temporal boundaries and action categories, which are labor-intensive and time-consuming. Consequently, unsupervised temporal action localization (UTAL) has gained popularity. However, current methods face two main challenges: 1) Classification pre-trained features overly focus on highly discriminative regions; 2) Solely relying on visual modality information makes it difficult to determine contextual boundaries. To address these issues, we propose a CLIP-assisted cross-view audiovisual enhanced UTAL method. Specifically, we introduce visual language pre-training (VLP) and classification pre-training-based collaborative enhancement to avoid excessive focus on highly discriminative regions; we also incorporate audio perception to provide richer contextual boundary information. Finally, we introduce a self-supervised cross-view learning paradigm to achieve multi-view perceptual enhancement without additional annotations. Extensive experiments on two public datasets demonstrate our model's superiority over several state-of-the-art competitors.
Abstract (translated)
时间动作定位（TAL）在信息检索领域引起了广泛的关注。现有的监督或弱监督方法高度依赖于标记的时间边界和动作类别，这需要大量的劳动时间和资源。因此，无监督时间动作定位（UTAL）变得越来越受欢迎。然而，当前的方法面临着两大挑战：1) 预训练的分类特征过于关注高区分性的区域；2) 单纯依靠视觉模态的信息难以确定上下文边界。为了解决这些问题，我们提出了一种CLIP辅助的跨视角音频-视觉增强UTAL方法。具体来说，我们引入了视觉语言预训练（VLP）和基于分类预训练的合作增强机制来避免对高区分性区域的过度关注；同时，我们还融入了音频感知以提供更丰富的上下文边界信息。最后，我们提出了一种自监督跨视角学习范式，在不增加额外标注的情况下实现多视角感知增强。在两个公开数据集上的广泛实验表明，我们的模型优于多个最先进的竞争对手。
URL
https://arxiv.org/abs/2505.23524
PDF
-
DSAGL: Dual-Stream Attention-Guided Learning for Weakly Supervised Whole Slide Image Classification
2025-05-29 11:07:16Daoxi Cao, Hangbei Cheng, Yijin Li, Ruolin Zhou, Xinyi Li, Xuehan Zhang, Binwei Li, Xuancheng Gu, Xueyu Liu, Yongfei WuAbstract
Whole-slide images (WSIs) are critical for cancer diagnosis due to their ultra-high resolution and rich semantic content. However, their massive size and the limited availability of fine-grained annotations pose substantial challenges for conventional supervised learning. We propose DSAGL (Dual-Stream Attention-Guided Learning), a novel weakly supervised classification framework that combines a teacher-student architecture with a dual-stream design. DSAGL explicitly addresses instance-level ambiguity and bag-level semantic consistency by generating multi-scale attention-based pseudo labels and guiding instance-level learning. A shared lightweight encoder (VSSMamba) enables efficient long-range dependency modeling, while a fusion-attentive module (FASA) enhances focus on sparse but diagnostically relevant regions. We further introduce a hybrid loss to enforce mutual consistency between the two streams. Experiments on CIFAR-10, NCT-CRC, and TCGA-Lung datasets demonstrate that DSAGL consistently outperforms state-of-the-art MIL baselines, achieving superior discriminative performance and robustness under weak supervision.
Abstract (translated)
全滑动图像（WSIs）对于癌症诊断至关重要，因为它们具有超高分辨率和丰富的语义内容。然而，由于其庞大的大小以及细粒度注释的有限可用性，传统监督学习面临着重大挑战。我们提出了DSAGL（双流注意力引导学习），这是一种新型弱监督分类框架，结合了教师-学生架构与双流设计。DSAGL通过生成基于多尺度注意力的伪标签并指导实例级学习来明确解决实例层面的模糊性和袋层级的语义一致性问题。一个共享的轻量级编码器（VSSMamba）能够有效地进行长程依赖建模，而融合注意模块（FASA）则增强了对稀疏但诊断相关的区域的关注。我们进一步引入了一种混合损失函数来强制执行两流之间的相互一致性。在CIFAR-10、NCT-CRC和TCGA-Lung数据集上的实验表明，DSAGL始终优于最先进的MIL基准模型，在弱监督条件下实现了优异的判别性能和鲁棒性。
URL
https://arxiv.org/abs/2505.23341
PDF
