Weakly_Supervised
-
Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation
2025-05-28 11:05:50Zhisong Wang, Yiwen Ye, Ziyang Chen, Yong XiaAbstract
Weakly supervised semantic segmentation (WSSS) in medical imaging struggles with effectively using sparse annotations. One promising direction for WSSS leverages gaze annotations, captured via eye trackers that record regions of interest during diagnostic procedures. However, existing gaze-based methods, such as GazeMedSeg, do not fully exploit the rich information embedded in gaze data. In this paper, we propose GradTrack, a framework that utilizes physicians' gaze track, including fixation points, durations, and temporal order, to enhance WSSS performance. GradTrack comprises two key components: Gaze Track Map Generation and Track Attention, which collaboratively enable progressive feature refinement through multi-level gaze supervision during the decoding process. Experiments on the Kvasir-SEG and NCI-ISBI datasets demonstrate that GradTrack consistently outperforms existing gaze-based methods, achieving Dice score improvements of 3.21\% and 2.61\%, respectively. Moreover, GradTrack significantly narrows the performance gap with fully supervised models such as nnUNet.
Abstract (translated)
在医学成像中的弱监督语义分割（WSSS）面临有效利用稀疏标注的挑战。一个有前景的研究方向是利用通过眼动追踪器记录下的诊断过程中感兴趣区域的眼动数据来增强WSSS效果。然而，现有的基于眼动的方法（如GazeMedSeg）未能充分挖掘眼动数据中的丰富信息。在本文中，我们提出了GradTrack框架，该框架利用了医生的注视轨迹，包括固定点、持续时间和时间顺序等信息，以提升弱监督语义分割的表现。 GradTrack包含两个关键组件：注视轨迹图生成和轨迹注意力机制，这两个部分合作通过多级眼动监督来逐步改进特征，在解码过程中实现渐进式的特征精炼。在Kvasir-SEG和NCI-ISBI数据集上的实验表明，GradTrack相比现有的基于眼动的方法表现出显著的优势，分别取得了3.21% 和 2.61% 的Dice分数提升。此外，GradTrack大幅度缩小了与全监督模型（如nnUNet）的性能差距。
URL
https://arxiv.org/abs/2505.22230
PDF
-
Weakly Supervised Data Refinement and Flexible Sequence Compression for Efficient Thai LLM-based ASR
2025-05-28 07:39:25Mingchen Shao, Xinfa Zhu, Chengyou Wang, Bingshen Mu, Hai Li, Ying Yan, Junhui Liu, Danming Xie, Lei XieAbstract
Despite remarkable achievements, automatic speech recognition (ASR) in low-resource scenarios still faces two challenges: high-quality data scarcity and high computational demands. This paper proposes EThai-ASR, the first to apply large language models (LLMs) to Thai ASR and create an efficient LLM-based ASR system. EThai-ASR comprises a speech encoder, a connection module and a Thai LLM decoder. To address the data scarcity and obtain a powerful speech encoder, EThai-ASR introduces a self-evolving data refinement strategy to refine weak labels, yielding an enhanced speech encoder. Moreover, we propose a pluggable sequence compression module used in the connection module with three modes designed to reduce the sequence length, thus decreasing computational demands while maintaining decent performance. Extensive experiments demonstrate that EThai-ASR has achieved state-of-the-art accuracy in multiple datasets. We release our refined text transcripts to promote further research.
Abstract (translated)
尽管自动语音识别（ASR）在资源有限的场景中已经取得了显著成就，但仍面临两大挑战：高质量数据稀缺和计算需求高。本文提出了EThai-ASR，这是首个将大型语言模型（LLMs）应用于泰语ASR并创建高效LLM基ASR系统的方案。EThai-ASR包含一个语音编码器、连接模块和一个泰语LLM解码器。为了应对数据稀缺问题，并增强语音编码器的能力，EThai-ASR引入了一种自我演进的数据精炼策略，用于改进弱标签，从而产生了一个性能更佳的语音编码器。此外，我们还提出了一种可插拔序列压缩模块，在连接模块中使用该模块并设计了三种模式以减少序列长度，从而在保持良好性能的同时降低计算需求。大量的实验表明，EThai-ASR已经在多个数据集中达到了最先进的准确率水平。我们将发布我们的精炼文本转录本，以促进进一步的研究。
URL
https://arxiv.org/abs/2505.22063
PDF
-
From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
2025-05-26 15:11:23Zuyao Chen, Jinlin Wu, Zhen Lei, Chang Wen ChenAbstract
We present OvSGTR, a novel transformer-based framework for fully open-vocabulary scene graph generation that overcomes the limitations of traditional closed-set models. Conventional methods restrict both object and relationship recognition to a fixed vocabulary, hindering their applicability to real-world scenarios where novel concepts frequently emerge. In contrast, our approach jointly predicts objects (nodes) and their inter-relationships (edges) beyond predefined categories. OvSGTR leverages a DETR-like architecture featuring a frozen image backbone and text encoder to extract high-quality visual and semantic features, which are then fused via a transformer decoder for end-to-end scene graph prediction. To enrich the model's understanding of complex visual relations, we propose a relation-aware pre-training strategy that synthesizes scene graph annotations in a weakly supervised manner. Specifically, we investigate three pipelines--scene parser-based, LLM-based, and multimodal LLM-based--to generate transferable supervision signals with minimal manual annotation. Furthermore, we address the common issue of catastrophic forgetting in open-vocabulary settings by incorporating a visual-concept retention mechanism coupled with a knowledge distillation strategy, ensuring that the model retains rich semantic cues during fine-tuning. Extensive experiments on the VG150 benchmark demonstrate that OvSGTR achieves state-of-the-art performance across multiple settings, including closed-set, open-vocabulary object detection-based, relation-based, and fully open-vocabulary scenarios. Our results highlight the promise of large-scale relation-aware pre-training and transformer architectures for advancing scene graph generation towards more generalized and reliable visual understanding.
Abstract (translated)
我们介绍了OvSGTR，这是一种基于变压器的全新框架，用于全开放式词汇量场景图生成，克服了传统封闭集模型的局限性。传统的方法将物体和关系识别限制在一个固定的词汇表中，这妨碍了它们在新概念频繁出现的真实世界场景中的应用。相比之下，我们的方法同时预测超出预定义类别的对象（节点）及其相互关系（边）。OvSGTR采用类似于DETR的架构，包括冻结的图像骨干网络和文本编码器来提取高质量的视觉和语义特征，并通过变压器解码器融合这些特征以进行端到端场景图预测。为了丰富模型对复杂视觉关系的理解，我们提出了一种基于关系感知的预训练策略，在弱监督下综合生成场景图注释。具体而言，我们研究了三种管道——基于场景解析器、基于大型语言模型（LLM）和多模态LLM的方法——以利用最少的手动标注生成可转移的监督信号。此外，为了解决开放式词汇设置中常见的灾难性遗忘问题，我们引入了一种结合视觉概念保留机制与知识蒸馏策略的方法，在微调过程中确保模型保持丰富的语义线索。在VG150基准测试上的广泛实验表明，OvSGTR在封闭集、基于开放词汇对象检测的、关系驱动型和完全开放式词汇量等多种设置下均达到了最先进的性能水平。我们的结果强调了大规模关系感知预训练和变压器架构对于推进场景图生成向更通用和可靠视觉理解方向发展的前景。
URL
https://arxiv.org/abs/2505.20106
PDF
-
I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
2025-05-25 15:34:29Jiayi Xin, Sukwon Yun, Jie Peng, Inyoung Choi, Jenna L. Ballard, Tianlong Chen, Qi LongAbstract
Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at this https URL.
Abstract (translated)
模态融合是多模态学习的核心，它能够整合来自不同数据源的信息。然而，传统的融合方法在（1）无法处理模态之间的异构交互以及（2）缺乏解释性来揭示数据中固有的多模态交互方面存在局限性。为此，我们提出了一种新的框架I2MoE（可解释的多模态交互感知专家混合模型），这是一种端到端的专家混合框架，旨在通过显式建模多种多模态交互以及在局部和全局层面提供解释来增强模态融合。 首先，I2MoE 使用带有弱监督交互损失的不同交互专家来以数据驱动的方式学习多模态交互。其次，I2MoE 部署了一个重新加权模型，该模型为每个交互专家的输出分配重要性分数，这提供了样本级别和数据集级别的解释。 对医疗和通用多模态数据集进行广泛的评估显示，I2MoE 具有足够的灵活性与不同的融合技术结合使用，并且在各种现实场景中持续改进任务性能并提供解释。代码可在以下链接获取：[此 URL](https://this https URL)（请将此处的“此 https URL”替换为实际提供的具体链接）。
URL
https://arxiv.org/abs/2505.19190
PDF
-
WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation
2025-05-24 13:05:17Yang Liu, Silin Cheng, Xinwei He, Sebastien Ourselin, Lei Tan, Gen LuoAbstract
Weakly supervised referring expression comprehension(WREC) and segmentation(WRES) aim to learn object grounding based on a given expression using weak supervision signals like image-text pairs. While these tasks have traditionally been modeled separately, we argue that they can benefit from joint learning in a multi-task framework. To this end, we propose WeakMCN, a novel multi-task collaborative network that effectively combines WREC and WRES with a dual-branch architecture. Specifically, the WREC branch is formulated as anchor-based contrastive learning, which also acts as a teacher to supervise the WRES branch. In WeakMCN, we propose two innovative designs to facilitate multi-task collaboration, namely Dynamic Visual Feature Enhancement(DVFE) and Collaborative Consistency Module(CCM). DVFE dynamically combines various pre-trained visual knowledge to meet different task requirements, while CCM promotes cross-task consistency from the perspective of optimization. Extensive experimental results on three popular REC and RES benchmarks, i.e., RefCOCO, RefCOCO+, and RefCOCOg, consistently demonstrate performance gains of WeakMCN over state-of-the-art single-task alternatives, e.g., up to 3.91% and 13.11% on RefCOCO for WREC and WRES tasks, respectively. Furthermore, experiments also validate the strong generalization ability of WeakMCN in both semi-supervised REC and RES settings against existing methods, e.g., +8.94% for semi-REC and +7.71% for semi-RES on 1% RefCOCO. The code is publicly available at this https URL.
Abstract (translated)
弱监督指代表达理解（WREC）和分割（WRES）的目标是基于给定的描述学习目标对象定位，利用诸如图像文本对之类的弱监督信号。虽然这些任务传统上分别建模，但我们认为在多任务框架中进行联合学习可以带来好处。为此，我们提出了WeakMCN，这是一种新颖的多任务协作网络，采用双分支架构有效结合了WREC和WRES。具体来说，WREC支路被设计为基于锚点的对比学习，并且也作为教师来监督WRES支路。在WeakMCN中，我们提出两项创新设计以促进多任务合作：动态视觉特征增强（DVFE）和协作一致性模块（CCM）。DVFE动态结合各种预训练的视觉知识，以满足不同的任务需求；而CCM则从优化的角度促进了跨任务的一致性。 广泛的实验结果在三个流行的REC和RES基准测试数据集上得到验证，即RefCOCO、RefCOCO+ 和 RefCOCOg。这些结果显示了WeakMCN相对于最先进的单任务替代方法的性能提升，例如，在RefCOCO上的WREC任务中提升了3.91%，在WRES任务中提升了13.11%。 此外，实验还验证了WeakMCN在半监督REC和RES设置中的强大泛化能力，这比现有方法更优。例如，在1% RefCOCO的情况下，对于半监督REC提高了8.94%，而对于半监督RES则提高了7.71%。 代码可以在提供的URL上公开获取。
URL
https://arxiv.org/abs/2505.18686
PDF
-
Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling
2025-05-23 14:48:32Bryan Wong, Jong Woo Kim, Huazhu Fu, Mun Yong YiAbstract
Vision-language models (VLMs) have recently been integrated into multiple instance learning (MIL) frameworks to address the challenge of few-shot, weakly supervised classification of whole slide images (WSIs). A key trend involves leveraging multi-scale information to better represent hierarchical tissue structures. However, existing methods often face two key limitations: (1) insufficient modeling of interactions within the same modalities across scales (e.g., 5x and 20x) and (2) inadequate alignment between visual and textual modalities on the same scale. To address these gaps, we propose HiVE-MIL, a hierarchical vision-language framework that constructs a unified graph consisting of (1) parent-child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes on the same scale. To further enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism that removes weakly correlated patch-text pairs, and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments on TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MIL consistently outperforms both traditional MIL and recent VLM-based MIL approaches, achieving gains of up to 4.1% in macro F1 under 16-shot settings. Our results demonstrate the value of jointly modeling hierarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data. The code is available at this https URL
Abstract (translated)
最近，视觉-语言模型（VLMs）被整合到多实例学习（MIL）框架中，以解决全滑动图像（WSI）的少量样本、弱监督分类问题。一个关键的趋势是利用跨尺度的信息来更好地表示组织结构的层次性。然而，现有方法通常面临两个主要限制：(1) 在同一模态内（例如5x和20x）不同尺度之间的交互建模不足；(2) 同一尺度上视觉与文本模态间的对齐度不够。为解决这些问题，我们提出了HiVE-MIL，这是一个层次化视觉-语言框架，它构建了一个统一的图结构，其中包括：(1) 粗略（5x）和精细（20x）视觉/文本节点之间的父-子链接，以捕捉层次关系；以及 (2) 同一尺度上的异构内部边连接同一尺度下的视觉与文本节点。为了进一步增强语义一致性，HiVE-MIL引入了一种两阶段、由文本引导的动态过滤机制，用于移除相关性较弱的图块-文本对，并引入了跨尺度的层次对比损失来调整不同尺度上的文本语义。在TCGA乳腺癌、肺癌和肾癌数据集上进行的大量实验表明，HiVE-MIL始终优于传统的MIL方法以及近期基于VLM的MIL方法，在16个样本设置下实现了高达4.1%的宏F1值提升。我们的结果证明了同时建模层次结构和多模态对齐对于从有限病理学数据中进行高效、可扩展学习的价值。代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2505.17982
PDF
-
Semantic segmentation with reward
2025-05-23 13:52:15Xie Ting, Ye Huang, Zhilin Liu, Lixin DuanAbstract
In real-world scenarios, pixel-level labeling is not always available. Sometimes, we need a semantic segmentation network, and even a visual encoder can have a high compatibility, and can be trained using various types of feedback beyond traditional labels, such as feedback that indicates the quality of the parsing results. To tackle this issue, we proposed RSS (Reward in Semantic Segmentation), the first practical application of reward-based reinforcement learning on pure semantic segmentation offered in two granular levels (pixel-level and image-level). RSS incorporates various novel technologies, such as progressive scale rewards (PSR) and pair-wise spatial difference (PSD), to ensure that the reward facilitates the convergence of the semantic segmentation network, especially under image-level rewards. Experiments and visualizations on benchmark datasets demonstrate that the proposed RSS can successfully ensure the convergence of the semantic segmentation network on two levels of rewards. Additionally, the RSS, which utilizes an image-level reward, outperforms existing weakly supervised methods that also rely solely on image-level signals during training.
Abstract (translated)
在实际场景中，像素级标注并不总是可用的。有时我们需要一个语义分割网络，并且即使是一个视觉编码器也能具有很高的兼容性，可以通过传统标签之外的各种反馈类型进行训练，例如指示解析结果质量的反馈。为了解决这个问题，我们提出了RSS（语义分割中的奖励），这是第一个在纯语义分割任务中实现基于奖励的强化学习的实际应用，提供了两个粒度层次（像素级和图像级）的解决方案。RSS整合了多种新技术，如逐步尺度奖励（PSR）和成对空间差异（PSD），以确保奖励能够促进语义分割网络的收敛，特别是在使用图像级别奖励时。 在基准数据集上的实验和可视化结果表明，所提出的RSS能够在两个层次的奖励下成功地保证语义分割网络的收敛性。此外，采用图像级奖励的RSS优于现有仅依赖于训练过程中图像信号进行弱监督的方法。
URL
https://arxiv.org/abs/2505.17905
PDF
-
Sketchy Bounding-box Supervision for 3D Instance Segmentation
2025-05-22 08:49:49Qian Deng, Le Hui, Jin Xie, Jian YangAbstract
Bounding box supervision has gained considerable attention in weakly supervised 3D instance segmentation. While this approach alleviates the need for extensive point-level annotations, obtaining accurate bounding boxes in practical applications remains challenging. To this end, we explore the inaccurate bounding box, named sketchy bounding box, which is imitated through perturbing ground truth bounding box by adding scaling, translation, and rotation. In this paper, we propose Sketchy-3DIS, a novel weakly 3D instance segmentation framework, which jointly learns pseudo labeler and segmentator to improve the performance under the sketchy bounding-box supervisions. Specifically, we first propose an adaptive box-to-point pseudo labeler that adaptively learns to assign points located in the overlapped parts between two sketchy bounding boxes to the correct instance, resulting in compact and pure pseudo instance labels. Then, we present a coarse-to-fine instance segmentator that first predicts coarse instances from the entire point cloud and then learns fine instances based on the region of coarse instances. Finally, by using the pseudo instance labels to supervise the instance segmentator, we can gradually generate high-quality instances through joint training. Extensive experiments show that our method achieves state-of-the-art performance on both the ScanNetV2 and S3DIS benchmarks, and even outperforms several fully supervised methods using sketchy bounding boxes. Code is available at this https URL.
Abstract (translated)
边界框监督在弱监督下的三维实例分割中受到了广泛的关注。尽管这种方法减轻了对大量点级别标注的需求，但在实际应用中获取准确的边界框仍然具有挑战性。为此，我们探索了一种不精确的边界框，称为草图边界框，它是通过对地面真实边界框添加缩放、平移和旋转扰动来模仿得到的。 在本文中，我们提出了一种新颖的弱监督三维实例分割框架——Sketchy-3DIS，该框架通过联合学习伪标签生成器（pseudo labeler）和分割器（segmentator），以改善草图边界框指导下的性能。具体而言，首先我们提出了一个自适应的框到点的伪标签生成器，它能够自适应地将位于两个不精确边界框重叠部分中的点分配给正确的实例，从而产生紧凑且纯净的伪实例标签。然后，我们提出了一种由粗到精的实例分割器，该分割器首先从整个点云中预测粗略实例，然后再基于这些粗略实例区域学习精细实例。最后，通过使用伪实例标签监督实例分割器，并进行联合训练，可以逐步生成高质量的实例。 大量的实验表明，我们的方法在ScanNetV2和S3DIS基准测试上取得了最先进的性能，甚至超过了几个采用草图边界框的完全监督的方法。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2505.16399
PDF
-
Self-Classification Enhancement and Correction for Weakly Supervised Object Detection
2025-05-22 06:45:58Yufei Yin, Lechao Cheng, Wengang Zhou, Jiajun Deng, Zhou Yu, Houqiang LiAbstract
In recent years, weakly supervised object detection (WSOD) has attracted much attention due to its low labeling cost. The success of recent WSOD models is often ascribed to the two-stage multi-class classification (MCC) task, i.e., multiple instance learning and online classification refinement. Despite achieving non-trivial progresses, these methods overlook potential classification ambiguities between these two MCC tasks and fail to leverage their unique strengths. In this work, we introduce a novel WSOD framework to ameliorate these two issues. For one thing, we propose a self-classification enhancement module that integrates intra-class binary classification (ICBC) to bridge the gap between the two distinct MCC tasks. The ICBC task enhances the network's discrimination between positive and mis-located samples in a class-wise manner and forges a mutually reinforcing relationship with the MCC task. For another, we propose a self-classification correction algorithm during inference, which combines the results of both MCC tasks to effectively reduce the mis-classified predictions. Extensive experiments on the prevalent VOC 2007 & 2012 datasets demonstrate the superior performance of our framework.
Abstract (translated)
近年来，由于弱监督目标检测（WSOD）的标注成本低而备受关注。最近成功的WSOD模型通常归功于两阶段多类分类（MCC）任务的成功，即多实例学习和在线分类细化。尽管这些方法已经取得了显著的进步，但它们忽视了这两个MCC任务之间的潜在分类模糊性，并未能充分利用各自的独特优势。在这项工作中，我们引入了一个新颖的WSOD框架来改善这些问题。 一方面，我们提出了一种自我分类增强模块，该模块通过在类内进行二元分类（ICBC）来弥合这两项不同MCC任务之间的差距。ICBC任务以类别为单位增强了网络对正样本和错误定位样本之间区别的能力，并与MCC任务形成了相互强化的关系。 另一方面，在推理过程中，我们提出了一个自我分类校正算法，该算法结合了两个MCC任务的结果，有效地减少了误分类的预测结果。 在流行的VOC 2007及2012数据集上的广泛实验展示了我们的框架具有优越的性能。
URL
https://arxiv.org/abs/2505.16294
PDF
-
Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
2025-05-21 12:19:55Jianyuan Guo, Peike Li, Trevor CohnAbstract
Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation. Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process. This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss. We train our SLT mode, which consists of a vision encoder and a translator, through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language. Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.
Abstract (translated)
手语翻译（SLT）旨在将手语视频映射为口语文本。一种常见的方法依赖于词素注释作为中间表示，将SLT分解为两个子任务：视频到词素识别和词素到文本的翻译。尽管这一范式有效，但它依赖于专家标注的词素标签，这些标签成本高昂且在现有数据集中很少见，这限制了其可扩展性。为了应对这一挑战，我们提出了一种无词素的伪词素生成框架，该框架消除了对手工标注词素的需求，同时保留结构化的中间表示形式。 具体而言，我们使用少量示例文本-词素对提示大型语言模型（LLM），并通过上下文学习从口语文本中生成草案手语词素。为了增强LLM生成的伪词素与视频中的手势序列之间的对应关系，我们通过弱监督学习过程纠正伪词素中的顺序以实现更好的对齐。这种重新排序促进了辅助对齐目标的集成，并允许使用连接时序分类（CTC）损失进行高效的监督。 我们的SLT模型由视觉编码器和翻译器组成，我们通过三阶段管道对其进行训练，逐步缩小手语与口语之间的模式差距。尽管该方法很简单，但在两个SLT基准测试中，它超越了以前最先进的无词素框架，并且在与基于词素的方法的比较中取得了具有竞争力的结果。
URL
https://arxiv.org/abs/2505.15438
PDF
-
From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data
2025-05-20 20:50:28Ahmed Adel Attia, Dorottya Demszky, Jing Liu, Carol Espy-WilsonAbstract
Recent progress in speech recognition has relied on models trained on vast amounts of labeled data. However, classroom Automatic Speech Recognition (ASR) faces the real-world challenge of abundant weak transcripts paired with only a small amount of accurate, gold-standard data. In such low-resource settings, high transcription costs make re-transcription impractical. To address this, we ask: what is the best approach when abundant inexpensive weak transcripts coexist with limited gold-standard data, as is the case for classroom speech data? We propose Weakly Supervised Pretraining (WSP), a two-step process where models are first pretrained on weak transcripts in a supervised manner, and then fine-tuned on accurate data. Our results, based on both synthetic and real weak transcripts, show that WSP outperforms alternative methods, establishing it as an effective training methodology for low-resource ASR in real-world scenarios.
Abstract (translated)
近期的语音识别进展主要依赖于大量标注数据训练的模型。然而，在教室自动语音识别（ASR）中，面临的一大现实挑战是存在大量的低成本、质量较低的转录文本与少量高质量标准数据并存的情况。在资源匮乏的情况下，高昂的转录成本使得重新转录变得不切实际。为解决这一问题，我们提出：当大量廉价但质量较差的转录音频和有限的高标准数据同时出现时（如教室语音数据），最有效的处理方法是什么？为此，我们提出了弱监督预训练（WSP）的方法，这是一种两步过程，首先在有监督的情况下使用低质量转录文本对模型进行预训练，然后基于准确的数据对其进行微调。我们的实验结果表明，在合成和真实的弱标记数据上，WSP方法优于其他替代方案，证明了其作为资源匮乏环境下有效训练方法的潜力。
URL
https://arxiv.org/abs/2505.17088
PDF
-
Bronchovascular Tree-Guided Weakly Supervised Learning Method for Pulmonary Segment Segmentation
2025-05-20 04:23:12Ruijie Zhao (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Zuopeng Tan (Canon Medical Systems), Xiao Xue (Canon Medical Systems), Longfei Zhao (Canon Medical Systems), Bing Li (Canon Medical Systems), Zicheng Liao (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Ying Ming (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Jiaru Wang (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Ran Xiao (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Sirong Piao (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Rui Zhao (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China), Qiqi Xu (Canon Medical Systems), Wei Song (Department of Radiology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China)Abstract
Pulmonary segment segmentation is crucial for cancer localization and surgical planning. However, the pixel-wise annotation of pulmonary segments is laborious, as the boundaries between segments are indistinguishable in medical images. To this end, we propose a weakly supervised learning (WSL) method, termed Anatomy-Hierarchy Supervised Learning (AHSL), which consults the precise clinical anatomical definition of pulmonary segments to perform pulmonary segment segmentation. Since pulmonary segments reside within the lobes and are determined by the bronchovascular tree, i.e., artery, airway and vein, the design of the loss function is founded on two principles. First, segment-level labels are utilized to directly supervise the output of the pulmonary segments, ensuring that they accurately encompass the appropriate bronchovascular tree. Second, lobe-level supervision indirectly oversees the pulmonary segment, ensuring their inclusion within the corresponding lobe. Besides, we introduce a two-stage segmentation strategy that incorporates bronchovascular priori information. Furthermore, a consistency loss is proposed to enhance the smoothness of segment boundaries, along with an evaluation metric designed to measure the smoothness of pulmonary segment boundaries. Visual inspection and evaluation metrics from experiments conducted on a private dataset demonstrate the effectiveness of our method.
Abstract (translated)
肺段分割对于癌症定位和手术规划至关重要。然而，由于医学图像中各肺段之间的边界不明显，对其进行逐像素标注是一项费时的工作。为此，我们提出了一种弱监督学习（WSL）方法，称为解剖层次监督学习（AHSL），该方法利用精确的临床解剖定义来执行肺段分割。鉴于肺段位于肺叶内，并由支气管血管树（即动脉、气道和静脉）决定，损失函数的设计基于两个原则。首先，使用段级标签直接监管输出的肺段结果，确保其准确地包含相应的支气管血管结构。其次，通过肺叶级别的监督间接监管各肺段，确保它们位于对应的肺叶内。 此外，我们还引入了一种两阶段分割策略，该策略结合了支气管血管先验信息。另外，提出了一致性损失函数以增强肺段边界的平滑度，并设计了一个评估指标来衡量肺段边界平滑度的大小。通过在私有数据集上进行实验并使用视觉检查和评估指标，证明了我们方法的有效性。
URL
https://arxiv.org/abs/2505.13911
PDF
-
4D-ROLLS: 4D Radar Occupancy Learning via LiDAR Supervision
2025-05-20 04:12:44Ruihan Liu, Xiaoyi Wu, Xijun Chen, Liang Hu, Yunjiang LouAbstract
A comprehensive understanding of 3D scenes is essential for autonomous vehicles (AVs), and among various perception tasks, occupancy estimation plays a central role by providing a general representation of drivable and occupied space. However, most existing occupancy estimation methods rely on LiDAR or cameras, which perform poorly in degraded environments such as smoke, rain, snow, and fog. In this paper, we propose 4D-ROLLS, the first weakly supervised occupancy estimation method for 4D radar using the LiDAR point cloud as the supervisory signal. Specifically, we introduce a method for generating pseudo-LiDAR labels, including occupancy queries and LiDAR height maps, as multi-stage supervision to train the 4D radar occupancy estimation model. Then the model is aligned with the occupancy map produced by LiDAR, fine-tuning its accuracy in occupancy estimation. Extensive comparative experiments validate the exceptional performance of 4D-ROLLS. Its robustness in degraded environments and effectiveness in cross-dataset training are qualitatively demonstrated. The model is also seamlessly transferred to downstream tasks BEV segmentation and point cloud occupancy prediction, highlighting its potential for broader applications. The lightweight network enables 4D-ROLLS model to achieve fast inference speeds at about 30 Hz on a 4060 GPU. The code of 4D-ROLLS will be made available at this https URL.
Abstract (translated)
对三维场景的全面理解对于自动驾驶车辆（AVs）至关重要，在各种感知任务中，占据估计通过提供可行驶和被占用空间的一般表示起着核心作用。然而，大多数现有的占据估计方法依赖于激光雷达或摄像头，在烟雾、雨雪和大雾等退化环境中表现不佳。在本文中，我们提出了4D-ROLLS，这是第一个使用激光雷达点云作为监督信号的弱监督4D雷达占据估计方法。 具体而言，我们引入了一种生成伪激光雷达标签的方法，包括占据查询和激光雷达高度图，作为多阶段监督来训练4D雷达占据估计模型。然后将该模型与由激光雷达产生的占据地图对齐，进一步微调其占据估计的准确性。广泛的对比实验验证了4D-ROLLS的卓越性能，其在退化环境中的鲁棒性和跨数据集训练的有效性得到了定性的证明。 此外，该模型还无缝转移至下游任务如鸟瞰图分割和点云占据预测中，凸显了它在更广泛的应用领域的潜力。轻量级网络使4D-ROLLS模型能够在配备NVIDIA 4060 GPU的设备上实现大约30Hz的快速推理速度。4D-ROLLS代码将在[此链接](https://example.com/4DROLLS)提供。 （注：上述翻译中提到的URL为示例，实际请替换为正确的网址。）
URL
https://arxiv.org/abs/2505.13905
PDF
-
Completely Weakly Supervised Class-Incremental Learning for Semantic Segmentation
2025-05-16 01:43:36David Minkwan Kim, Soeun Lee, Byeongkeun KangAbstract
This work addresses the task of completely weakly supervised class-incremental learning for semantic segmentation to learn segmentation for both base and additional novel classes using only image-level labels. While class-incremental semantic segmentation (CISS) is crucial for handling diverse and newly emerging objects in the real world, traditional CISS methods require expensive pixel-level annotations for training. To overcome this limitation, partially weakly-supervised approaches have recently been proposed. However, to the best of our knowledge, this is the first work to introduce a completely weakly-supervised method for CISS. To achieve this, we propose to generate robust pseudo-labels by combining pseudo-labels from a localizer and a sequence of foundation models based on their uncertainty. Moreover, to mitigate catastrophic forgetting, we introduce an exemplar-guided data augmentation method that generates diverse images containing both previous and novel classes with guidance. Finally, we conduct experiments in three common experimental settings: 15-5 VOC, 10-10 VOC, and COCO-to-VOC, and in two scenarios: disjoint and overlap. The experimental results demonstrate that our completely weakly supervised method outperforms even partially weakly supervised methods in the 15-5 VOC and 10-10 VOC settings while achieving competitive accuracy in the COCO-to-VOC setting.
Abstract (translated)
这项工作致力于完全弱监督的类增量学习在语义分割中的任务，利用仅有的图像级别标签来为基本和新增的类别进行语义分割。虽然类增量语义分割（CISS）对于处理现实世界中多样且不断出现的新物体至关重要，但传统的方法需要昂贵的像素级标注来进行训练。为了克服这一限制，最近提出了一些部分弱监督方法。然而，据我们所知，这是首次介绍了一种完全弱监督方法用于CISS。 为实现这一点，我们提议通过结合来自定位器和一系列基础模型的伪标签，并根据它们的不确定性进行选择来生成稳健的伪标签。此外，为了减轻灾难性遗忘问题，我们引入了一种基于样本引导的数据增强方法，该方法能生成包含先前类与新类别多样图像。 最后，我们在三种常见的实验设置下进行了测试：15-5 VOC、10-10 VOC 和 COCO-to-VOC，并且在两种场景中进行验证：离散和重叠。实验结果表明，在 15-5 VOC 和 10-10 VOC 设置下，我们的完全弱监督方法超越了部分弱监督的方法，同时在 COCO-to-VOC 设置中也取得了具有竞争力的准确性。
URL
https://arxiv.org/abs/2505.10781
PDF
-
AC-PKAN: Attention-Enhanced and Chebyshev Polynomial-Based Physics-Informed Kolmogorov-Arnold Networks
2025-05-13 15:46:10Hangwei Zhang, Zhimu Huang, Yan WangAbstract
Kolmogorov-Arnold Networks (KANs) have recently shown promise for solving partial differential equations (PDEs). Yet their original formulation is computationally and memory intensive, motivating the introduction of Chebyshev Type-I-based KANs (Chebyshev1KANs). Although Chebyshev1KANs have outperformed the vanilla KANs architecture, our rigorous theoretical analysis reveals that they still suffer from rank collapse, ultimately limiting their expressive capacity. To overcome these limitations, we enhance Chebyshev1KANs by integrating wavelet-activated MLPs with learnable parameters and an internal attention mechanism. We prove that this design preserves a full-rank Jacobian and is capable of approximating solutions to PDEs of arbitrary order. Furthermore, to alleviate the loss instability and imbalance introduced by the Chebyshev polynomial basis, we externally incorporate a Residual Gradient Attention (RGA) mechanism that dynamically re-weights individual loss terms according to their gradient norms and residual magnitudes. By jointly leveraging internal and external attention, we present AC-PKAN, a novel architecture that constitutes an enhancement to weakly supervised Physics-Informed Neural Networks (PINNs) and extends the expressive power of KANs. Experimental results from nine benchmark tasks across three domains show that AC-PKAN consistently outperforms or matches state-of-the-art models such as PINNsFormer, establishing it as a highly effective tool for solving complex real-world engineering problems in zero-data or data-sparse regimes. The code will be made publicly available upon acceptance.
Abstract (translated)
科洛莫戈罗夫-阿诺尔德网络（KANs）最近在解决偏微分方程（PDEs）方面显示出潜力。然而，它们的原始形式计算和内存消耗较大，因此引入了基于切比雪夫Type-I的KANs（Chebyshev1KANs）。虽然Chebyshev1KANs的表现优于传统的KAN架构，但我们的严格理论分析揭示出这些网络仍然存在秩塌陷的问题，这最终限制了它们的表达能力。为克服这一局限性，我们通过集成具有可学习参数和内部注意力机制的波浪激活多层感知机（MLP），对Chebyshev1KANs进行了增强。我们证明这种设计能够保持全秩雅可比矩阵，并且有能力近似任意阶数的偏微分方程的解。 为了缓解切比雪夫多项式基底引入的损失不稳定性和不平衡问题，我们在外部加入了残差梯度注意力（RGA）机制，该机制可以根据各个损失项的梯度范数和残差幅度动态重新加权。通过同时利用内部和外部注意机制，我们提出了AC-PKAN这一新架构，它构成了对弱监督物理信息神经网络（PINNs）的一种增强，并扩展了KANs的表达能力。 九个跨三个领域的基准任务实验结果表明，AC-PKAN在无数据或数据稀疏环境下始终优于或等同于现有最佳模型如PINNsFormer，在解决复杂的现实世界工程问题方面具有高度有效性。代码将在接受后公开发布。
URL
https://arxiv.org/abs/2505.08687
PDF
-
SimMIL: A Universal Weakly Supervised Pre-Training Framework for Multi-Instance Learning in Whole Slide Pathology Images
2025-05-10 17:23:36Yicheng Song, Tiancheng Lin, Die Peng, Su Yang, Yi XuAbstract
Various multi-instance learning (MIL) based approaches have been developed and successfully applied to whole-slide pathological images (WSI). Existing MIL methods emphasize the importance of feature aggregators, but largely neglect the instance-level representation learning. They assume that the availability of a pre-trained feature extractor can be directly utilized or fine-tuned, which is not always the case. This paper proposes to pre-train feature extractor for MIL via a weakly-supervised scheme, i.e., propagating the weak bag-level labels to the corresponding instances for supervised learning. To learn effective features for MIL, we further delve into several key components, including strong data augmentation, a non-linear prediction head and the robust loss function. We conduct experiments on common large-scale WSI datasets and find it achieves better performance than other pre-training schemes (e.g., ImageNet pre-training and self-supervised learning) in different downstream tasks. We further show the compatibility and scalability of the proposed scheme by deploying it in fine-tuning the pathological-specific models and pre-training on merged multiple datasets. To our knowledge, this is the first work focusing on the representation learning for MIL.
Abstract (translated)
多种基于多实例学习（MIL）的方法已经被开发并成功应用于全片病理图像（WSI）。现有的MIL方法强调特征聚合器的重要性，但往往忽视了实例级别表示学习。它们假设可以直接使用或微调预训练的特征提取器，但这在实际情况中并不总是成立的。本文提出了一种通过弱监督方案为MIL预先训练特征提取器的方法，即通过将弱级联标签传播到相应的实例进行有监督学习。为了有效地为MIL学习特征，我们进一步深入探讨了几个关键组件，包括强大的数据增强、非线性预测头和鲁棒的损失函数。我们在常见的大规模WSI数据集上进行了实验，发现该方案在不同的下游任务中比其他预训练方案（如ImageNet预训练和自监督学习）表现更好。此外，我们通过将其应用于微调特定病理模型以及在合并多个数据集上的预训练来展示了所提方案的兼容性和可扩展性。据我们所知，这是第一项专注于MIL表示学习的工作。
URL
https://arxiv.org/abs/2505.06710
PDF
-
Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining
2025-05-10 08:03:00Lu Dong, Haiyu Zhang, Hongjie Zhang, Yifei Huang, Zhen-Hua Ling, Yu Qiao, Limin Wang, Yali WangAbstract
The task of weakly supervised temporal sentence grounding (WSTSG) aims to detect temporal intervals corresponding to a language description from untrimmed videos with only video-level video-language correspondence. For an anchor sample, most existing approaches generate negative samples either from other videos or within the same video for contrastive learning. However, some training samples are highly similar to the anchor sample, directly regarding them as negative samples leads to difficulties for optimization and ignores the correlations between these similar samples and the anchor sample. To address this, we propose Positive Sample Mining (PSM), a novel framework that mines positive samples from the training set to provide more discriminative supervision. Specifically, for a given anchor sample, we partition the remaining training set into semantically similar and dissimilar subsets based on the similarity of their text queries. To effectively leverage these correlations, we introduce a PSM-guided contrastive loss to ensure that the anchor proposal is closer to similar samples and further from dissimilar ones. Additionally, we design a PSM-guided rank loss to ensure that similar samples are closer to the anchor proposal than to the negative intra-video proposal, aiming to distinguish the anchor proposal and the negative intra-video proposal. Experiments on the WSTSG and grounded VideoQA tasks demonstrate the effectiveness and superiority of our method.
Abstract (translated)
弱监督时间句子定位（WSTSG）任务的目标是从未经修剪的视频中检测与语言描述相对应的时间间隔，仅使用视频级别的视频-文本对应关系。对于一个锚样本，现有的大多数方法通过从其他视频或同一视频内生成负样本来进行对比学习。然而，一些训练样本与锚样本高度相似，直接将它们视为负样本会导致优化困难，并忽略了这些类似样本与锚样本之间的相关性。 为了解决这个问题，我们提出了正样本挖掘（PSM），这是一种新颖的框架，旨在从训练集中挖掘出更多判别性的正样本以提供更有区分度的监督。具体而言，对于给定的一个锚样本，我们会基于其文本查询的相似性将剩余的训练集划分为语义上相似和不相似的子集。为了有效地利用这些相关性，我们引入了一种由PSM引导的对比损失来确保锚提案与相似样本更接近，并且远离不相似样本。此外，我们设计了另一种由PSM指导的排名损失，以确保相似样本比负例内视频提案更靠近锚提案，从而区分出锚提案和负例内视频提案。 在WSTSG和基于视频的问题回答（grounded VideoQA）任务上的实验结果展示了我们的方法的有效性和优越性。
URL
https://arxiv.org/abs/2505.06557
PDF
-
Pro2SAM: Mask Prompt to SAM with Grid Points for Weakly Supervised Object Localization
2025-05-08 02:44:53Xi Yang, Songsong Duan, Nannan Wang, Xinbo GaoAbstract
Weakly Supervised Object Localization (WSOL), which aims to localize objects by only using image-level labels, has attracted much attention because of its low annotation cost in real applications. Current studies focus on the Class Activation Map (CAM) of CNN and the self-attention map of transformer to identify the region of objects. However, both CAM and self-attention maps can not learn pixel-level fine-grained information on the foreground objects, which hinders the further advance of WSOL. To address this problem, we initiatively leverage the capability of zero-shot generalization and fine-grained segmentation in Segment Anything Model (SAM) to boost the activation of integral object regions. Further, to alleviate the semantic ambiguity issue accrued in single point prompt-based SAM, we propose an innovative mask prompt to SAM (Pro2SAM) network with grid points for WSOL task. First, we devise a Global Token Transformer (GTFormer) to generate a coarse-grained foreground map as a flexible mask prompt, where the GTFormer jointly embeds patch tokens and novel global tokens to learn foreground semantics. Secondly, we deliver grid points as dense prompts into SAM to maximize the probability of foreground mask, which avoids the lack of objects caused by a single point/box prompt. Finally, we propose a pixel-level similarity metric to come true the mask matching from mask prompt to SAM, where the mask with the highest score is viewed as the final localization map. Experiments show that the proposed Pro2SAM achieves state-of-the-art performance on both CUB-200-2011 and ILSVRC, with 84.03\% and 66.85\% Top-1 Loc, respectively.
Abstract (translated)
弱监督目标定位（WSOL）旨在仅通过图像级别的标签来定位对象，由于其在实际应用中的低标注成本而备受关注。当前的研究主要集中在卷积神经网络（CNN）的类激活图（CAM）和变压器的自注意力图上以识别物体区域。然而，无论是CAM还是自注意力映射都无法学习前景对象上的像素级细粒度信息，这限制了WSOL技术的进步。为了解决这个问题，我们首次利用Segment Anything Model (SAM) 的零样本泛化能力和细粒度分割能力来增强整体目标区域的激活程度。 为了缓解基于单点提示的SAM在语义模糊性方面的问题，我们提出了一种创新性的mask prompt到SAM（Pro2SAM）网络，并使用网格点来进行WSOL任务。首先，我们设计了一个全局令牌变压器（GTFormer），以生成一个粗粒度的前景图作为灵活的掩码提示，在该过程中，GTFormer同时嵌入了补丁令牌和新的全球令牌来学习前景语义。 其次，我们将密集的网格点作为提示输入到SAM中，以此最大化前景掩码的概率，从而避免由单个点/框提示所引起的对象缺失问题。最后，我们提出了一种像素级相似度度量方法来实现从mask prompt到SAM的掩码匹配过程，其中得分最高的掩码被视为最终的目标定位图。 实验表明，在CUB-200-2011和ILSVRC数据集上，所提出的Pro2SAM网络实现了最先进的性能表现，其Top-1精度分别达到了84.03% 和 66.85%。
URL
https://arxiv.org/abs/2505.04905
PDF
-
Learning from Similarity Proportion Loss for Classifying Skeletal Muscle Recovery Stages
2025-05-07 06:02:27Yu Yamaoka or Weng Ian Chan, Shigeto Seno, Soichiro Fukada, Hideo MatsudaAbstract
Evaluating the regeneration process of damaged muscle tissue is a fundamental analysis in muscle research to measure experimental effect sizes and uncover mechanisms behind muscle weakness due to aging and disease. The conventional approach to assessing muscle tissue regeneration involves whole-slide imaging and expert visual inspection of the recovery stages based on the morphological information of cells and fibers. There is a need to replace these tasks with automated methods incorporating machine learning techniques to ensure a quantitative and objective analysis. Given the limited availability of fully labeled data, a possible approach is Learning from Label Proportions (LLP), a weakly supervised learning method using class label proportions. However, current LLP methods have two limitations: (1) they cannot adapt the feature extractor for muscle tissues, and (2) they treat the classes representing recovery stages and cell morphological changes as nominal, resulting in the loss of ordinal information. To address these issues, we propose Ordinal Scale Learning from Similarity Proportion (OSLSP), which uses a similarity proportion loss derived from two bag combinations. OSLSP can update the feature extractor by using class proportion attention to the ordinal scale of the class. Our model with OSLSP outperforms large-scale pre-trained and fine-tuning models in classification tasks of skeletal muscle recovery stages.
Abstract (translated)
评估受损肌肉组织再生过程是肌体研究中的基础分析，旨在衡量实验效应大小并揭示因衰老和疾病导致的肌肉无力背后的机制。传统上，评估肌肉组织再生的方法包括整体幻灯片成像以及基于细胞和纤维形态信息的专业人员视觉检查。为了确保定量且客观的分析，有必要用结合机器学习技术的自动化方法取代这些任务。鉴于完全标注数据的有限可用性，一种可能的方法是使用类标签比例进行弱监督学习（LLP）。然而，当前的LLP方法存在两个局限：(1) 无法根据肌肉组织调整特征提取器；(2) 将表示恢复阶段和细胞形态变化的类别视为名义类别，从而丧失了序数信息。为了解决这些问题，我们提出了从相似性比例学习（OSLSP），该方法使用由两种袋组合推导出的相似性比例损失函数。OSLSP能够通过类比例注意力更新特征提取器以适应有序尺度上的变化。在骨骼肌恢复阶段分类任务中，我们的模型结合了OSLSP，在性能上超过了大规模预训练和微调模型。
URL
https://arxiv.org/abs/2505.04150
PDF
-
3DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignment
2025-05-03 12:45:00Xiaoqi Li, Jiaming Liu, Nuowei Han, Liang Heng, Yandong Guo, Hao Dong, Yang LiuAbstract
The 3D weakly-supervised visual grounding task aims to localize oriented 3D boxes in point clouds based on natural language descriptions without requiring annotations to guide model learning. This setting presents two primary challenges: category-level ambiguity and instance-level complexity. Category-level ambiguity arises from representing objects of fine-grained categories in a highly sparse point cloud format, making category distinction challenging. Instance-level complexity stems from multiple instances of the same category coexisting in a scene, leading to distractions during grounding. To address these challenges, we propose a novel weakly-supervised grounding approach that explicitly differentiates between categories and instances. In the category-level branch, we utilize extensive category knowledge from a pre-trained external detector to align object proposal features with sentence-level category features, thereby enhancing category awareness. In the instance-level branch, we utilize spatial relationship descriptions from language queries to refine object proposal features, ensuring clear differentiation among objects. These designs enable our model to accurately identify target-category objects while distinguishing instances within the same category. Compared to previous methods, our approach achieves state-of-the-art performance on three widely used benchmarks: Nr3D, Sr3D, and ScanRef.
Abstract (translated)
三维弱监督视觉定位任务的目标是在没有注释引导模型学习的情况下，根据自然语言描述在点云中定位定向的3D边界框。这种设置提出了两个主要挑战：类别级模糊性和实例级复杂性。类别级模糊性源于以高度稀疏的点云格式表示细粒度类别的对象，这使得分类变得困难。实例级复杂性则来源于同一场景中存在相同类别的多个实例，导致定位时产生干扰。 为了应对这些挑战，我们提出了一种新颖的弱监督定位方法，该方法明确地区分了类别和实例。在类别级分支中，我们利用预训练外部检测器提供的广泛类别知识来对齐对象建议特征与句子级别的类别特征，从而增强模型对类别的识别能力。在实例级分支中，我们利用语言查询中的空间关系描述来细化对象提议的特征，确保不同对象之间的清晰区分。 这些设计使得我们的模型能够准确地识别目标类别的对象，并在同一类别内区分不同的实例。相较于先前的方法，在三个广泛使用的基准测试（Nr3D、Sr3D和ScanRef）上，我们的方法达到了最先进的性能水平。
URL
https://arxiv.org/abs/2505.01809
PDF
