Abstract
Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image, requiring both precise intra-class localization to pinpoint objects and effective inter-class reasoning to model complex category dependencies. While Vision-Language Pre-training (VLP) models offer a strong open-vocabulary foundation, they often struggle with fine-grained localization under weak supervision and typically fail to explicitly leverage structured relational knowledge beyond basic semantics, limiting performance especially for unseen classes. To overcome these limitations, we propose the Dual Adaptive Refinement Transfer (DART) framework. DART enhances a frozen VLP backbone via two synergistic adaptive modules. For intra-class refinement, an Adaptive Refinement Module (ARM) refines patch features adaptively, coupled with a novel Weakly Supervised Patch Selecting (WPS) loss that enables discriminative localization using only image-level labels. Concurrently, for inter-class transfer, an Adaptive Transfer Module (ATM) leverages a Class Relationship Graph (CRG), constructed using structured knowledge mined from a Large Language Model (LLM), and employs graph attention network to adaptively transfer relational information between class representations. DART is the first framework, to our knowledge, to explicitly integrate external LLM-derived relational knowledge for adaptive inter-class transfer while simultaneously performing adaptive intra-class refinement under weak supervision for OV-MLR. Extensive experiments on challenging benchmarks demonstrate that our DART achieves new state-of-the-art performance, validating its effectiveness.
Abstract (translated)
开放词汇多标签识别（OV-MLR）的目标是在图像中识别多种已知和未知的对象类别，这需要精确的类内定位以确定对象的位置，并有效地进行跨类推理以建模复杂的类别依赖关系。虽然视觉语言预训练（VLP）模型提供了强大的开放词汇基础，但它们在弱监督下的细粒度定位方面常常遇到困难，并且通常无法显式地利用超出基本语义的结构化关联知识，这限制了其在未见过类别的表现能力。 为了克服这些局限性，我们提出了双自适应细化传输（DART）框架。DART通过两个协同作用的自适应模块增强了冻结的VLP骨干网络。对于类内精炼，自适应精炼模块（ARM）能够根据图像级别的标签自适应地精炼块特征，并且结合了一种新颖的弱监督块选择损失（WPS），仅使用图像级别标签即可实现有区分性的定位。与此同时，在跨类别迁移方面，自适应传输模块（ATM）利用了一个基于大型语言模型（LLM）挖掘出的结构化知识构建的类关系图（CRG），并通过图形注意力网络来适应性地在类表示之间传递关联信息。 据我们所知，DART是第一个显式集成外部由LLM产生的关系知识进行自适应跨类别迁移的同时，在弱监督条件下执行自适应内类别精炼以实现OV-MLR的框架。在具有挑战性的基准测试中进行的一系列实验表明，我们的DART达到了新的最先进性能水平，验证了其有效性。 这个介绍展示了DART框架如何通过结合视觉语言预训练模型和大型语言模型的知识来改进开放词汇多标签识别任务中的定位精度和类别关系理解能力。
URL
https://arxiv.org/abs/2508.05585
