Abstract
Fine-grained Visual Recognition (FGVR) involves distinguishing between visually similar categories, which is inherently challenging due to subtle inter-class differences and the need for large, expert-annotated datasets. In domains like medical imaging, such curated datasets are unavailable due to issues like privacy concerns and high annotation costs. In such scenarios lacking labeled data, an FGVR model cannot rely on a predefined set of training labels, and hence has an unconstrained output space for predictions. We refer to this task as Vocabulary-Free FGVR (VF-FGVR), where a model must predict labels from an unconstrained output space without prior label information. While recent Multimodal Large Language Models (MLLMs) show potential for VF-FGVR, querying these models for each test input is impractical because of high costs and prohibitive inference times. To address these limitations, we introduce \textbf{Nea}rest-Neighbor Label \textbf{R}efinement (NeaR), a novel approach that fine-tunes a downstream CLIP model using labels generated by an MLLM. Our approach constructs a weakly supervised dataset from a small, unlabeled training set, leveraging MLLMs for label generation. NeaR is designed to handle the noise, stochasticity, and open-endedness inherent in labels generated by MLLMs, and establishes a new benchmark for efficient VF-FGVR.
Abstract (translated)
细粒度视觉识别（FGVR）涉及在图像中区分具有细微差异的相似类别，这一任务因其类间细微差别和需要大量专家标注的数据集而变得极具挑战性。在医学成像等领域中，由于隐私问题和高昂的注释成本，这样的数据集难以获取。在这种缺乏标记数据的情况下，一个FGVR模型无法依赖预定义的训练标签，因此其预测结果的空间是无约束的。我们称这一任务为“词汇无关细粒度视觉识别”（VF-FGVR），即在没有先验标签信息的情况下，模型必须从无约束输出空间中进行预测。 虽然最近出现的多模态大型语言模型（MLLM）在处理VF-FGVR方面表现出潜力，但由于查询这些模型的成本高且推理时间长，因此将其用于每个测试输入是不切实际的。为了克服这些问题，我们引入了一种新方法**Nea**rest-Neighbor Label **R**efinement (NeaR)，这种方法通过使用MLLM生成标签来微调下游CLIP模型。我们的方法利用少量未标记的数据集构建了一个弱监督数据集，并借助MLLM进行标签生成。 NeaR被设计用来处理由MLLM产生的标签中存在的噪声、随机性和开放性，从而为高效的VF-FGVR建立了新的基准标准。
URL
https://arxiv.org/abs/2505.01064
