Abstract
Out-of-distribution (OOD) problems in few-shot classification (FSC) occur when novel classes sampled from testing distributions differ from base classes drawn from training distributions, which considerably degrades the performance of deep learning models deployed in real-world applications. Recent studies suggest that the OOD problems in FSC mainly including: (a) cross-domain few-shot classification (CD-FSC) and (b) spurious-correlation few-shot classification (SC-FSC). Specifically, CD-FSC occurs when a classifier learns transferring knowledge from base classes drawn from seen training distributions but recognizes novel classes sampled from unseen testing distributions. In contrast, SC-FSC arises when a classifier relies on non-causal features (or contexts) that happen to be correlated with the labels (or concepts) in base classes but such relationships no longer hold during the model deployment. Despite CD-FSC has been extensively studied, SC-FSC remains understudied due to lack of the corresponding evaluation benchmarks. To this end, we present Meta Concept Context (MetaCoCo), a benchmark with spurious-correlation shifts collected from real-world scenarios. Moreover, to quantify the extent of spurious-correlation shifts of the presented MetaCoCo, we further propose a metric by using CLIP as a pre-trained vision-language model. Extensive experiments on the proposed benchmark are performed to evaluate the state-of-the-art methods in FSC, cross-domain shifts, and self-supervised learning. The experimental results show that the performance of the existing methods degrades significantly in the presence of spurious-correlation shifts. We open-source all codes of our benchmark and hope that the proposed MetaCoCo can facilitate future research on spurious-correlation shifts problems in FSC. The code is available at: this https URL.
Abstract (translated)
翻译：在少样本分类（FSC）中，异分布（OOD）问题主要包括：（a）跨域少样本分类（CD-FSC）和（b）伪相关少样本分类（SC-FSC）。具体来说，当分类器从可见训练分布中的基类学习知识，但同时从未见测试分布中采样新类时，就会出现CD-FSC。相反，当分类器依赖于与基类标签（或概念）相关但这种关系在模型部署时不再成立时，就会出现SC-FSC。尽管CD-FSC已经得到了广泛研究，但SC-FSC仍然因为没有相应的评估基准而备受忽视。为此，我们提出了元概念上下文（MetaCoCo），一个从现实世界场景中收集到的伪相关转移的基准。此外，为了量化所提出的MetaCoCo中伪相关转移的程度，我们进一步使用CLIP作为预训练的视觉语言模型提出了一个指标。对所提出的基准进行的大量实验用于评估FSC中的最先进方法、跨域转移和自监督学习的状态。实验结果表明，在存在伪相关转移的情况下，现有方法的性能显著下降。我们开源了我们的基准代码，并希望所提出的MetaCoCo能够促进未来关于FSC中伪相关转移问题的研究。代码可用在这个链接上：https://this URL。
URL
https://arxiv.org/abs/2404.19644
