Abstract
In-context learning (ICL) enables Large Vision-Language Models (LVLMs) to adapt to new tasks without parameter updates, using a few demonstrations from a large support set. However, selecting informative demonstrations leads to high computational and memory costs. While some methods explore selecting a small and representative coreset in the text classification, evaluating all support set samples remains costly, and discarded samples lead to unnecessary information loss. These methods may also be less effective for image classification due to differences in feature spaces. Given these limitations, we propose Key-based Coreset Optimization (KeCO), a novel framework that leverages untapped data to construct a compact and informative coreset. We introduce visual features as keys within the coreset, which serve as the anchor for identifying samples to be updated through different selection strategies. By leveraging untapped samples from the support set, we update the keys of selected coreset samples, enabling the randomly initialized coreset to evolve into a more informative coreset under low computational cost. Through extensive experiments on coarse-grained and fine-grained image classification benchmarks, we demonstrate that KeCO effectively enhances ICL performance for image classification task, achieving an average improvement of more than 20\%. Notably, we evaluate KeCO under a simulated online scenario, and the strong performance in this scenario highlights the practical value of our framework for resource-constrained real-world scenarios.
Abstract (translated)
上下文学习（ICL）使大型视觉-语言模型（LVLMs）能够在不更新参数的情况下适应新任务，通过从一个大的支持集合中使用少量的演示案例来实现这一点。然而，选择具有信息性的演示案例会导致计算和内存成本高昂。虽然有一些方法探索了在文本分类中选取小而有代表性的核心集的方法，但评估所有支持集合样本仍然代价高昂，并且被丢弃的样本会导致不必要的信息损失。这些方法也可能因为特征空间的不同而在图像分类任务上效果较差。 鉴于上述限制，我们提出了一种基于键的关键核心优化（KeCO）框架，该框架利用未充分利用的数据来构建一个紧凑而有信息量的核心集。我们在核心集中引入了视觉特征作为“键”，用作通过不同选择策略识别需要更新的样本的基础。通过利用支持集合中未使用的样本，我们更新选定核心集样本中的“键”，从而使随机初始化的核心集在较低计算成本下演变为更有信息量的核心集。 通过在粗粒度和细粒度图像分类基准上的广泛实验，我们证明了KeCO能够有效地提高ICL在图像分类任务上的性能，平均提高了超过20%。值得注意的是，我们在模拟的在线场景中评估了KeCO，并且该框架在此场景中的出色表现突显了它对于资源受限的真实世界应用场景的实际价值。
URL
https://arxiv.org/abs/2504.14200
