Abstract
Instance-level recognition (ILR) concerns distinguishing individual instances from one another, with person re-identification as a prominent example. Despite the impressive visual perception capabilities of modern VLMs, we find their performance on ILR unsatisfactory, often dramatically underperforming domain-specific ILR models. This limitation hinders many practical application of VLMs, e.g. where recognizing familiar people and objects is crucial for effective visual understanding. Existing solutions typically learn to recognize instances one at a time using instance-specific datasets, which not only incur substantial data collection and training costs but also struggle with fine-grained discrimination. In this work, we propose IIR-VLM, a VLM enhanced for In-context Instance-level Recognition. We integrate pre-trained ILR expert models as auxiliary visual encoders to provide specialized features for learning diverse instances, which enables VLMs to learn new instances in-context in a one-shot manner. Further, IIR-VLM leverages this knowledge for instance-aware visual understanding. We validate IIR-VLM's efficacy on existing instance personalization benchmarks. Finally, we demonstrate its superior ILR performance on a challenging new benchmark, which assesses ILR capabilities across varying difficulty and diverse categories, with person, face, pet and general objects as the instances at task.
Abstract (translated)
实例级识别（ILR）涉及区分个体实例，其中行人再识别是一个显著的例子。尽管现代视觉语言模型（VLMs）在视觉感知能力方面表现出色，但我们在其实例级识别性能上发现令人不满意的结果，通常大幅落后于特定领域的ILR模型。这种限制阻碍了VLM在许多实际应用中的有效性，例如，在有效视觉理解中需要认出熟悉的人和物体的场景中。现有的解决方案通常通过使用特定于每个实例的数据集一次一个地学习来实现实例识别，这不仅会产生大量的数据收集和训练成本，而且难以进行细微的区分。 为此，我们提出了IIR-VLM（In-context Instance-level Recognition增强型视觉语言模型），这是一种经过优化以在上下文中执行一次性实例级识别任务的VLM。我们在该模型中集成了预训练的ILR专家模型作为辅助视觉编码器，为学习多样化的实例提供了专门的功能特征，这使得VLM能够在一次输入后学会新的实例（即无需额外数据训练）。此外，IIR-VLM利用这种知识来实现对实例的理解。 我们已经在现有的实例个性化基准测试上验证了IIR-VLM的有效性。最后，我们在一个具有挑战性的新基准上展示了其优越的ILR性能评估，在这个基准中，评估了不同难度和多样类别下的ILR能力，任务中的实例包括人、面部、宠物以及一般对象。
URL
https://arxiv.org/abs/2601.14188
