Abstract
Recent progress in large pre-trained vision language models (VLMs) has reached state-of-the-art performance on several object detection benchmarks and boasts strong zero-shot capabilities, but for optimal performance on specific targets some form of finetuning is still necessary. While the initial VLM weights allow for great few-shot transfer learning, this usually involves the loss of the original natural language querying and zero-shot capabilities. Inspired by the success of Textual Inversion (TI) in personalizing text-to-image diffusion models, we propose a similar formulation for open-vocabulary object detection. TI allows extending the VLM vocabulary by learning new or improving existing tokens to accurately detect novel or fine-grained objects from as little as three examples. The learned tokens are completely compatible with the original VLM weights while keeping them frozen, retaining the original model's benchmark performance, and leveraging its existing capabilities such as zero-shot domain transfer (e.g., detecting a sketch of an object after training only on real photos). The storage and gradient calculations are limited to the token embedding dimension, requiring significantly less compute than full-model fine-tuning. We evaluated whether the method matches or outperforms the baseline methods that suffer from forgetting in a wide variety of quantitative and qualitative experiments.
Abstract (translated)
近期，在大型预训练视觉语言模型（VLMs）方面取得的进展已经在多个目标检测基准测试中达到了最先进的性能，并且具备强大的零样本能力，但为了在特定任务上达到最佳效果，仍需要某种形式的微调。虽然最初的VLM权重允许进行出色的少量样例迁移学习，但这通常会导致丧失原始的自然语言查询能力和零样本能力。 受Textual Inversion（TI）成功个性化文本到图像扩散模型的启发，我们提出了一种类似的开放词汇目标检测方法。TI通过从最少三个示例中学习新或改进现有标记来扩展VLM词汇表，以准确检测新颖或细粒度的对象。所学得的标记与原始VLM权重完全兼容，并且保持这些权重不变，同时保留模型在基准测试中的性能和现有的能力，例如零样本领域转移（即，在仅使用真实照片进行训练后识别物体素描）。存储和梯度计算仅限于标记嵌入维度，所需计算量远少于全模型微调。 我们在广泛的定量和定性实验中评估了该方法是否能与那些因遗忘问题而表现不佳的基线方法相匹配或超越。
URL
https://arxiv.org/abs/2508.05323
