Abstract
Foundation models, pre-trained on a large amount of data have demonstrated impressive zero-shot capabilities in various downstream tasks. However, in object detection and instance segmentation, two fundamental computer vision tasks heavily reliant on extensive human annotations, foundation models such as SAM and DINO struggle to achieve satisfactory performance. In this study, we reveal that the devil is in the object boundary, \textit{i.e.}, these foundation models fail to discern boundaries between individual objects. For the first time, we probe that CLIP, which has never accessed any instance-level annotations, can provide a highly beneficial and strong instance-level boundary prior in the clustering results of its particular intermediate layer. Following this surprising observation, we propose $\textbf{Zip}$ which $\textbf{Z}$ips up CL$\textbf{ip}$ and SAM in a novel classification-first-then-discovery pipeline, enabling annotation-free, complex-scene-capable, open-vocabulary object detection and instance segmentation. Our Zip significantly boosts SAM's mask AP on COCO dataset by 12.5% and establishes state-of-the-art performance in various settings, including training-free, self-training, and label-efficient finetuning. Furthermore, annotation-free Zip even achieves comparable performance to the best-performing open-vocabulary object detecters using base annotations. Code is released at this https URL
Abstract (translated)
基础模型，在大量数据上预训练，已经在各种下游任务中展示了出色的零样本能力。然而，在目标检测和实例分割这两个对大量人类标注依赖的基本计算机视觉任务中，基础模型如SAM和DINO很难实现令人满意的成绩。在这项研究中，我们揭示了对象边界就在那里，即这些基础模型无法区分单个对象的边界。对于第一个，我们观察到CLIP，它从未访问过任何实例级别的标注，在其特定中间层的聚类结果中可以提供高度有益的实例级别边界先验。接着，我们提出了Zip，它将CLIP和SAM在一种新颖的分类先于发现的数据管道中结合，实现无标注、复杂场景 capable的开放词汇目标检测和实例分割。我们的Zip显著提高了SAM在COCO数据集上的掩码AP，并建立了各种设置中的最先进性能，包括无需训练、自训练和标签效率微调。此外，无标注的Zip甚至与使用基本注释的最佳性能对象检测器相当。代码发布在https://这个URL上。
URL
https://arxiv.org/abs/2404.11957
