Abstract
There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an image. In this work, we introduce Llip, Latent Language Image Pretraining, which models the diversity of captions that could match an image. Llip's vision encoder outputs a set of visual features that are mixed into a final representation by conditioning on information derived from the text. We show that Llip outperforms non-contextualized baselines like CLIP and SigLIP on a variety of tasks even with large-scale encoders. Llip improves zero-shot classification by an average of 2.9% zero-shot classification benchmarks with a ViT-G/14 encoder. Specifically, Llip attains a zero-shot top-1 accuracy of 83.5% on ImageNet outperforming a similarly sized CLIP by 1.4%. We also demonstrate improvement on zero-shot retrieval on MS-COCO by 6.0%. We provide a comprehensive analysis of the components introduced by the method and demonstrate that Llip leads to richer visual representations.
Abstract (translated)
图像的标题有很多种方式。相比之下，Contrastive Language Pretraining（CLIP）通过将图像及其标题映射到单个向量上来限制CLIP类似模型在描述图像多样方式方面的表现。在本文中，我们引入了Llip，Latent Language Image Pretraining，该模型能够表示与图像相匹配的标题的多样性。Llip的视觉编码器输出一系列视觉特征，通过在文本信息的基础上进行条件处理，将其混合为最终表示。我们证明了，即使在大型编码器的情况下，Llip在各种任务上也要优于无背景的基线，如CLIP和SigLIP。Llip通过ViT-G/14编码器平均提高了2.9%的零散数据基准的零散分类性能。具体来说，Llip在ImageNet上的零散分类顶点为83.5%，超越了同样大小的CLIP。我们还证明了在MS-COCO上，Llip在零散检索方面提高了6.0%的性能。我们全面分析了该方法引入的组件，并表明Llip能够产生更丰富的视觉表示。
URL
https://arxiv.org/abs/2405.00740
