Abstract
Vision--Language Models (VLMs) have demonstrated success across diverse applications, yet their potential to assist in relevance judgments remains uncertain. This paper assesses the relevance estimation capabilities of VLMs, including CLIP, LLaVA, and GPT-4V, within a large-scale \textit{ad hoc} retrieval task tailored for multimedia content creation in a zero-shot fashion. Preliminary experiments reveal the following: (1) Both LLaVA and GPT-4V, encompassing open-source and closed-source visual-instruction-tuned Large Language Models (LLMs), achieve notable Kendall's $\tau \sim 0.4$ when compared to human relevance judgments, surpassing the CLIPScore metric. (2) While CLIPScore is strongly preferred, LLMs are less biased towards CLIP-based retrieval systems. (3) GPT-4V's score distribution aligns more closely with human judgments than other models, achieving a Cohen's $\kappa$ value of around 0.08, which outperforms CLIPScore at approximately -0.096. These findings underscore the potential of LLM-powered VLMs in enhancing relevance judgments.
Abstract (translated)
视觉语言模型（VLMs）在各种应用领域都取得了成功，但它们在辅助进行相关性判断方面的潜力仍然不确定。本文评估了包括CLIP、LLaVA和GPT-4V在内的VLMs在大型临时检索任务中的相关性估计能力。初步实验结果如下：（1）LLaVA和GPT-4V，包括开源和闭源的视觉指令调整的大型语言模型（LLMs），在比较人类相关性判断时，实现了显著的Kendall分数$\tau \approx 0.4$，超过了CLIPScore指标。（2）虽然CLIPScore受到了很高的偏好，但LLM在CLIP基于检索系统上的偏见较小。（3）GPT-4V的得分分布与人类判断更加接近其他模型，实现了科恩分数$\kappa$值约为0.08，这超过了CLIPScore约-0.096。这些发现强调了LLM驱动的VLMs在增强相关性判断方面的潜力。
URL
https://arxiv.org/abs/2408.01363
