Abstract
Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.
Abstract (translated)
多模态大型语言模型（MLLMs）如GPT-4V和Gemini Pro在视觉问答（VQA）中面临挑战，尤其是在要求对物体身份、位置或属性的精细理解的对象导向感知任务中。这主要是由于它们在有效地将复杂视觉线索与文本信息相结合的能力方面有限。在本文中，我们提出了一个新方法，联合视觉和文本提示（VTPrompt），该方法利用细粒度的视觉信息来增强MLLMs在VQA中的能力，尤其是对于物体导向感知。VTPrompt将视觉和文本提示合并以提取文本问题中的关键概念，并使用检测模型在图像中突出相关的物体作为视觉提示。处理后的图像和文本提示随后输入MLLMs以产生更准确的答案。我们对GPT-4V和Gemini Pro在三个基准测试（MME，MMB和POPE）的实验结果表明，我们的方法取得了显著的改善。特别是，我们的方法使GPT-4V在MME上的得分提高了183.5，提高了GPT-4V和Gemini Pro在MMB上的8.17\%和15.69\%。
URL
https://arxiv.org/abs/2404.04514
