Abstract
Large Vision-Language Models (LVLMs) show significant strides in general-purpose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMT-Bench comprises $31,325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $30$ LVLMs such as the proprietary GPT-4V, GeminiProVision, and open-sourced InternVL-Chat, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.
Abstract (translated)
大视图语言模型（LVLMs）在诸如视觉对话和 embodied 导航等通用多模态应用方面取得了显著的进步。然而，现有的多模态评估基准测试的项目数量有限，无法跟踪 LVLM 的开发。在这项研究中，我们提出了 MMT-Bench，一个全面的多模态基准，旨在评估 LVLMs 在需要专家知识和故意视觉识别、定位、推理和规划的大型多模态任务中的能力。MMT-Bench 包括来自各种多模态场景的 $31,325$ 个精心策划的多选题视觉问题，涵盖了 $32$ 个核心元任务和 $162$ 个亚任务的多模态理解。由于其广泛的任务覆盖，MMT-Bench 使使用任务图评估 LVLMs 成为可能，促进发现领域内和领域外任务。评估了 $30$ 个 LVLM，如专有 GPT-4V、GeminiProVision 和开源的 InternVL-Chat，结果表明 MMT-Bench 带来了重大挑战。我们预计，MMT-Bench 将激发社区开发指向实现通用多模态智能的下一代多模态基础模型。
URL
https://arxiv.org/abs/2404.16006
