Abstract
Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach.
Abstract (translated)
多模态LLM是LLM的自然演变，并扩大其功能以实现超越纯文本模态。在设计新颖架构和视觉与语言适配器的研究过程中，本文重点关注为这样的模型赋予回答需要外部知识的问题的能力。我们称之为Wiki-LLaVA的方法旨在通过分层检索管道访问外部知识源，为LLM提供额外的上下文，提高生成对话的有效性和精确度。我们在针对视觉问题回答的外部数据集上进行广泛的实验，证明了我们的方法的合适性。
URL
https://arxiv.org/abs/2404.15406
