Abstract
The advent of Large Multimodal Models (LMMs) has sparked a surge in research aimed at harnessing their remarkable reasoning abilities. However, for understanding text-rich images, challenges persist in fully leveraging the potential of LMMs, and existing methods struggle with effectively processing high-resolution images. In this work, we propose TextCoT, a novel Chain-of-Thought framework for text-rich image understanding. TextCoT utilizes the captioning ability of LMMs to grasp the global context of the image and the grounding capability to examine local textual regions. This allows for the extraction of both global and local visual information, facilitating more accurate question-answering. Technically, TextCoT consists of three stages, including image overview, coarse localization, and fine-grained observation. The image overview stage provides a comprehensive understanding of the global scene information, and the coarse localization stage approximates the image area containing the answer based on the question asked. Then, integrating the obtained global image descriptions, the final stage further examines specific regions to provide accurate answers. Our method is free of extra training, offering immediate plug-and-play functionality. Extensive experiments are conducted on a series of text-rich image question-answering benchmark datasets based on several advanced LMMs, and the results demonstrate the effectiveness and strong generalization ability of our method. Code is available at this https URL.
Abstract (translated)
大规模多模态模型（LMMs）的出现引发了旨在充分利用其非凡推理能力的研究高潮。然而，对于理解富含文本的图像，要完全利用LMMs的潜力仍然具有挑战性，现有的方法在处理高分辨率图像时也存在困难。在这项工作中，我们提出了TextCoT，一种用于文本丰富图像理解的全新链式思维框架。TextCoT利用LMMs的摘要能力来把握图像的全局上下文和定位能力来检查局部文本区域。这使得可以提取全局和局部视觉信息，从而促进更准确的问题回答。从技术上讲，TextCoT由三个阶段组成，包括图像概述、粗略定位和细粒度观察。图像概述阶段提供了对全局场景信息的全面理解，粗略定位阶段根据提出的问题估算包含答案的图像区域。然后，将获得的全局图像描述集成到其中，最后的阶段进一步检查具体区域以提供准确答案。我们的方法无需额外训练，具有即插即用的功能。在多个基于先进LMM的文本丰富图像问题回答基准数据集上进行了广泛的实验，结果表明，我们的方法具有有效性和强大的泛化能力。代码可以从该链接处获取。
URL
https://arxiv.org/abs/2404.09797
