Abstract
With the advent of large language models(LLMs) enhanced by the chain-of-thought(CoT) methodology, visual reasoning problem is usually decomposed into manageable sub-tasks and tackled sequentially with various external tools. However, such a paradigm faces the challenge of the potential "determining hallucinations" in decision-making due to insufficient visual information and the limitation of low-level perception tools that fail to provide abstract summaries necessary for comprehensive reasoning. We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks. This paper delves into the realm of multimodal CoT to solve intricate visual reasoning tasks with multimodal large language models(MLLMs) and their cognitive capability. To this end, we propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture. Cantor first acts as a decision generator and integrates visual inputs to analyze the image and problem, ensuring a closer alignment with the actual context. Furthermore, Cantor leverages the advanced cognitive functions of MLLMs to perform as multifaceted experts for deriving higher-level information, enhancing the CoT generation process. Our extensive experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance across two complex visual reasoning datasets, without necessitating fine-tuning or ground-truth rationales. Project Page: this https URL .
Abstract (translated)
随着大型语言模型（LLMs）通过链式思考（CoT）方法的优势进入，通常将视觉推理问题分解为可管理的小子任务，并使用各种外部工具按顺序解决。然而，这种范式面临着由于缺乏视觉信息而导致的决策中潜在的“感知错觉”以及低级感知工具无法提供全面推理所需抽象摘要的局限。我们认为，收敛的视觉上下文获取和逻辑推理对于解决视觉推理任务至关重要。本文将深入研究多模态CoT，使用多模态大型语言模型（MMLMs）及其认知能力解决复杂的视觉推理任务。为此，我们提出了一个创新的多模态CoT框架，称为Cantor，其特点是一个感知决策架构。Cantor首先充当一个决策生成器，并将视觉输入整合分析图像和问题，确保更贴近实际上下文。此外，Cantor利用MLLMs的先进认知功能执行多面手特征，提高CoT生成过程。我们进行的广泛实验证明，所提出的框架的有效性，表明在两个复杂的视觉推理数据集上，多模态CoT性能得到了显著的提高，而无需进行微调或目标推理的 ground-truth 理由。项目页面：https:// this URL.
URL
https://arxiv.org/abs/2404.16033
