Abstract
The financial domain poses substantial challenges for vision-language models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.
Abstract (translated)
金融领域对视觉-语言模型（VLMs）提出了重大挑战，原因是其特有的图表格式和知识密集型推理需求。然而，现有的金融基准测试主要集中在单一回合上，并依赖于有限的问题类型，这限制了在现实应用场景中的全面评估。为了弥补这一不足，我们提出了一种多轮多媒体基准测试——FinMTM，该测试通过数据和任务两个维度扩展多样性。 在数据方面，我们整理并标注了11,133对双语（中文和英文）金融问答对，这些问答基于财务可视化图表，包括蜡烛图、统计图和报告图形。在任务层面，FinMTM涵盖了单选题和多选题、多轮开放式对话以及代理任务。 我们进一步设计了特定于任务的评估协议，包括针对多项选择问题的一套重叠评分规则、用于多轮对话中的回合级和会话级得分加权组合方法，以及结合规划质量和最终成果的综合度量标准，专门针对代理任务。 通过22种视觉-语言模型（VLMs）的大规模实验评估显示了它们在细粒度视觉感知、长上下文推理以及复杂代理工作流程方面的局限性。
URL
https://arxiv.org/abs/2602.03130
