Abstract
With their high information density and intuitive readability, charts have become the de facto medium for data analysis and communication across disciplines. Recent multimodal large language models (MLLMs) have made notable progress in automated chart understanding, yet they remain heavily dependent on explicit textual annotations and the performance degrades markedly when key numerals are absent. To address this limitation, we introduce ChartAgent, a chart understanding framework grounded in Tool-Integrated Reasoning (TIR). Inspired by human cognition, ChartAgent decomposes complex chart analysis into a sequence of observable, replayable steps. Supporting this architecture is an extensible, modular tool library comprising more than a dozen core tools, such as keyelement detection, instance segmentation, and optical character recognition (OCR), which the agent dynamically orchestrates to achieve systematic visual parsing across diverse chart types. Leveraging TIRs transparency and verifiability, ChartAgent moves beyond the black box paradigm by standardizing and consolidating intermediate outputs into a structured Evidence Package, providing traceable and reproducible support for final conclusions. Experiments show that ChartAgent substantially improves robustness under sparse annotation settings, offering a practical path toward trustworthy and extensible systems for chart understanding.
Abstract (translated)
凭借其高信息密度和直观易读性，图表已成为跨学科数据解析与交流的公认媒介。近年来，多模态大型语言模型（MLLMs）在自动化图表理解方面取得了显著进展，但它们仍然严重依赖明确的文字注释，并且当关键数字缺失时性能会明显下降。为了解决这一局限性，我们引入了基于工具集成推理（TIR）的图表理解框架ChartAgent。受到人类认知启发，ChartAgent将复杂的图表分析分解成一系列可观测、可重演的操作步骤。支撑这种架构的是一个扩展性强且模块化的工具库，包括关键元素检测、实例分割和光学字符识别（OCR）等超过十种核心工具，这些工具由代理动态协调以实现不同类型的系统化视觉解析。通过TIR的透明性和可验证性，ChartAgent超越了黑盒模型范式，通过标准化并整合中间输出为结构化的证据包，提供最终结论的可追溯性和可重复性支持。实验表明，在注释稀疏的情况下，ChartAgent显著提高了鲁棒性，并为图表理解提供了值得信赖且具有扩展性的系统实现途径。
URL
https://arxiv.org/abs/2512.14040
