Abstract
The rise of Visual-Language Models (LVLMs) has unlocked new possibilities for seamlessly integrating visual and textual information. However, their ability to interpret cartographic maps remains largely unexplored. In this paper, we introduce CartoMapQA, a benchmark specifically designed to evaluate LVLMs' understanding of cartographic maps through question-answering tasks. The dataset includes over 2000 samples, each composed of a cartographic map, a question (with open-ended or multiple-choice answers), and a ground-truth answer. These tasks span key low-, mid- and high-level map interpretation skills, including symbol recognition, embedded information extraction, scale interpretation, and route-based reasoning. Our evaluation of both open-source and proprietary LVLMs reveals persistent challenges: models frequently struggle with map-specific semantics, exhibit limited geospatial reasoning, and are prone to Optical Character Recognition (OCR)-related errors. By isolating these weaknesses, CartoMapQA offers a valuable tool for guiding future improvements in LVLM architectures. Ultimately, it supports the development of models better equipped for real-world applications that depend on robust and reliable map understanding, such as navigation, geographic search, and urban planning. Our source code and data are openly available to the research community at: this https URL
Abstract (translated)
视觉语言模型（LVLMs）的兴起开启了无缝融合视觉和文本信息的新可能。然而，这些模型解读地图的能力仍然鲜有探索。本文介绍了一个专门设计用于通过问答任务评估LVLM对地图理解能力的基准测试——CartoMapQA。该数据集包括超过2000个样本，每个样本由一张地图、一个问题（开放或多项选择答案）以及一个真实答案组成。这些任务涵盖了关键的地图解读技能，从低级到高级，包括符号识别、嵌入信息提取、比例尺解读和路线推理等。 我们对开源及专有LVLM的评估揭示了持续存在的挑战：模型在处理地图特定语义时经常遇到困难，展现出有限的空间推理能力，并且容易出现与光学字符识别（OCR）相关的错误。通过隔离这些弱点，CartoMapQA为未来改进LVLM架构提供了一个有价值的工具。最终，它支持开发更适合于依赖可靠地图理解的真实世界应用的模型，例如导航、地理搜索和城市规划。 我们的源代码和数据对研究社区开放获取：[此URL](this https URL)
URL
https://arxiv.org/abs/2512.03558
