Abstract
Visual Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an evaluation that tests vision-language models' capacity for nonlocal visual reasoning -- reasoning that requires chaining evidence collected from multiple, possibly distant, regions of an image. We isolate three distinct forms of non-local vision: comparative perception, which demands holding two images in working memory and comparing them; saccadic search, which requires making discrete, evidence-driven jumps to locate successive targets; and smooth visual search, which involves searching smoothly along a continuous contour. Flagship models (e.g., Gemini 2.5 Pro, Claude Vision 3.7, GPT-o4-mini), even those that perform well on prior primitive-vision benchmarks, fail these tests and barely exceed random accuracy on two variants of our tasks that are trivial for humans. Our structured evaluation suite allows us to test if VLMs can perform similar visual algorithms to humans. Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
Abstract (translated)
视觉语言模型（VLMs）在复杂视觉任务如视觉问答（VQA）和图表理解方面表现出色，但最近的研究表明它们在简单的感知测试中表现不佳。我们提出了一种评估方法，用于检验视觉-语言模型进行非局部视觉推理的能力——这种推理需要从图像中的多个、可能相距甚远的区域收集证据并将其串联起来。我们将非局部视觉分为三种不同的形式：比较感知（要求将两张图片保留在工作记忆中并进行对比），扫视搜索（需要根据证据做出离散跳跃以定位连续的目标），以及平滑视觉搜索（涉及沿着连续轮廓进行平滑搜索）。 即使是一些旗舰模型（如Gemini 2.5 Pro、Claude Vision 3.7 和 GPT-o4-mini），在先前的基础视觉基准测试中表现良好，但在我们设计的任务变体中也失败了。这些任务对人类来说是微不足道的，但模型的表现仅略高于随机猜测水平。 我们的结构化评估套件使我们能够检验VLM是否可以执行与人类相似的视觉算法。研究结果表明，尽管在基本视觉敏锐度方面有所提升，目前的模型仍然缺乏核心的视觉推理能力。
URL
https://arxiv.org/abs/2507.13361
