Abstract
Theory of Mind (ToM) -- the ability to attribute beliefs, desires, and emotions to others -- is fundamental for human social intelligence, yet remains a major challenge for artificial agents. Existing Vision-Language Models (VLMs) are increasingly applied in socially grounded tasks, but their capacity for cross-cultural ToM reasoning is largely unexplored. In this work, we introduce CulturalToM-VQA, a new evaluation benchmark containing 5095 questions designed to probe ToM reasoning across diverse cultural contexts through visual question answering. The dataset captures culturally grounded cues such as rituals, attire, gestures, and interpersonal dynamics, enabling systematic evaluation of ToM reasoning beyond Western-centric benchmarks. Our dataset is built through a VLM-assisted human-in-the-loop pipeline, where human experts first curate culturally rich images across traditions, rituals, and social interactions; a VLM then assist in generating structured ToM-focused scene descriptions, which are refined into question-answer pairs spanning a taxonomy of six ToM tasks and four graded complexity levels. The resulting dataset covers diverse theory of mind facets such as mental state attribution, false belief reasoning, non-literal communication, social norm violations, perspective coordination, and multi-agent reasoning.
Abstract (translated)
理论思维（ToM）——即能够赋予他人信念、欲望和情感的能力——是人类社交智能的基础，但对于人工代理来说仍是一项重大挑战。现有的视觉语言模型（VLMs）越来越多地被应用于社会环境中，但它们在跨文化背景下进行理论思维推理的能力尚待探索。 在此研究中，我们引入了CulturalToM-VQA，这是一个新的评估基准，包含5095个问题，旨在通过视觉问答的方式探究跨不同文化背景下的理论思维能力。该数据集捕捉到了如仪式、服饰、手势和人际动态等具有文化基础的线索，从而能够系统地评估超出西方中心主义基准范围内的理论思维推理能力。 我们的数据集是通过一个VLM辅助的人机协作管道构建而成，在这个过程中，人类专家首先整理出跨不同传统、礼仪和社会互动的文化丰富图像；随后，VLM协助生成结构化的关注于场景描述的理论思维内容，这些内容被进一步转化为跨越六个理论任务类别和四个分级复杂度水平的问题-答案对。最终的数据集涵盖了多样化的理论思维方面，如心理状态归属、虚假信念推理、非字面沟通、社会规范违规、视角协调以及多主体推理。
URL
https://arxiv.org/abs/2512.17394
