Abstract
When visual evidence is ambiguous, vision models must decide whether to interpret face-like patterns as meaningful. Face pareidolia, the perception of faces in non-face objects, provides a controlled probe of this behavior. We introduce a representation-level diagnostic framework that analyzes detection, localization, uncertainty, and bias across class, difficulty, and emotion in face pareidolia images. Under a unified protocol, we evaluate six models spanning four representational regimes: vision-language models (VLMs; CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B), pure vision classification (ViT), general object detection (YOLOv8), and face detection (RetinaFace). Our analysis reveals three mechanisms of interpretation under ambiguity. VLMs exhibit semantic overactivation, systematically pulling ambiguous non-human regions toward the Human concept, with LLaVA-1.5-7B producing the strongest and most confident over-calls, especially for negative emotions. ViT instead follows an uncertainty-as-abstention strategy, remaining diffuse yet largely unbiased. Detection-based models achieve low bias through conservative priors that suppress pareidolia responses even when localization is controlled. These results show that behavior under ambiguity is governed more by representational choices than score thresholds, and that uncertainty and bias are decoupled: low uncertainty can signal either safe suppression, as in detectors, or extreme over-interpretation, as in VLMs. Pareidolia therefore provides a compact diagnostic and a source of ambiguity-aware hard negatives for probing and improving the semantic robustness of vision-language systems. Code will be released upon publication.
Abstract (translated)
当视觉证据模棱两可时，视觉模型必须决定是否将面部样式的图案视为有意义的。面对似真症（在非面部对象中感知到面孔）为这种行为提供了一个可控的研究方法。我们引入了一种表示层面的诊断框架，该框架分析了在人脸似真症图像中的检测、定位、不确定性和偏差，涵盖了类别、难度和情绪等多个方面。在统一协议下，我们评估了六种模型，这些模型跨越了四种表示范式：视觉-语言模型（VLMs；CLIP-B/32, CLIP-L/14, LLaVA-1.5-7B）、纯视觉分类（ViT）、通用对象检测（YOLOv8）和面部检测（RetinaFace）。我们的分析揭示了三种在模棱两可情况下进行解释的机制。VLMs表现出语义过度激活，系统地将模棱两可的人脸区域拉向“人类”概念，其中LLaVA-1.5-7B产生的过度解读最强且最自信，特别是在负面情绪的情况下。相比之下，ViT则遵循了不确定性即放弃策略，在很大程度上保持中立并且偏差较小。基于检测的模型通过保守的先验知识抑制似真反应来实现低偏差，即使在定位控制良好的情况下也是如此。 这些结果表明，模棱两可下的行为更多地受到表示选择的影响而非得分阈值，并且不确定性和偏见是解耦的：低不确定性既可以表示安全的抑制（如检测器），也可以表示极端过度解读（如VLMs）。因此，似真症提供了一个紧凑的诊断工具和一个模棱两可负面样本来源，用于探测并提高视觉-语言系统的语义鲁棒性。论文发布后将公开代码。
URL
https://arxiv.org/abs/2603.03989
