Abstract
Deep learning algorithms lack human-interpretable accounts of how they transform raw visual input into a robust semantic understanding, which impedes comparisons between different architectures, training objectives, and the human brain. In this work, we take inspiration from neuroscience and employ representational approaches to shed light on how neural networks encode information at low (visual saliency) and high (semantic similarity) levels of abstraction. Moreover, we introduce a custom image dataset where we systematically manipulate salient and semantic information. We find that ResNets are more sensitive to saliency information than ViTs, when trained with object classification objectives. We uncover that networks suppress saliency in early layers, a process enhanced by natural language supervision (CLIP) in ResNets. CLIP also enhances semantic encoding in both architectures. Finally, we show that semantic encoding is a key factor in aligning AI with human visual perception, while saliency suppression is a non-brain-like strategy.
Abstract (translated)
深度学习算法缺乏对它们如何将原始视觉输入转换为稳健的语义理解进行人类可解释的说明，这阻碍了不同架构、训练目标和人类大脑之间的比较。在本文中，我们从神经科学中汲取灵感，并采用表示方法阐明了神经网络在低（视觉显着性）和高（语义相似性）抽象水平上编码信息的过程。此外，我们还引入了一个自定义图像数据集，我们系统地操纵显着性和语义信息。我们发现，在以物体分类为目标训练ResNets时，ResNets对显着性信息的敏感性比ViTs更高。我们发现，在ResNets中，网络在早期层抑制显着性信息，这一过程通过自然语言监督（CLIP）得到了增强。CLIP还增强了两种架构的语义编码。最后，我们证明了语义编码是使AI与人类视觉感知保持一致的关键因素，而显着性抑制是一种非脑部策略。
URL
https://arxiv.org/abs/2404.18772
