Abstract
Despite the tremendous success of deep learning in computer vision, models still fall behind humans in generalizing to new input distributions. Existing benchmarks do not investigate the specific failure points of models by analyzing performance under many controlled conditions. Our study systematically dissects where and why models struggle with contour integration -- a hallmark of human vision -- by designing an experiment that tests object recognition under various levels of object fragmentation. Humans (n=50) perform at high accuracy, even with few object contours present. This is in contrast to models which exhibit substantially lower sensitivity to increasing object contours, with most of the over 1,000 models we tested barely performing above chance. Only at very large scales ($\sim5B$ training dataset size) do models begin to approach human performance. Importantly, humans exhibit an integration bias -- a preference towards recognizing objects made up of directional fragments over directionless fragments. We find that not only do models that share this property perform better at our task, but that this bias also increases with model training dataset size, and training models to exhibit contour integration leads to high shape bias. Taken together, our results suggest that contour integration is a hallmark of object vision that underlies object recognition performance, and may be a mechanism learned from data at scale.
Abstract (translated)
尽管深度学习在计算机视觉领域取得了巨大成功，但模型仍然无法像人类一样将知识泛化到新的输入分布中。现有的基准测试没有通过分析多种受控条件下的性能来具体调查模型的失败点。我们的研究系统地分析了模型为何以及在哪方面难以处理轮廓整合——这是人类视觉的一个标志性特征——我们设计了一个实验，该实验在不同程度的对象碎片化情况下测试物体识别能力。50名参与者（n=50）即使在仅存在少量对象轮廓的情况下也能以高准确率完成任务，这与大多数深度学习模型形成了鲜明对比，后者对增加的轮廓敏感度显著降低，我们在超过1,000个模型中进行测试时发现，大多数表现甚至低于随机猜测水平。只有当训练数据集规模达到非常大的程度（约5B）时，这些模型才开始接近人类的表现。 更重要的是，人类表现出一种整合偏见——倾向于识别由定向碎片构成的对象而非无方向性的碎片。我们发现不仅那些具有这种性质的模型在我们的任务中表现得更好，而且随着训练数据集规模的增加，这种偏向性也在增强，并且通过训练使模型展示轮廓整合会导致高形状偏好。 综上所述，我们的研究结果表明，轮廓整合是物体视觉的一个标志特征，对对象识别性能起着关键作用，并可能是从大规模数据中学习到的一种机制。
URL
https://arxiv.org/abs/2504.05253
