Abstract
CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it "see" the same way humans do - especially when interpreting artworks? In this paper, we investigate CLIP's ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions: content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP's responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP's visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role.
Abstract (translated)
CLIP作为一种能够通过联合嵌入连接图像和文本的强大多模态模型已经出现，但其“视觉”能力在多大程度上与人类相似——特别是在解读艺术品时？本文探讨了CLIP从绘画中提取高层次语义和风格信息的能力，包括由人创作的和AI生成的图像。我们通过多个维度评估其感知力：内容、场景理解、艺术风格、历史时期以及视觉变形或瑕疵的存在。通过设计有针对性的探测任务，并将CLIP的响应与人类注释和专家基准进行比较，我们探讨了它在人类知觉和语境理解方面的对齐情况。我们的研究结果揭示了CLIP在视觉表示上的强项和局限性，特别是在美学线索和艺术意图方面。此外，我们还讨论了这些见解对于使用CLIP作为生成过程（如风格转换或基于提示的图像合成）中的指导机制的影响。我们的工作强调了多模态系统在应用于需要微妙性和主观性的创意领域时对更深解读的需求。
URL
https://arxiv.org/abs/2505.05229
