Abstract
Automatic art analysis aims to classify and retrieve artistic representations from a collection of images by using computer vision and machine learning techniques. In this work, we propose to enhance visual representations from neural networks with contextual artistic information. Whereas visual representations are able to capture information about the content and the style of an artwork, our proposed context-aware embeddings additionally encode relationships between different artistic attributes, such as author, school, or historical period. We design two different approaches for using context in automatic art analysis. In the first one, contextual data is obtained through a multi-task learning model, in which several attributes are trained together to find visual relationships between elements. In the second approach, context is obtained through an art-specific knowledge graph, which encodes relationships between artistic attributes. An exhaustive evaluation of both of our models in several art analysis problems, such as author identification, type classification, or cross-modal retrieval, show that performance is improved by up to 7.3% in art classification and 37.24% in retrieval when context-aware embeddings are used.
Abstract (translated)
自动艺术分析的目的是通过使用计算机视觉和机器学习技术从一组图像中分类和检索艺术表现。在这项工作中，我们建议用上下文艺术信息增强神经网络的视觉表现。虽然视觉表现能够捕捉到关于艺术品内容和风格的信息，但我们提出的上下文感知嵌入还编码了不同艺术属性（如作者、学校或历史时期）之间的关系。我们设计了两种在自动艺术分析中使用上下文的不同方法。在第一个模型中，通过多任务学习模型获得上下文数据，其中几个属性一起训练，以发现元素之间的视觉关系。在第二种方法中，上下文是通过一个特定于艺术的知识图获得的，它编码了艺术属性之间的关系。对我们的两个模型在几个艺术分析问题（如作者识别、类型分类或跨模式检索）中的详尽评估表明，当使用上下文感知嵌入时，艺术分类的性能提高了7.3%，检索的性能提高了37.24%。
URL
https://arxiv.org/abs/1904.04985
