Abstract
This paper presents a framework for localization or grounding of phrases in images using a large collection of linguistic and visual cues. We model the appearance, size, and position of entity bounding boxes, adjectives that contain attribute information, and spatial relationships between pairs of entities connected by verbs or prepositions. Special attention is given to relationships between people and clothing or body part mentions, as they are useful for distinguishing individuals. We automatically learn weights for combining these cues and at test time, perform joint inference over all phrases in a caption. The resulting system produces state of the art performance on phrase localization on the Flickr30k Entities dataset and visual relationship detection on the Stanford VRD dataset.
Abstract (translated)
本文提出了一个使用大量语言和视觉线索为图像中的短语定位或搁置的框架。我们模拟实体边界框的外观，大小和位置，包含属性信息的形容词以及由动词或介词连接的实体对之间的空间关系。特别注意人与服装或身体部位之间的关系，因为它们有助于区分个体。我们会自动学习合并这些线索的权重，并在测试时间对标题中的所有短语执行联合推断。由此产生的系统可以在Flickr30k实体数据集上进行短语定位和Stanford VRD数据集上的视觉关系检测。
URL
https://arxiv.org/abs/1611.06641
