Abstract
In this work, we propose a new technique that combines appearance and text in a Convolutional Neural Network (CNN), with the aim of detecting regions of different textual categories. We define a novel visual representation of the semantic meaning of text that allows a seamless integration in a standard CNN architecture. This representation, referred to as text-map, is integrated with the actual image to provide a much richer input to the network. Text-maps are colored with different intensities depending on the relevance of the words recognized over the image. Concretely, these words are previously extracted using Optical Character Recognition (OCR) and they are colored according to the probability of belonging to a textual category of interest. In this sense, this solution is especially relevant in the context of item coding for supermarket products, where different types of textual categories must be identified, such as ingredients or nutritional facts. We evaluated our solution in the proprietary item coding dataset of Nielsen Brandbank, which contains more than 10,000 images for train and 2,000 images for test. The reported results demonstrate that our approach focused on visual and textual data outperforms state-of-the-art algorithms only based on appearance, such as standard Faster R-CNN. These enhancements are reflected in precision and recall, which are improved in 42 and 33 points respectively.
Abstract (translated)
在这项工作中，我们提出了一种新的技术，结合外观和文本在卷积神经网络（CNN），目的是检测不同文本类别的区域。我们定义了一种新颖的文本语义视觉表示，允许在标准CNN架构中无缝集成。这种表示法被称为文本映射，它与实际图像集成在一起，为网络提供更丰富的输入。根据图像上识别的单词的相关性，文本地图的颜色具有不同的强度。具体地说，这些单词以前是使用光学字符识别（OCR）提取的，它们是根据属于感兴趣的文本类别的概率着色的。从这个意义上讲，这个解决方案在超市产品的项目编码环境中尤其相关，在这种环境中，必须识别不同类型的文本类别，如成分或营养成分。我们在Nielsen Brandbank的专有项目编码数据集中评估了我们的解决方案，该数据集包含10000多张列车图像和2000多张测试图像。报告的结果表明，我们的方法专注于视觉和文本数据，优于仅基于外观的最先进算法，如标准更快的R-CNN。这些改进体现在精确性和召回方面，分别提高了42分和33分。
URL
https://arxiv.org/abs/1905.10858
