Abstract
Internet memes are characterised by the interspersing of text amongst visual elements. State-of-the-art multimodal meme classifiers do not account for the relative positions of these elements across the two modalities, despite the latent meaning associated with where text and visual elements are placed. Against two meme sentiment classification datasets, we systematically show performance gains from incorporating the spatial position of visual objects, faces, and text clusters extracted from memes. In addition, we also present facial embedding as an impactful enhancement to image representation in a multimodal meme classifier. Finally, we show that incorporating this spatial information allows our fully automated approaches to outperform their corresponding baselines that rely on additional human validation of OCR-extracted text.
Abstract (translated)
互联网Meme的特点是将文本穿插于视觉元素之间。最先进的多模式Meme分类器并未考虑到这些元素在两种模式之间的相对位置,尽管它们在放置文本和视觉元素时的潜在含义相关。在面对两个Meme情感分类数据集时,我们系统地展示了从Meme中提取的视觉对象、人脸和文本簇的 spatial 位置对图像表示的影响。此外,我们还提出了面部嵌入作为多模式Meme分类器中图像表示的一种重要增强方法。最后,我们表明,包括这种空间信息使我们的完全自动化方法能够比依赖于人工验证OCR提取的文本的对应基线表现更好。
URL
https://arxiv.org/abs/2303.01781
