Abstract
We introduce VisualNews, a large-scale dataset collected from four news agencies consisting of more than one million news images along with associated news articles, image captions, author information, and other metadata. We also propose VisualNews-Captioner, a model for the task of news image captioning. Unlike the standard image captioning task, news images depict situations where people, locations, andevents are of paramount importance. Our proposed method is able to effectively combine visual and textual features to generate captions with richer information such as events and entities. More specifically, we propose an Entity-Aware module along with an Entity-Guide attention layer to encourage more accurate predictions for named entities. Our method achieves new state-of-the-art results on both GoodNews and VisualNews datasets while having significantly fewer parameters than competing methods. Our larger and more diverse VisualNews dataset further highlights the remaining challenges in news image captioning.
Abstract (translated)
URL
https://arxiv.org/abs/2010.03743
