Abstract
There has been significant attention to the research on dense video captioning, which aims to automatically localize and caption all events within untrimmed video. Several studies introduce methods by designing dense video captioning as a multitasking problem of event localization and event captioning to consider inter-task relations. However, addressing both tasks using only visual input is challenging due to the lack of semantic content. In this study, we address this by proposing a novel framework inspired by the cognitive information processing of humans. Our model utilizes external memory to incorporate prior knowledge. The memory retrieval method is proposed with cross-modal video-to-text matching. To effectively incorporate retrieved text features, the versatile encoder and the decoder with visual and textual cross-attention modules are designed. Comparative experiments have been conducted to show the effectiveness of the proposed method on ActivityNet Captions and YouCook2 datasets. Experimental results show promising performance of our model without extensive pretraining from a large video dataset.
Abstract (translated)
近年来，对密集视频标注的研究受到了广泛关注，该研究旨在自动将未剪辑视频中的所有事件进行定位和标注。几项研究通过将密集视频标注作为一个多任务问题，将事件定位和事件标注相结合，来考虑任务之间的关系。然而，仅通过视觉输入来解决这两个任务是具有挑战性的，因为缺乏语义内容。在本研究中，我们通过提议一个以人类认知信息处理为基础的新框架来解决这个问题。我们的模型利用外部记忆来包含先验知识。提出了跨模态视频-文本匹配的内存检索方法。为了有效地包括检索到的文本特征，设计了一个灵活的编码器和一个具有视觉和文本ual跨注意力的解码器。对活动网络标注和YouCook2数据集进行了比较实验，以展示所提出方法的有效性。实验结果表明，在没有大量预训练视频数据的情况下，我们的模型具有鼓舞人心的性能。
URL
https://arxiv.org/abs/2404.07610
