Abstract
A major challenge for video captioning is to combine audio and visual cues. Existing multi-modal fusion methods have shown encouraging results in video understanding. However, the temporal structures of multiple modalities at different granularities are rarely explored, and how to selectively fuse the multi-modal representations at different levels of details remains uncharted. In this paper, we propose a novel hierarchically aligned cross-modal attention (HACA) framework to learn and selectively fuse both global and local temporal dynamics of different modalities. Furthermore, for the first time, we validate the superior performance of the deep audio features on the video captioning task. Finally, our HACA model significantly outperforms the previous best systems and achieves new state-of-the-art results on the widely used MSR-VTT dataset.
Abstract (translated)
视频字幕的一个主要挑战是结合音频和视觉线索。现有的多模式融合方法在视频理解上显示出令人鼓舞的结果。然而，不同粒度的多种模态的时间结构很少被探索，如何有选择地融合不同细节层面上的多模态表征仍然是未知的。在本文中，我们提出了一种新型的层次化对齐的跨模态关注（HACA）框架，以学习和选择性融合不同形式的全球和当地时间动态。此外，我们还首次验证了视频字幕任务中深度音频功能的卓越性能。最后，我们的HACA模型显着优于以前的最佳系统，并在广泛使用的MSR-VTT数据集上实现了最新的最新成果。
URL
https://arxiv.org/abs/1804.05448
