Abstract
This paper proposes the progressive attention memory network (PAMN) for movie story question answering (QA). Movie story QA is challenging compared to VQA in two aspects: (1) pinpointing the temporal parts relevant to answer the question is difficult as the movies are typically longer than an hour, (2) it has both video and subtitle where different questions require different modality to infer the answer. To overcome these challenges, PAMN involves three main features: (1) progressive attention mechanism that utilizes cues from both question and answer to progressively prune out irrelevant temporal parts in memory, (2) dynamic modality fusion that adaptively determines the contribution of each modality for answering the current question, and (3) belief correction answering scheme that successively corrects the prediction score on each candidate answer. Experiments on publicly available benchmark datasets, MovieQA and TVQA, demonstrate that each feature contributes to our movie story QA architecture, PAMN, and improves performance to achieve the state-of-the-art result. Qualitative analysis by visualizing the inference mechanism of PAMN is also provided.
Abstract (translated)
本文提出了一种用于电影故事问答的渐进式注意力记忆网络（PAMN）。与vqa相比，电影故事qa在两个方面具有挑战性：（1）确定与回答问题相关的时间部分是困难的，因为电影通常超过一个小时；（2）它有视频和字幕，其中不同的问题需要不同的形式来推断答案。为了克服这些挑战，PAMN包含三个主要特征：（1）渐进式注意机制，利用问题和答案的线索，逐步删除记忆中不相关的时间部分；（2）动态模态融合，自适应地确定每种模态对回答当前问题的贡献；（3）被Lief修正回答方案，连续修正每个候选答案的预测分数。对公开的基准数据集movie qa和tvqa进行的实验表明，每一个特性都有助于我们的电影故事qa体系结构pamn，并提高性能以实现最先进的结果。并对PAMN的推理机制进行了可视化的定性分析。
URL
https://arxiv.org/abs/1904.08607
