Abstract
This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answering is expensive and the existing benchmarks are relatively small to provide sufficient supervision. To overcome this challenge, this paper proposes a multi-task learning method which is composed of three main components: (1) multi-modal video question answering network that answers the question based on the both video and subtitle feature, (2) temporal retrieval network that predicts the time in the video clip where the question was generated from and (3) modality alignment network that solves metric learning problem to find correct association of video and subtitle modalities. By simultaneously solving related auxiliary tasks with hierarchically shared intermediate layers, the extra synergistic supervisions are provided. Motivated by curriculum learning, multi task ratio scheduling is proposed to learn easier task earlier to set inductive bias at the beginning of the training. The experiments on publicly available dataset TVQA shows state-of-the-art results, and ablation studies are conducted to prove the statistical validity.
Abstract (translated)
本文提出了一种通过多任务学习获得额外监督的多模式视频问答方法。多模态视频问答是一项旨在实现视觉和语言共同理解的重要任务。然而，建立大规模的多模式视频问答数据集代价高昂，现有的基准相对较小，无法提供足够的监督。为了克服这一挑战，本文提出了一种多任务学习方法，该方法由三个主要部分组成：（1）基于视频和字幕特征的多模式视频问答网络；（2）时间检索网络，预测视频剪辑中生成问题的时间。OM和（3）模态对齐网络，解决了度量学习问题，以找到视频和副标题模态的正确关联。通过同时解决具有层次共享中间层的相关辅助任务，提供了额外的协同监控。在课程学习的激励下，提出了多任务比调度的方法，以便在培训开始时提前学习更容易的任务，从而设置归纳偏差。在公共可用数据集TVCA上进行的实验显示了最新的结果，并进行了消融研究以证明统计有效性。
URL
https://arxiv.org/abs/1905.13540
