Abstract
We address the problem of video captioning by grounding language generation on object interactions in the video. Existing work mostly focuses on overall scene understanding with often limited or no emphasis on object interactions to address the problem of video understanding. In this paper, we propose SINet-Caption that learns to generate captions grounded over higher-order interactions between arbitrary groups of objects for fine-grained video understanding. We discuss the challenges and benefits of such an approach. We further demonstrate state-of-the-art results on the ActivityNet Captions dataset using our model, SINet-Caption based on this approach.
Abstract (translated)
我们通过将语言生成与视频中的对象交互相结合来解决视频字幕问题。现有的工作主要集中在整体场景理解上，通常很少或不重视对象交互以解决视频理解问题。在本文中，我们提出了SINet-Caption，它可以学习如何生成基于任意对象组之间高阶相互作用的字幕，以实现细致的视频理解。我们讨论这种方法的挑战和好处。我们使用我们的模型，基于这种方法的SINet-Caption，进一步在ActivityNet Captions数据集上展示了最新的结果。
URL
https://arxiv.org/abs/1711.06354
