Abstract
Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos.
Abstract (translated)
时间关系推理，即随时间推移对象或实体的有意义转换的能力，是智能物种的基本属性。在本文中，我们介绍了一个有效且可解释的网络模块，即时间关系网络（TRN），旨在学习和推理多个时间尺度的视频帧之间的时间依赖性。我们使用三个最近的视频数据集 - Something-Something，Jester和Charades - 评估TRN配备的网络活动识别任务，这些数据集从根本上依赖于时间关系推理。我们的结果表明，所提出的TRN使卷积神经网络具有发现视频中时间关系的显着能力。通过仅稀疏采样的视频帧，配备TRN的网络可以准确地预测Something-Something数据集中的人 - 物体相互作用，并在Jester数据集上识别具有极具竞争性的各种人类姿势。配备TRN的网络在识别Charades数据集中的日常活动方面也优于双流网络和3D卷积网络。进一步的分析表明，模型在视频中学习直观和可解释的视觉常识知识。
URL
https://arxiv.org/abs/1711.08496
