Abstract
Existing video prediction methods mainly rely on observing multiple historical frames or focus on predicting the next one-frame. In this work, we study the problem of generating consecutive multiple future frames by observing one single still image only. We formulate the multi-frame prediction task as a multiple time step flow (multi-flow) prediction phase followed by a flow-to-frame synthesis phase. The multi-flow prediction is modeled in a variational probabilistic manner with spatial-temporal relationships learned through 3D convolutions. The flow-to-frame synthesis is modeled as a generative process in order to keep the predicted results lying closer to the manifold shape of real video sequence. Such a two-phase design prevents the model from directly looking at the high-dimensional pixel space of the frame sequence and is demonstrated to be more effective in predicting better and diverse results. Extensive experimental results on videos with different types of motion show that the proposed algorithm performs favorably against existing methods in terms of quality, diversity and human perceptual evaluation.
Abstract (translated)
现有的视频预测方法主要依赖于观察多个历史帧或专注于预测下一个帧。在这项工作中，我们通过仅观察一个单一静止图像来研究生成连续多个未来帧的问题。我们将多帧预测任务制定为多时间步流（多流）预测阶段，然后是流到帧合成阶段。多流预测以变分概率方式建模，其中通过3D卷积学习时空关系。流 - 帧合成被建模为生成过程，以便使预测结果更接近真实视频序列的流形。这种两相设计防止模型直接观察帧序列的高维像素空间，并且被证明在预测更好和多样化的结果方面更有效。对具有不同类型运动的视频的广泛实验结果表明，所提出的算法在质量，多样性和人类感知评估方面对现有方法表现出良好的效果。
URL
https://arxiv.org/abs/1807.09755
