Abstract
The drastic variation of motion in spatial and temporal dimensions makes the video prediction task extremely challenging. Existing RNN models obtain higher performance by deepening or widening the model. They obtain the multi-scale features of the video only by stacking layers, which is inefficient and brings unbearable training costs (such as memory, FLOPs, and training time). Different from them, this paper proposes a spatiotemporal multi-scale model called MS-LSTM wholly from a multi-scale perspective. On the basis of stacked layers, MS-LSTM incorporates two additional efficient multi-scale designs to fully capture spatiotemporal context information. Concretely, we employ LSTMs with mirrored pyramid structures to construct spatial multi-scale representations and LSTMs with different convolution kernels to construct temporal multi-scale representations. Detailed comparison experiments with eight baseline models on four video datasets show that MS-LSTM has better performance but lower training costs.
Abstract (translated)
在空间和时间维度上的变化剧烈使得视频预测任务极其具有挑战性。现有的RNN模型通过加深或拓宽模型来获得更高的性能。他们只能通过堆叠层来获取视频的多尺度特征，这非常低效并且带来了难以承受的训练成本(如内存、Flops、训练时间)。与它们不同，本文提出了一种名为MS-LSTM的时空多尺度模型，完全从多尺度的角度提出。基于堆叠的层，MS-LSTM引入了两个高效的多尺度设计，以完全捕捉时空上下文信息。具体而言，我们使用具有 mirrorPyramid结构的LSTM构建空间多尺度表示，使用不同的卷积核构建时间多尺度表示。在四个视频数据集上与8个基准模型进行详细的比较实验，结果表明MS-LSTM具有更好的性能，但训练成本更低。
URL
https://arxiv.org/abs/2304.07724
