Abstract
Stochastic video prediction enables the consideration of uncertainty in future motion, thereby providing a better reflection of the dynamic nature of the environment. Stochastic video prediction methods based on image auto-regressive recurrent models need to feed their predictions back into the latent space. Conversely, the state-space models, which decouple frame synthesis and temporal prediction, proves to be more efficient. However, inferring long-term temporal information about motion and generalizing to dynamic scenarios under non-stationary assumptions remains an unresolved challenge. In this paper, we propose a state-space decomposition stochastic video prediction model that decomposes the overall video frame generation into deterministic appearance prediction and stochastic motion prediction. Through adaptive decomposition, the model's generalization capability to dynamic scenarios is enhanced. In the context of motion prediction, obtaining a prior on the long-term trend of future motion is crucial. Thus, in the stochastic motion prediction branch, we infer the long-term motion trend from conditional frames to guide the generation of future frames that exhibit high consistency with the conditional frames. Experimental results demonstrate that our model outperforms baselines on multiple datasets.
Abstract (translated)
随机视频预测使得可以考虑未来运动中的不确定性，从而更好地反映了环境的动态性质。基于图像自回归循环模型的随机视频预测方法需要将预测结果反馈到潜在空间中。相反，基于状态空间模型的方法被证明更加有效。然而，根据非平稳假设推断运动和泛化长期信息仍然是一个未解决的挑战。在本文中，我们提出了一种基于状态空间分解的随机视频预测模型，将整个视频帧生成分解为确定性可见预测和随机运动预测。通过自适应分解，模型的泛化能力得到增强。在运动预测方面，获得对未来运动的长期趋势的先前了解至关重要。因此，在随机运动预测分支中，我们从条件帧中推断长期运动趋势，以指导生成与条件帧具有高一致性的未来帧。实验结果表明，我们的模型在多个数据集上的表现优于基线。
URL
https://arxiv.org/abs/2404.11576
