Abstract
Predicting future video frames is essential for decision-making systems, yet RGB frames alone often lack the information needed to fully capture the underlying complexities of the real world. To address this limitation, we propose a multi-modal framework for Synchronous Video Prediction (SyncVP) that incorporates complementary data modalities, enhancing the richness and accuracy of future predictions. SyncVP builds on pre-trained modality-specific diffusion models and introduces an efficient spatio-temporal cross-attention module to enable effective information sharing across modalities. We evaluate SyncVP on standard benchmark datasets, such as Cityscapes and BAIR, using depth as an additional modality. We furthermore demonstrate its generalization to other modalities on SYNTHIA with semantic information and ERA5-Land with climate data. Notably, SyncVP achieves state-of-the-art performance, even in scenarios where only one modality is present, demonstrating its robustness and potential for a wide range of applications.
Abstract (translated)
预测未来的视频帧对于决策系统至关重要，然而仅依靠RGB图像通常无法充分捕捉现实世界的复杂性。为了解决这一局限性，我们提出了一种多模态框架——同步视频预测（SyncVP），该框架整合了互补的数据模式，增强了对未来预测的丰富性和准确性。SyncVP基于预训练的特定模式扩散模型，并引入了一个高效的时空交叉注意力模块，以促进不同模式间的信息共享。我们在标准基准数据集上评估了SyncVP的效果，例如Cityscapes和BAIR，在这些数据集中使用深度信息作为额外模态。此外，我们还展示了它在SYNTHIA（带有语义信息）和ERA5-Land（带有气候数据）上的跨其他模式的泛化能力。值得注意的是，即使在仅存在单一模态的情况下，SyncVP也实现了最先进的性能，这表明其鲁棒性和广泛应用潜力。
URL
https://arxiv.org/abs/2503.18933
