Abstract
Video Frame Interpolation (VFI) aims to predict the intermediate frame $I_n$ (we use n to denote time in videos to avoid notation overload with the timestep $t$ in diffusion models) based on two consecutive neighboring frames $I_0$ and $I_1$. Recent approaches apply diffusion models (both image-based and video-based) in this task and achieve strong performance. However, image-based diffusion models are unable to extract temporal information and are relatively inefficient compared to non-diffusion methods. Video-based diffusion models can extract temporal information, but they are too large in terms of training scale, model size, and inference time. To mitigate the above issues, we propose Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation (TLB-VFI), an efficient video-based diffusion model. By extracting rich temporal information from video inputs through our proposed 3D-wavelet gating and temporal-aware autoencoder, our method achieves 20% improvement in FID on the most challenging datasets over recent SOTA of image-based diffusion models. Meanwhile, due to the existence of rich temporal information, our method achieves strong performance while having 3times fewer parameters. Such a parameter reduction results in 2.3x speed up. By incorporating optical flow guidance, our method requires 9000x less training data and achieves over 20x fewer parameters than video-based diffusion models. Codes and results are available at our project page: this https URL.
Abstract (translated)
视频帧插值（VFI）的目标是基于两个连续的邻近帧$I_0$和$I_1$预测中间帧$I_n$。最近的方法将扩散模型（包括图像型和视频型）应用于此任务，并取得了很好的效果。然而，基于图像的扩散模型无法提取时间信息，并且与非扩散方法相比效率较低。而基于视频的扩散模型可以提取时间信息，但它们在训练规模、模型大小以及推理时间方面过大。 为了缓解上述问题，我们提出了一种称为“时空感知隐变量布朗桥扩散视频帧插值”（TLB-VFI）的方法，这是一种高效的基于视频的扩散模型。通过使用我们提出的3D小波门控和时域感知自编码器从视频输入中提取丰富的时序信息，我们的方法在最具挑战性的数据集上比最近的最佳图像型扩散模型提高了20%的FID分数（一种衡量生成图像质量的标准）。同时，由于存在丰富的时序信息，我们的方法即使参数量减少了三倍也能取得强性能。这种参数减少带来了大约2.3倍的速度提升。通过引入光学流指导，我们的方法只需要视频型扩散模型1/9000的数据进行训练，并且参数数量少于其1/20。 代码和结果可以在我们的项目页面上找到：[此处插入具体网址]。
URL
https://arxiv.org/abs/2507.04984
