Abstract
Video Frame Interpolation (VFI) is a fundamental yet challenging task in computer vision, particularly under conditions involving large motion, occlusion, and lighting variation. Recent advancements in event cameras have opened up new opportunities for addressing these challenges. While existing event-based VFI methods have succeeded in recovering large and complex motions by leveraging handcrafted intermediate representations such as optical flow, these designs often compromise high-fidelity image reconstruction under subtle motion scenarios due to their reliance on explicit motion modeling. Meanwhile, diffusion models provide a promising alternative for VFI by reconstructing frames through a denoising process, eliminating the need for explicit motion estimation or warping operations. In this work, we propose EventDiff, a unified and efficient event-based diffusion model framework for VFI. EventDiff features a novel Event-Frame Hybrid AutoEncoder (HAE) equipped with a lightweight Spatial-Temporal Cross Attention (STCA) module that effectively fuses dynamic event streams with static frames. Unlike previous event-based VFI methods, EventDiff performs interpolation directly in the latent space via a denoising diffusion process, making it more robust across diverse and challenging VFI scenarios. Through a two-stage training strategy that first pretrains the HAE and then jointly optimizes it with the diffusion model, our method achieves state-of-the-art performance across multiple synthetic and real-world event VFI datasets. The proposed method outperforms existing state-of-the-art event-based VFI methods by up to 1.98dB in PSNR on Vimeo90K-Triplet and shows superior performance in SNU-FILM tasks with multiple difficulty levels. Compared to the emerging diffusion-based VFI approach, our method achieves up to 5.72dB PSNR gain on Vimeo90K-Triplet and 4.24X faster inference.
Abstract (translated)
视频帧插值（VFI）是计算机视觉中一个基础但具有挑战性的任务，尤其是在处理大运动、遮挡和光照变化的情况下。最近在事件相机领域的进展为解决这些挑战提供了新的机会。现有的基于事件的VFI方法通过利用诸如光流等手工设计的中间表示来成功恢复大量复杂动作，但在细微运动场景下的高保真图像重建方面往往因依赖显式的运动建模而有所妥协。同时，扩散模型通过去噪过程重构帧提供了一种有前景的替代方案，消除了对显式运动估计或扭曲操作的需求。在这项工作中，我们提出了EventDiff，这是一种统一且高效的基于事件的扩散模型框架用于VFI。EventDiff具备一种新颖的事件-帧混合自编码器（HAE），配有一个轻量级的空间-时间交叉注意（STCA）模块，该模块能够有效地融合动态事件流与静态图像帧。不同于以往的基于事件的VFI方法，EventDiff通过在潜在空间中进行去噪扩散过程直接执行插值操作，在各种复杂和挑战性的VFI场景下表现出更强的鲁棒性。 我们采用两阶段训练策略：首先预训练HAE，然后将其与扩散模型共同优化。本方法在多个合成及真实世界事件VFI数据集上实现了最先进的性能。相较于现有的基于事件的VFI方法，我们的方法在Vimeo90K-Triplet上的PSNR值提高了高达1.98dB，并且在SNU-FILM任务中不同难度级别的表现更为优越。与新兴的基于扩散的VFI方法相比，在Vimeo90K-Triplet上本方法实现了高达5.72dB的PSNR增益，同时推理速度提升了4.24倍。
URL
https://arxiv.org/abs/2505.08235
