Abstract
Video frame prediction extrapolates future frames from previous frames, but suffers from prediction errors in dynamic scenes due to the lack of information about the next frame. Event cameras address this limitation by capturing per-pixel brightness changes asynchronously with high temporal resolution. Prior research on event-based video frame prediction has leveraged motion information from event data, often by predicting event-based optical flow and reconstructing frames via pixel warping. However, such approaches introduce holes and blurring when pixel displacement is inaccurate. To overcome this limitation, we propose DESSERT, a diffusion-based event-driven single-frame synthesis framework via residual training. Leveraging a pre-trained Stable Diffusion model, our method is trained on inter-frame residuals to ensure temporal consistency. The training pipeline consists of two stages: (1) an Event-to-Residual Alignment Variational Autoencoder (ER-VAE) that aligns the event frame between anchor and target frames with the corresponding residual, and (2) a diffusion model that denoises the residual latent conditioned on event data. Furthermore, we introduce Diverse-Length Temporal (DLT) augmentation, which improves robustness by training on frame segments of varying temporal lengths. Experimental results demonstrate that our method outperforms existing event-based reconstruction, image-based video frame prediction, event-based video frame prediction, and one-sided event-based video frame interpolation methods, producing sharper and more temporally consistent frame synthesis.
Abstract (translated)
视频帧预测通过从前一帧中推断未来帧来进行，但在动态场景中会因为缺乏下一帧的信息而导致预测误差。事件相机（event cameras）通过异步捕捉每个像素的亮度变化来解决这一问题，从而提供高时间分辨率的数据。先前基于事件的数据进行视频帧预测的研究通常利用事件数据中的运动信息，例如通过预测基于事件的光流并使用像素扭曲重建帧的方式实现。然而，此类方法在像素位移不准确时会产生孔洞和模糊。 为了克服这些限制，我们提出了DESSERT框架，这是一个基于扩散模型（diffusion-based）并通过残差训练进行单帧合成的事件驱动框架。该方法利用预训练的Stable Diffusion模型，在帧间差异的基础上进行训练以确保时间一致性。训练流程分为两个阶段： 1. 事件到残差对齐变分自编码器（ER-VAE），将锚定帧和目标帧之间的事件帧与对应的残差点对齐。 2. 扩散模型，基于事件数据清理残差隐变量的噪声。 此外，我们还引入了多样时间长度增强技术（DLT augmentation），通过在不同时间长度的帧段上进行训练来提高鲁棒性。实验结果显示，我们的方法超过了现有的事件基重构、图像基视频帧预测、事件基视频帧预测以及单向事件基视频帧插值的方法，在生成更清晰且时间一致性更好的合成帧方面表现出色。
URL
https://arxiv.org/abs/2512.17323
