Abstract
Video frame interpolation (VFI) that leverages the bio-inspired event cameras as guidance has recently shown better performance and memory efficiency than the frame-based methods, thanks to the event cameras' advantages, such as high temporal resolution. A hurdle for event-based VFI is how to effectively deal with non-linear motion, caused by the dynamic changes in motion direction and speed within the scene. Existing methods either use events to estimate sparse optical flow or fuse events with image features to estimate dense optical flow. Unfortunately, motion errors often degrade the VFI quality as the continuous motion cues from events do not align with the dense spatial information of images in the temporal dimension. In this paper, we find that object motion is continuous in space, tracking local regions over continuous time enables more accurate identification of spatiotemporal feature correlations. In light of this, we propose a novel continuous point tracking-based VFI framework, named TimeTracker. Specifically, we first design a Scene-Aware Region Segmentation (SARS) module to divide the scene into similar patches. Then, a Continuous Trajectory guided Motion Estimation (CTME) module is proposed to track the continuous motion trajectory of each patch through events. Finally, intermediate frames at any given time are generated through global motion optimization and frame refinement. Moreover, we collect a real-world dataset that features fast non-linear motion. Extensive experiments show that our method outperforms prior arts in both motion estimation and frame interpolation quality.
Abstract (translated)
视频帧插值（VFI）技术利用受生物启发的事件相机作为指导，近期表现出优于基于帧的方法的性能和内存效率，这得益于事件相机的优势，如高时间分辨率。然而，基于事件的VFI的一个挑战是如何有效处理非线性运动，这是由于场景中运动方向和速度的动态变化引起的。现有方法要么利用事件来估计稀疏光流，要么融合事件与图像特征以估计密集光流。不幸的是，这些方法中的运动误差往往降低了VFI的质量，因为连续的运动线索（来自事件）在时间维度上并不总是与图像的空间信息对齐。 在这篇论文中，我们发现物体的运动在空间上是连续的，并且通过持续的时间跟踪局部区域能够更准确地识别时空特征的相关性。基于这一发现，我们提出了一种新颖的基于连续点追踪的VFI框架，名为TimeTracker。具体来说，我们首先设计了一个场景感知区域分割（SARS）模块来将场景划分为相似的小块。然后，提出了一个连续轨迹引导运动估计（CTME）模块，通过事件跟踪每个小块的持续运动轨迹。最后，通过全局运动优化和帧细化生成任意时间点上的中间帧。 此外，我们收集了一个具有快速非线性运动的真实世界数据集。广泛的实验表明，我们的方法在运动估计和插值帧质量方面都优于现有的技术。
URL
https://arxiv.org/abs/2505.03116
