Abstract
We introduce VideoFlow, a novel optical flow estimation framework for videos. In contrast to previous methods that learn to estimate optical flow from two frames, VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. The information of the frame triplet is iteratively fused onto the center frame. To extend TROF for handling more frames, we further propose a MOtion Propagation (MOP) module that bridges multiple TROFs and propagates motion features between adjacent TROFs. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP. By effectively exploiting video information, VideoFlow presents extraordinary performance, ranking 1st on all public benchmarks. On the Sintel benchmark, VideoFlow achieves 1.649 and 0.991 average end-point-error (AEPE) on the final and clean passes, a 15.1% and 7.6% error reduction from the best published results (1.943 and 1.073 from FlowFormer++). On the KITTI-2015 benchmark, VideoFlow achieves an F1-all error of 3.65%, a 19.2% error reduction from the best published result (4.52% from FlowFormer++).
Abstract (translated)
我们介绍了VideoFlow,一个用于视频的全新的光学流估计框架。与以前的方法和学习从两帧中估计光学流不同,VideoFlow通过充分利用时间线索,同时估计多帧中可用的双向光学流,具体方法是提出TROF模块,以三帧的方式估计中心帧的双向光学流。将帧triplet的信息迭代地融合到中心帧上。为了扩展TROF处理更多帧,我们还提出MOtion Propagation(MOP)模块,将多个TROF连接并传递相邻TROF之间的运动特征。通过迭代的流估计优化,将融合在单个TROF中的信息通过MOP传播到整个序列。通过有效地利用视频信息,VideoFlow表现出卓越的性能,在所有公开基准中排名第一。在Sintel基准测试中,VideoFlow在最终干净帧上的AEPE平均为1.649和0.991,比最好的 published 结果(1.943和1.073,来自Flow former++)减少了15.1%和7.6%。在KITTI-2015基准测试中,VideoFlow的F1-全部误差为3.65%,比最好的 published 结果(4.52%,来自Flow former++)减少了19.2%。
URL
https://arxiv.org/abs/2303.08340
