Abstract
Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, we propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. Thanks to these modules, our EDVR wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. EDVR also demonstrates superior performance to state-of-the-art published methods on video super-resolution and deblurring. The code is available at https://github.com/xinntao/EDVR.
Abstract (translated)
视频恢复任务,包括超分辨率、去模糊等,越来越受到计算机视觉界的关注。在ntire19挑战中发布了一个名为Reds的具有挑战性的基准测试。这项新的基准测试从两个方面挑战了现有的方法:(1)如何在大运动下对齐多个帧;(2)如何有效地融合不同运动和模糊的帧。在这项工作中,我们提出了一个新的视频恢复框架与增强变形网络,称为EDVR,以解决这些挑战。首先,为了处理大的运动,我们设计了一个金字塔、级联和可变形(PCD)对齐模块,在这个模块中,帧对齐是在特征级别使用可变形卷积以粗到细的方式完成的。第二,我们提出了一个时空注意融合模块,它将注意力从时间和空间两个方面加以应用,以强调后续恢复的重要特征。得益于这些模块,我们的EDVR在NTIR19视频恢复和增强挑战中赢得了冠军,并在所有四个曲目中以巨大的优势超越了第二名。EDVR在视频超分辨率和去模糊方面的表现也优于最先进的公开方法。该代码可在https://github.com/xinnato/edvr上找到。
URL
https://arxiv.org/abs/1905.02716