Abstract
Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features associated with the corrupted regions are degraded and incur inaccurate self attention. This problem, termed query degradation, may be mitigated by first completing optical flows and then using the flows to guide the self attention, which was verified in our previous work - flow-guided transformer (FGT). We further exploit the flow guidance and propose FGT++ to pursue more effective and efficient video inpainting. First, we design a lightweight flow completion network by using local aggregation and edge loss. Second, to address the query degradation, we propose a flow guidance feature integration module, which uses the motion discrepancy to enhance the features, together with a flow-guided feature propagation module that warps the features according to the flows. Third, we decouple the transformer along the temporal and spatial dimensions, where flows are used to select the tokens through a temporally deformable MHSA mechanism, and global tokens are combined with the inner-window local tokens through a dual perspective MHSA mechanism. FGT++ is experimentally evaluated to be outperforming the existing video inpainting networks qualitatively and quantitatively.
Abstract (translated)
Transformer因其多目自主关注机制(MHSA)而被广泛用于视频处理。然而,MHSA机制在视频修复方面遇到了内在的困难,因为与损坏区域相关的特征受到了破坏,并引起了不准确的自主关注。这个问题被称为查询退化,可以通过完成光学流并利用流指导自主关注来缓解,这在我们以前的工作中——流引导Transformer(FGT)得到了验证。我们进一步利用流指导并提出了FGT++,以追求更有效、更高效的视频修复。首先,我们使用 local 聚合和边缘损失设计了一个轻量级的流完成网络。其次,为了解决查询退化问题,我们提出了一个流指导特征整合模块,它利用运动差异来增强特征,并使用流指导特征传播模块根据流来扭曲特征。第三,我们沿着时间空间和空间维度分离Transformer,其中流通过一个可变形的MHSA机制来选择代币,全球代币通过双重MHSA机制与内窗口本地代币组合。FGT++进行了实验评估,被认为在定性和定量方面超越了现有的视频修复网络。
URL
https://arxiv.org/abs/2301.10048
