Abstract
Image style transfer models based on convolutional neural networks usually suffer from high temporal inconsistency when applied to videos. Some video style transfer models have been proposed to improve temporal consistency, yet they fail to guarantee fast processing speed, nice perceptual style quality and high temporal consistency at the same time. In this paper, we propose a novel real-time video style transfer model, ReCoNet, which can generate temporally coherent style transfer videos while maintaining favorable perceptual styles. A novel luminance warping constraint is added to the temporal loss at the output level to capture luminance changes between consecutive frames and increase stylization stability under illumination effects. We also purpose a novel feature-map-level temporal loss to further enhance temporal consistency on traceable objects. Experimental results indicate that our model exhibits outstanding performance both qualitatively and quantitatively.
Abstract (translated)
基于卷积神经网络的图像样式传递模型在应用于视频时通常遭受高时间不一致性。已经提出了一些视频样式传递模型来改善时间一致性，但是它们不能保证快速处理速度，良好的感知风格质量和高时间一致性。在本文中，我们提出了一种新颖的实时视频风格传输模型ReCoNet，它可以生成时间上连贯的风格转移视频，同时保持良好的感知风格。将新颖的亮度变形约束添加到输出电平的时间损耗，以捕获连续帧之间的亮度变化，并增加照明效果下的样式化稳定性。我们还针对一种新颖的特征 - 地图级时间损失，以进一步增强可追踪对象的时间一致性。实验结果表明，我们的模型在质量和数量上都表现出优异的性能。
URL
https://arxiv.org/abs/1807.01197
