Abstract
In the latest years, videoconferencing has taken a fundamental role in interpersonal relations, both for personal and business purposes. Lossy video compression algorithms are the enabling technology for videoconferencing, as they reduce the bandwidth required for real-time video streaming. However, lossy video compression decreases the perceived visual quality. Thus, many techniques for reducing compression artifacts and improving video visual quality have been proposed in recent years. In this work, we propose a novel GAN-based method for compression artifacts reduction in videoconferencing. Given that, in this context, the speaker is typically in front of the camera and remains the same for the entire duration of the transmission, we can maintain a set of reference keyframes of the person from the higher-quality I-frames that are transmitted within the video stream and exploit them to guide the visual quality improvement; a novel aspect of this approach is the update policy that maintains and updates a compact and effective set of reference keyframes. First, we extract multi-scale features from the compressed and reference frames. Then, our architecture combines these features in a progressive manner according to facial landmarks. This allows the restoration of the high-frequency details lost after the video compression. Experiments show that the proposed approach improves visual quality and generates photo-realistic results even with high compression rates. Code and pre-trained networks are publicly available at this https URL.
Abstract (translated)
在最近几年，视频会议在个人和商务目的中已经取得了基本的地位。实现视频会议的关键技术是实现低带宽视频流实时视频传输的压缩算法，因为它们减少了所需的带宽。然而，损失性的视频压缩降低了视觉质量。因此，近年来提出了许多减少压缩伪影和提高视频视觉质量的技术。在这项工作中，我们提出了一个基于GAN的新方法来减少视频会议中的压缩伪影。 由于在当前视频中，讲话者通常在摄像头前，且在整个传输过程中保持不变，我们可以从传输中的高质量I帧中提取一系列参考关键帧，并利用它们来指导视觉质量的提高。这种方法的一个新颖之处是更新策略，它维护并更新了一个紧凑而有效的参考关键帧集。首先，我们从压缩和参考帧中提取多尺度特征。然后，根据面部特征点将这些特征以渐进的方式组合在一起。这允许在视频压缩后恢复高频细节。实验证明，与高压缩率相比，所提出的方法可以提高视觉质量并产生照片现实的结果。代码和预训练网络可以在该https URL上获取。
URL
https://arxiv.org/abs/2311.04263
