Abstract
Multiple complex degradations are coupled in low-quality video faces in the real world. Therefore, blind video face restoration is a highly challenging ill-posed problem, requiring not only hallucinating high-fidelity details but also enhancing temporal coherence across diverse pose variations. Restoring each frame independently in a naive manner inevitably introduces temporal incoherence and artifacts from pose changes and keypoint localization errors. To address this, we propose the first blind video face restoration approach with a novel parsing-guided temporal-coherent transformer (PGTFormer) without pre-alignment. PGTFormer leverages semantic parsing guidance to select optimal face priors for generating temporally coherent artifact-free results. Specifically, we pre-train a temporal-spatial vector quantized auto-encoder on high-quality video face datasets to extract expressive context-rich priors. Then, the temporal parse-guided codebook predictor (TPCP) restores faces in different poses based on face parsing context cues without performing face pre-alignment. This strategy reduces artifacts and mitigates jitter caused by cumulative errors from face pre-alignment. Finally, the temporal fidelity regulator (TFR) enhances fidelity through temporal feature interaction and improves video temporal consistency. Extensive experiments on face videos show that our method outperforms previous face restoration baselines. The code will be released on \href{this https URL}{this https URL}.
Abstract (translated)
在现实生活中，低质量的视频脸部存在多个复杂降解。因此，盲视频脸部修复是一项高度具有挑战性的 ill-posed 问题，不仅需要高保真度的图像，还需要增强跨不同姿态变化的时间一致性。在 naive 的独立修复每个帧的方式下，难免引入了姿态变化和关键点定位误差带来的时间不一致和伪影。为了解决这个问题，我们提出了第一个没有预对齐的盲视频脸部修复方法——具有新颖的分词引导的时间一致性变换器 (PGTFormer)。PGTFormer 利用语义分词指导来选择生成具有时间一致性伪影的最佳人脸 prior。具体来说，我们在高质量视频脸数据集上预训练一个时间-空间向量量化自编码器，以提取充满表达性上下文的 prior。然后，基于姿态解码指导的代码本预测器 (TPCP) 根据姿态解码上下文预测不同姿态的脸部。这种策略通过时间特征交互减少了伪影，并减轻了由于预对齐误差累积造成的抖动。最后，时间 fidelity 调节器 (TFR) 通过时间特征交互增加了 fidelity，并改善了视频的时间一致性。在面部视频的广泛实验中，我们的方法超越了以前的面部修复基线。代码将发布在 \href{this <https://this URL>}{this <https://this URL>}。
URL
https://arxiv.org/abs/2404.13640
