Abstract
Generating images with both photorealism and multiview 3D consistency is crucial for 3D-aware GANs, yet existing methods struggle to achieve them simultaneously. Improving the photorealism via CNN-based 2D super-resolution can break the strict 3D consistency, while keeping the 3D consistency by learning high-resolution 3D representations for direct rendering often compromises image quality. In this paper, we propose a novel learning strategy, namely 3D-to-2D imitation, which enables a 3D-aware GAN to generate high-quality images while maintaining their strict 3D consistency, by letting the images synthesized by the generator's 3D rendering branch to mimic those generated by its 2D super-resolution branch. We also introduce 3D-aware convolutions into the generator for better 3D representation learning, which further improves the image generation quality. With the above strategies, our method reaches FID scores of 5.4 and 4.3 on FFHQ and AFHQ-v2 Cats, respectively, at 512x512 resolution, largely outperforming existing 3D-aware GANs using direct 3D rendering and coming very close to the previous state-of-the-art method that leverages 2D super-resolution. Project website: this https URL.
Abstract (translated)
生成具有照片写实性和多角度3D一致性的图像对于3D意识GAN至关重要,但现有方法却难以同时实现。通过基于卷积神经网络的2D超分辨率可以破坏严格的3D一致性,而通过学习直接渲染的高质量3D表示常常会导致图像质量下降。在本文中,我们提出了一种新的学习方法,即3D到2D模仿,该方法可以让3D意识GAN生成高质量图像,同时保持严格的3D一致性,通过让生成器3D渲染分支生成的图像模仿其2D超分辨率分支生成的图像。我们还将3D意识Convolutions引入生成器以更好地学习3D表示,这进一步改善了图像生成质量。通过以上方法,我们的方法和FFHQ和AFHQ-v2猫在512x512分辨率下分别获得了5.4和4.3的FID得分, largely outperforms existing 3D意识GAN使用直接3D渲染并接近于之前利用2D超分辨率的优势的方法的最先进的方法。项目网站:这个https URL。
URL
https://arxiv.org/abs/2303.09036
