Abstract
Generative Adversarial Networks (GANs), particularly StyleGAN and its variants, have demonstrated remarkable capabilities in generating highly realistic images. Despite their success, adapting these models to diverse tasks such as domain adaptation, reference-guided synthesis, and text-guided manipulation with limited training data remains challenging. Towards this end, in this study, we present a novel framework that significantly extends the capabilities of a pre-trained StyleGAN by integrating CLIP space via hypernetworks. This integration allows dynamic adaptation of StyleGAN to new domains defined by reference images or textual descriptions. Additionally, we introduce a CLIP-guided discriminator that enhances the alignment between generated images and target domains, ensuring superior image quality. Our approach demonstrates unprecedented flexibility, enabling text-guided image manipulation without the need for text-specific training data and facilitating seamless style transfer. Comprehensive qualitative and quantitative evaluations confirm the robustness and superior performance of our framework compared to existing methods.
Abstract (translated)
生成对抗网络（GANs），特别是StyleGAN及其变体，已经展示了在生成高度逼真图像方面的显著能力。尽管它们取得了成功，但在使用有限的训练数据进行领域适应、参考引导合成和文本引导操作等多样化任务上的应用仍然具有挑战性。为此，在这项研究中，我们提出了一种新颖框架，该框架通过超网络集成CLIP空间来大幅扩展预训练StyleGAN的能力。这种整合使得StyleGAN能够根据参考图像或文本描述动态地适应新领域。此外，我们还引入了由CLIP指导的判别器，以增强生成图像与目标领域的对齐度，确保更优的图像质量。我们的方法展示了前所未有的灵活性，使文本引导的图像操作无需特定于文本的训练数据，并促进了无缝风格迁移。全面的定性和定量评估证实了我们框架相对于现有方法的强大稳健性和卓越性能。
URL
https://arxiv.org/abs/2411.12832
