Abstract
Ultra-high quality artistic style transfer refers to repainting an ultra-high quality content image using the style information learned from the style image. Existing artistic style transfer methods can be categorized into style reconstruction-based and content-style disentanglement-based style transfer approaches. Although these methods can generate some artistic stylized images, they still exhibit obvious artifacts and disharmonious patterns, which hinder their ability to produce ultra-high quality artistic stylized images. To address these issues, we propose a novel artistic image style transfer method, U-StyDiT, which is built on transformer-based diffusion (DiT) and learns content-style disentanglement, generating ultra-high quality artistic stylized images. Specifically, we first design a Multi-view Style Modulator (MSM) to learn style information from a style image from local and global perspectives, conditioning U-StyDiT to generate stylized images with the learned style information. Then, we introduce a StyDiT Block to learn content and style conditions simultaneously from a style image. Additionally, we propose an ultra-high quality artistic image dataset, Aes4M, comprising 10 categories, each containing 400,000 style images. This dataset effectively solves the problem that the existing style transfer methods cannot produce high-quality artistic stylized images due to the size of the dataset and the quality of the images in the dataset. Finally, the extensive qualitative and quantitative experiments validate that our U-StyDiT can create higher quality stylized images compared to state-of-the-art artistic style transfer methods. To our knowledge, our proposed method is the first to address the generation of ultra-high quality stylized images using transformer-based diffusion.
Abstract (translated)
超高质量的艺术风格迁移是指使用从风格图像中学习到的样式信息来重绘一张超高清晰度的内容图片。现有的艺术风格迁移方法可以分为基于风格重建和内容-风格分离的两种类型。尽管这些方法能够生成一些具有艺术风格化的图像，但它们仍然表现出明显的伪影和不和谐模式，这限制了其生成超高质量的艺术风格化图像的能力。为了解决这些问题，我们提出了一种新的艺术图像风格迁移方法U-StyDiT，该方法基于变压器扩散模型（DiT）构建，并通过学习内容-风格分离来生成超高质量的艺术风格化图像。 具体来说，我们首先设计了一个多视图样式调制器（MSM），用于从局部和全局视角学习一个风格图像中的样式信息，从而使U-StyDiT能够利用学到的样式信息生成艺术化的图像。然后，我们引入了StyDiT模块，该模块可以从一幅风格图片中同时学习到内容与样式的条件。 此外，我们还提出了一套用于超高质量的艺术图像数据集Aes4M，包含10个类别，每个类别中有40万张风格化图片。这个数据集有效地解决了现有的艺术风格迁移方法由于数据集大小和数据集中图片质量的原因而无法生成高品质艺术风格化图片的问题。 最后，广泛的定性和定量实验验证了我们的U-StyDiT能够创建出比当前最先进的艺术风格化图像更高的质量的图像。据我们所知，这是第一次使用基于变压器扩散模型的方法来生成超高质量的艺术风格化图像。
URL
https://arxiv.org/abs/2503.08157
