Abstract
The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at this https URL.
Abstract (translated)
三维场景的风格化在3D视觉领域是一个越来越吸引人的研究主题。尽管图像风格迁移已广泛研究并取得了一定成果，但直接将2D风格迁移方法应用于3D场景往往无法保留3D环境的结构和多视图属性，导致不同视角下的图像出现不愉快的失真现象。为解决这些问题，我们利用基于扩散模型的强大生成先验，并提出一种新的风格迁移方法OSDiffST，该方法基于预训练的一步扩散模型（即SD-Turbo）来渲染3D场景中多视图图像的各种风格。为了高效地将预训练模型适应于小数据集上的多视角风格转移任务，我们引入了视觉条件模块以从参考风格图像中提取样式信息作为扩散模型的条件输入，并采用LoRA在扩散模型中进行适配。此外，我们通过两个特定的损失函数考虑了风格化和内容图像之间的颜色分布对齐与结构相似性。因此，我们的方法能够有效保持风格化图像中的结构信息和多视图一致性，无需任何3D信息。实验表明，我们在为3D场景多视角图像合成各种风格方面超越了其他有前景的风格迁移方法。由我们提出的方法生成的不同视角下的风格化图像具有更优的视觉质量，具备更好的结构性完整性和更低的失真度。源代码可在以下链接获取：[此https URL]。
URL
https://arxiv.org/abs/2411.10130
