Abstract
Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, Semantic Style Transfer, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, Semantix an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, Semantix be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, Semantix utilizes a meticulously crafted energy function to guide the sampling process, including three key components: Style Feature Guidance, Spatial Feature Guidance and Semantic Distance as a regularisation term. Experimental results demonstrate that Semantix not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields. The project website is available at this https URL
Abstract (translated)
最近在风格和外观转移方面的进展令人印象深刻，但大多数方法要么单独处理全局风格或局部外观的转换，要么忽视了语义对应关系。此外，图像和视频任务通常被孤立地处理，很少关注将它们集成以进行视频转移的方法。为了解决这些限制，我们引入了一个新的任务：语义风格迁移（Semantic Style Transfer），它涉及根据语义对应关系从参考图像中传输样式和外观特征到目标视觉内容。 随后，我们提出了一种无需训练的方法——Semantix，这是一个能量引导采样器，专门为语义风格转移设计。该方法基于预训练扩散模型的语义理解能力同时指导样式和外观转换。此外，作为采样器，Semantix可以无缝应用于图像和视频模型，使语义风格迁移能够在各种视觉媒体中通用。 具体而言，在将参考图和上下文图或视频通过SDEs反演到噪声空间后，Semantix利用精心设计的能量函数来指导采样过程。该能量函数包括三个关键组成部分：样式特征引导、空间特征引导以及作为正则化项的语义距离。 实验结果表明，Semantix不仅能够有效地完成图像和视频之间的语义风格迁移任务，并且在两个领域都超越了现有的最先进解决方案。该项目网站可在此处访问（请将“this https URL”替换为实际网址）。
URL
https://arxiv.org/abs/2503.22344
