Abstract
Diffusion models have achieved remarkable progress in image and video stylization. However, most existing methods focus on single-style transfer, while video stylization involving multiple styles necessitates seamless transitions between them. We refer to this smooth style transition between video frames as video style morphing. Current approaches often generate stylized video frames with discontinuous structures and abrupt style changes when handling such transitions. To address these limitations, we introduce SOYO, a novel diffusion-based framework for video style morphing. Our method employs a pre-trained text-to-image diffusion model without fine-tuning, combining attention injection and AdaIN to preserve structural consistency and enable smooth style transitions across video frames. Moreover, we notice that applying linear equidistant interpolation directly induces imbalanced style morphing. To harmonize across video frames, we propose a novel adaptive sampling scheduler operating between two style images. Extensive experiments demonstrate that SOYO outperforms existing methods in open-domain video style morphing, better preserving the structural coherence of video frames while achieving stable and smooth style transitions.
Abstract (translated)
扩散模型在图像和视频风格化方面取得了显著的进展。然而，大多数现有的方法主要集中在单一风格转换上，而涉及多风格的视频风格化则需要这些风格之间能够无缝过渡。我们称这种视频帧之间的平滑风格转变过程为“视频风格变形”。当前的方法往往在处理这类过渡时会产生结构不连续且风格变化突兀的问题。为了克服这些问题，我们引入了SOYO，这是一种用于视频风格变形的新颖扩散框架。我们的方法利用预先训练好的文本到图像的扩散模型而不进行微调，并结合注意力注入和AdaIN（归一化激活）技术以保持结构一致性并实现视频帧之间平滑的风格过渡。此外，我们注意到直接应用线性等距插值会导致不均衡的风格变形。为了在整个视频帧中保持一致，我们提出了一种新的自适应采样调度器，在两种样式图像之间运行。广泛的实验表明，SOYO在开放领域的视频风格变形方面优于现有方法，不仅更好地保留了视频帧的结构一致性，还实现了稳定且平滑的风格过渡。
URL
https://arxiv.org/abs/2503.06998
