Abstract
Stylizing a dynamic scene based on an exemplar image is critical for various real-world applications, including gaming, filmmaking, and augmented and virtual reality. However, achieving consistent stylization across both spatial and temporal dimensions remains a significant challenge. Most existing methods are designed for static scenes and often require an optimization process for each style image, limiting their adaptability. We introduce ZDySS, a zero-shot stylization framework for dynamic scenes, allowing our model to generalize to previously unseen style images at inference. Our approach employs Gaussian splatting for scene representation, linking each Gaussian to a learned feature vector that renders a feature map for any given view and timestamp. By applying style transfer on the learned feature vectors instead of the rendered feature map, we enhance spatio-temporal consistency across frames. Our method demonstrates superior performance and coherence over state-of-the-art baselines in tests on real-world dynamic scenes, making it a robust solution for practical applications.
Abstract (translated)
基于样例图像对动态场景进行风格化对于游戏、电影制作以及增强和虚拟现实等众多实际应用场景至关重要。然而，在空间和时间维度上保持一致的风格化仍然是一项重大挑战。大多数现有方法针对静态场景设计，通常需要为每张风格图片进行优化过程，限制了它们的适应性。我们引入了一种名为ZDySS的零样本动态场景风格化框架，使我们的模型能够在推理过程中推广到之前未见过的风格图像上。 该方法采用了高斯点绘技术来表示场景，并将每个高斯与一个学习得到的特征向量相关联，这个特征向量可以为任意视角和时间戳生成特征图。通过在学习得到的特征向量而不是渲染后的特征图上应用样式迁移，我们提高了帧间的空间-时间一致性。 我们的方法在针对真实世界动态场景的测试中展示了超越现有最先进技术基线的表现力和连贯性，使其成为实际应用场景中的一个稳健解决方案。
URL
https://arxiv.org/abs/2501.03875
