Abstract
Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.
Abstract (translated)
在扩散模型方面取得的进展显著提升了视频质量，但也引发了对精细化控制的关注。然而，许多现有的方法依赖于为特定任务微调大规模视频模型，这随着模型规模的增长变得越来越不切实际。在这项工作中，我们提出了帧引导（Frame Guidance），这是一种基于帧级信号（如关键帧、风格参考图像、草图或深度图）的无需训练即可实现可控视频生成的方法。为了提供实用的无训练指导，我们提出了一种简单的潜在处理方法，显著减少了内存使用，并应用了一种针对全局连贯视频生成设计的新颖潜在优化策略。 帧引导能够跨多种任务（包括关键帧引导、风格化和循环）进行有效控制，无需任何培训，并且与任何视频模型兼容。实验结果显示，帧引导可以为广泛的任务和输入信号产生高质量的可控视频。
URL
https://arxiv.org/abs/2506.07177
