Abstract
In the recent development of conditional diffusion models still require heavy supervised fine-tuning for performing control on a category of tasks. Training-free conditioning via guidance with off-the-shelf models is a favorable alternative to avoid further fine-tuning on the base model. However, the existing training-free guidance frameworks either have heavy memory requirements or offer sub-optimal control due to rough estimation. These shortcomings limit the applicability to control diffusion models that require intense computation, such as Text-to-Video (T2V) diffusion models. In this work, we propose Taming Inference Time Alignment for Guided Text-to-Video Diffusion Model, so-called TITAN-Guide, which overcomes memory space issues, and provides more optimal control in the guidance process compared to the counterparts. In particular, we develop an efficient method for optimizing diffusion latents without backpropagation from a discriminative guiding model. In particular, we study forward gradient descents for guided diffusion tasks with various options on directional directives. In our experiments, we demonstrate the effectiveness of our approach in efficiently managing memory during latent optimization, while previous methods fall short. Our proposed approach not only minimizes memory requirements but also significantly enhances T2V performance across a range of diffusion guidance benchmarks. Code, models, and demo are available at this https URL.
Abstract (translated)
近期，在条件扩散模型的发展中，为了在某一类任务上进行控制，仍然需要大量的监督微调。通过现成模型的引导来进行无需训练的条件设置是一种有利的选择，可以避免对基础模型进一步进行微调。然而，现有的无需训练的引导框架要么有较大的内存需求，要么由于粗略估计而导致提供次优的控制效果。这些缺点限制了其在需要大量计算的任务控制扩散模型（如文本到视频(Text-to-Video, T2V)扩散模型）中的应用。 为此，我们提出了一种名为TITAN-Guide的方法，用于引导式文本到视频扩散模型的时间推理对齐技术，该方法解决了内存空间问题，并且相比现有方法，在指导过程中提供了更优的控制效果。特别是，我们开发出一种无需通过判别性引导模型进行反向传播来优化扩散潜在变量的有效方法。特别地，我们在各种方向指令选项下研究了引导式扩散任务中的正向梯度下降法。 在实验中，我们展示了我们的方法在处理潜在变量优化时有效管理内存的优势，而先前的方法在这方面表现不足。我们提出的方法不仅将内存需求最小化，而且显著提高了多种扩散指导基准测试下的T2V性能。代码、模型和演示可在以下链接获得：[此URL]。 请注意，在原始文本中有一个未提供的URL，建议您访问该链接以获取更多的信息和技术细节。
URL
https://arxiv.org/abs/2508.00289
